How Much It Costs to Self-Host OTA Updates for 100 Million Users

This article is written by Zach Goldie

How much does it actually cost to self-host over-the-air (OTA) updates? Especially for an app with many millions of users.

It’s a useful number to know if you’ve got a growing Expo Updates bill.

This article prices an extreme case of 100 million monthly active users (MAU). Here is the monthly cost of hosting Codemagic Patch on DigitalOcean at a few sizes:

You can host Patch on any Linux compatible machine, such as an EC2 on AWS or a machine on Fly.io. We’re just using DigitalOcean as an example.

Requirements for hosting Patch

Patch is designed to be easy to run and adaptable to different requirements. The key pieces it needs are:

• A server for metrics, the dashboard, and publishing
• Object storage for the manifests and the bundles.
• Postgres for the metric rows and the release records.
• A CDN, optionally, in front of the storage hostname for manifests and bundles.

A small deployment can keep all of this on a single 2 GB machine. How to self-host OTA updates for React Native walks through standing that up. In these cases, the install wizard uses Caddy for HTTPS on the API and storage hostnames, and Silo for object storage. For 100 million users we will split the pieces apart.

Understanding the traffic for Patch

To plan our architecture, let’s talk about what requests and posts are made.

Requests

Devices will make a request for whether an update is available. The answer is provided as meta.json and manifest.json, a pair of files created along with each release that will be kept in object storage and served by the CDN. Designing scalable OTA update architecture for React Native explains why that check is a file on the CDN rather than a call to the server.

When an update is available, the bundle will also be fetched from the CDN. So, you won’t deal with millions of device requests hitting your server.

Posts

The posts are the phones reporting back. Receiving and installing an update posts Downloaded, then Ready and Applied. The phone also posts Active, to say what it is running, at most once per day.

These are initially stored as an individual row per device, then rolled up into daily totals after eight days.

Load estimates for 100M users

We will need to make some assumptions to decide the required spec.

We’ll say 1/4 of the monthly active users open the app on a given day, meaning 25 million update checks and Active posts per day.

Let’s also assume one OTA update per week. The amount of TB needed for devices to download these updates is Cloudflare’s problem, and won’t affect the bill or our deployment. On our end, releasing an update only means three extra post events per device.

The rates are those assumptions spread across a day.

Sizing the infrastructure

Do not size the droplet from the downloads. Size it from Active, then add the ship-day posts.

Server and throughput

A VM with a dedicated CPU is useful for the incoming metrics. The most basic one on DigitalOcean is 2 vCPUs with 8 GiB of RAM, at $63 a month.

A quick load test showed that this machine can post about 750 events per second to a basic managed Postgres. Faster than that, and the database couldn’t keep up.

To go beyond 750/s, it’s possible to add measures such as saving a batch of posts once a second. That isn’t currently included in Patch, but a quick test showed it increasing throughput to 4,000/s on the same setup.

Managed Postgres

Postgres is best on a separate managed database. You will need roughly 4 GiB per 1M MAU. About half of that is Active, and about half is the ship-day posts still inside the eight-day window.

For one million users, the basic managed Postgres is $15 a month. A $48 basic droplet beside it makes the $63 in the table above. At about 10 posts a second, that tier does not need the dedicated CPU.

For 100M users, we start needing one of the bigger offerings. 400GiB should be enough, but we’ll double it to 800GiB to be safe.

DigitalOcean has a PostgreSQL Advanced plan at $352 a month for 800 GiB, with 16 GiB of RAM and 4 vCPUs. It includes backups and point-in-time recovery.

Rollup time

On a busy day, rolling 50 million rows into the daily total would take about 25 minutes. Posts that arrive during that time are still stored.

Approaches such as partitioning the table by day could make that a quick task, but aren’t yet supported by Patch.

Adding redundancy

To keep publishing and metric posts going when a machine dies, add three pieces:

• A standby database node. It is promoted if the primary dies. Without one, the database is down while a replacement is provisioned. A standby does not take inserts itself.
• A second 2 vCPU droplet, so metric posts and publishing have a machine to land on. The first droplet already covers the busy hour.
• A load balancer in front of those droplets. In front of one droplet it does nothing when that droplet dies.

That covers a dead primary database or a dead droplet.

What’s not needed

Because requests don’t hit the server, pieces to deal with that load aren’t needed for Patch.

• No queue. The post is stored before the server acknowledges it. The phone retries on failure, and event_id keeps that retry from inserting a row that is already there.
• No HTTP cache in front of the API. The reads worth caching are the manifest and the bundle, which are already on the CDN. Posts are writes, so there is no response to cache.
• No Redis. It is often added for metadata, a cached update check, or metric counters. Metadata and metric rows are in Postgres, and the update check is those manifest files.
• No Kubernetes or API gateway. Two droplets and a load balancer cover a dead machine. Downloads stay on the CDN, so a gateway has no phone traffic to spread.
• No autoscaling. The droplet count does not follow download traffic, because that traffic never reaches the server.
• No multi-region Postgres. One database holds the eight-day window, and update checks do not read it.
• No timeseries database. A timeseries database such as Prometheus could give more granular analysis, but is option.

What you pay

At 100 million the month looks like this. Prices are the current droplet list, managed Postgres list, and load balancer price.

The license is the main item line, which is required above a million users.

Above that, the pricing page counts from the first million, in stacked bands.

Want to try out Patch?

To find out more about Patch and view the docs, visit patch.codemagic.io. It includes a local evaluation stack so you can play with it on your machine.

If you want to use it for over a million users, ask sales@codemagic.io for more information.

FAQ

Is the license free under a million users?

Yes, at or below 1,000,000 monthly active users. Above that, there’s a small fee per million MAU.

Does Cloudflare add a per-download bill?

Update checks and bundles are served from the CDN in front of the storage hostname. In this costing that line stays at $0.

Do I have to use DigitalOcean?

Patch runs on any Linux machine. DigitalOcean is just the example used here. You can also run it on platforms such as AWS, GCP or Fly.io.