One machine in Tokyo, on purpose
MapSocket's API, tile workers and edge proxy share a single Fly.io machine. Why that's the right shape for now, and how raw TCP lets customers bring their own tile domains.
Most architecture diagrams start with a load balancer and a fleet. Ours starts with one machine on Fly.io, in Tokyo. That isn’t an oversight. It’s what the product needs today, and the reasons are worth writing down.
What lives on it
Four kinds of process share the machine:
- The API, in Rust: accounts, geofencing, tracking, webhooks.
- tilefeed workers, one per tile source. Each one listens to a customer’s PostGIS database, regenerates the vector tiles a change touched, and serves them.
- A small Postgres with PostGIS and
ogr_fdw, for external data: GeoJSON, CSV and other files our users upload or point us at, read in place as foreign tables. - Caddy, the edge: TLS, routing by hostname, and the reverse proxy in front of everything else.
They share a volume at /data, and that’s the main reason they share a machine. A Fly volume attaches to exactly one
machine. The external-data database reads files the API writes, and tile workers keep their state on disk. Splitting
these apart would mean a network file system or object storage in the middle of every request, and we’d rather not
pay for that until scale asks for it. The volume gets daily snapshots, kept for a week.
If any process exits, the machine restarts. There’s no clever supervisor to get wrong.
Raw TCP, so Caddy holds the certificates
Fly can terminate TLS for you. We don’t let it. The app’s services are plain TCP on ports 80 and 443, passed through untouched to Caddy, which gets its own certificates from Let’s Encrypt.
That’s what makes custom tile domains work. A customer CNAMEs tiles.example.com to our edge hostname; the
first HTTPS request for that name reaches Caddy, which asks the API one question (is this hostname verified for an
account?) and, if the answer is yes, gets a certificate on the spot. Nobody files a ticket, and we never hold a
certificate for a name nobody verified.
Fly passes connections with a PROXY protocol header, so Caddy, and behind it the API, sees each visitor’s real IP address rather than Fly’s proxy. Per-IP rate limits depend on that.
Why Tokyo
Latency between the API and its control-plane database matters more than latency to any one user: a single API
request can make several database round trips. Our control plane runs on Supabase in Tokyo, so the machine sits in
Fly’s Tokyo region next to it. Tiles are cacheable (every tile carries an ETag and Cache-Control), so a CDN can
bring them closer to users later without moving anything else.
Why one, and when that changes
Rate limits are kept in memory and the volume belongs to one machine, so we deploy with high availability switched off, on purpose. A deploy restarts the machine, and live tile streams reconnect on their own.
This will change. The first step will probably be a CDN in front of tile traffic, then separating tile workers from the API once one machine’s CPU is the limit. The design keeps those steps cheap: the tile edge only needs a hostname, the API is stateless apart from rate limits, and the workers already run as separate processes.
Until then, one machine is easy to reason about, cheap to run, and quick to deploy.