Software Engineer — the frontend of a 4M DAU service, and the delivery path behind it
ScrollThe bill arrived with the code
I was writing frontend code for a service used by millions when the egress bill and the pager started landing on my desk as well. From then on I read the segments outside the viewport too — CDN, cache, servers, deploys, data.
Every figure here is a second measurement
I don't narrow a problem by intuition. I split it into segments with logs and events, find where the seconds and the bytes leak, change the structure, then measure the same metric again.
An LLM is one more system to verify
Not something special. Responses are checked against a schema, and quality is re-measured by replaying real sessions.
235GB/day
Cache keys and invalidation re-cut to how each resource changes, and a Service Worker took repeat requests off the network.
Seven events from request to click, defined and collected, then BigQuery to name the bottleneck. Then the bundle and the ad script changed places.
Faster meant more seen, more seen meant more clicked, more clicked meant more revenue. All three were verified in sequence.
253M
4.27 TB
Image delivery on CloudFront and Lambda@Edge. A different measurement from the cache figures above; the two are never summed.
1 ← 10+
Ten-plus folded into one multi-tenant service, migrated one DNS record at a time. Maintenance headcount settled at about 20% of before.
Raise the hit rate and requests stop earlier. Only the ones that reach the origin are billed as transfer.
235GB/day
56%
417 GB/day and 235 GB/day are the measured values. Everything between them is a linear model for illustration, not a bill.
A health check alert surfaced the fault; I connected to the server directly.
I checked state through Portainer and SSH and inspected Nginx. Nothing healthy was listening on the service's 8080 port. I brought production back up with Docker Compose.
I did not stop at recovery but went looking for why the process was gone — which is where the log accumulation in OPS-02 came from.
While inspecting server state I found PM2 logs had grown to roughly 21 GB.
I cleared the logs to reclaim the disk. Clearing is a reprieve rather than a fix, though: the same thing happens again.
I applied pm2-logrotate with max_size 10M, retain 7, compression, and date-based file management and rotation.
Ping-based monitoring calls a healthy service down when ICMP is blocked, and calls a host healthy when the HTTP service behind it is dead.
I separated ICMP reachability from application state, and reviewed HTTP-based checks with curl.
I set the standard that an alert only means something if server failure, service failure, network failure and monitoring failure can be told apart.
A network policy change blocked outbound connectivity on the port the production server used, and the service could not be changed on the spot.
I set up a temporary TCP relay path through a server that still had connectivity, keeping the service running.
Once normal network conditions returned I moved back to the original structure. The temporary path was not left in place.
Connection problems appeared around an RDS CA rotation and expiry — and results differed by client, with DataGrip connecting where the CLI failed.
I compared each client's trust store and TLS handling to establish which one verified the certificate how.
Recorded: in production, the difference between client environments is often the cause of a connection failure rather than the certificate itself.
A Docker container on the production server exited with code 1.
Cross-referencing container state, timestamps and logs pinned down when it exited.
The root cause was not established. An unverified cause is not written down as one.
Making technology decisions as SCG's president, I leaned hard at first toward what was already proven. A teammate told me I demanded a much higher standard of proof from new approaches than from the risk of the status quo.
They were right. So instead of asking whether something was stable, I changed the question to two: how far does this change reach, and how far can we roll it back if it goes wrong?
The person proposing an idea does not carry the whole burden of proving it safe. Deciding how much verification a change needs is the team's job.
Try it in a small scope first. Agreement should not cost more than the change.
Widen the review and the people in it — more so the less reversible the change is.
Confirm the path back, then go. The sequential DNS cutover was the same reasoning.