← All posts

Blog

The Bug That Waited 30 Days: What AI Code Costs When It Breaks Quietly

AI wrote a feature that cost pennies a day. On day 30 an expiry job woke up, ran millions of database reads and writes, and cost hundreds before we caught it.

3 min read

The Bug That Waited 30 Days: What AI Code Costs When It Breaks Quietly

For a month, the feature cost us pennies a day.

Then it cost us $47.60 in a single day, and it was climbing.

AI wrote the code. The code worked. It passed review, it shipped, and for 30 days it did exactly what we asked. Nothing looked wrong because nothing was wrong yet.

Daily cloud database cost for one AI-written feature: well under a dollar a day for weeks, then a jump on day 30 to over $10 a day, climbing to $47.60

What happened

The feature included a query to expire old records after 30 days. Reasonable. Every system that stores data needs some cleanup.

The first 30 days, there was nothing old enough to expire. So the query had nothing to do, and it cost nothing.

On day 30, it kicked in. It started doing millions of database reads and writes. The daily bill went from under a dollar to over ten, and it kept climbing every day after that. By the time we caught it, it had cost us a few hundred dollars.

A few hundred dollars will not sink anybody. The lesson was worth a lot more than that. On a bigger system, with more data, the same mistake is a five-figure surprise.

AI does the thing. It does not feel the bill.

AI is very good at writing code that does what you asked. It is much weaker at thinking through what that code does next month.

It wrote the expiry logic correctly for the case in front of it. It did not stop and ask what happens when the query runs against a full month of data, every day, on a database that charges per read and per write. Those are ramifications, and you only see ramifications if you are looking for them.

A senior engineer has been burned by this before. The scar tissue is what makes them ask, "What does this cost at scale?" AI has no scar tissue. It has never paid a cloud bill.

That is the part that makes AI-written code dangerous. The code is usually fine. The problem shows up later.

Cheap until it is expensive

Here is the uncomfortable math on AI in production.

When it is right, AI work is extremely cheap. A feature that used to take a week ships in an afternoon and costs pennies to run.

When it is wrong, it can be extremely expensive, and the cost does not show up when you would expect. Our problem had a 30-day fuse. Code review was a month behind it. Testing was a month behind it. The only thing that would have caught it in time was a cost alert, and that is the piece we had not set up.

Delayed failures are the hardest kind. Everyone has moved on. The pull request is closed. Nobody connects the spike on day 30 to the change from day one.

What we changed

This was a great lesson for our team, and it changed how we review AI work.

  • Double-check the work, including the future. Reviewing what the code does today is half the job. Ask what it does at day 30, day 90, and at 10 times the data.
  • Make sure all the scenarios are there. Empty tables, full tables, the first run of every scheduled job, the day the cleanup has a month of backlog. If a scenario involves time passing, write it down.
  • Put cost metrics on it. Every feature that touches a paid service should have a number attached: reads, writes, calls, dollars per day.
  • Install cost alerts. Daily spend alerts and budget caps on every cloud account. This is the cheapest insurance in tech, and it is the one that would have caught us on day one of the spike instead of day ten.

What to do this week

  1. Turn on billing alerts for every cloud provider you use. Set a daily threshold a little above normal and send it somewhere a human reads.
  2. List every scheduled job, expiry, cleanup, and TTL in your systems. Ask what each one does on its first real run.
  3. Add one question to code review for anything AI wrote: "What does this cost per day at 10x the data?"
  4. Look at your last 60 days of spend by service. Find the line that is growing faster than your users.

The point

Use AI. It is the biggest leverage shift I have seen in 28 years of building software. Our feature was cheap, fast, and correct for 30 days.

Then it was expensive. AI will happily write the code. Someone still has to own what it costs.

Have you been hit by a delayed bill like this? I would like to hear how you caught it.