Set Up Billing Alerts Before You Deploy Anything
The single highest-value fifteen minutes in cloud administration, and the one almost everybody postpones until after it would have helped.
Almost every alarming cloud bill story has the same shape. Something was
misconfigured, or left running, or attacked, and nobody found out until the
invoice arrived at the end of the month. The technical cause varies. The reason
it became expensive rather than merely annoying is always the same: there was no
alert.
This is a setup task, not a project. Do it before you deploy anything real.
Understand what an alert is and is not
A billing alert notifies you when spending passes a threshold. It does not
stop the spending. On most platforms there is no hard cap that shuts everything
down at a number you choose, and where something like that exists it usually
works by terminating resources, which is rarely what you want on a production
system.
So treat alerts as a smoke detector rather than a sprinkler. Their job is to
get a human looking at the console within hours instead of weeks.
The checklist
- Set a monthly budget at the figure you expect. Not the
figure that would be a disaster. The figure you have planned for. If you expect
to spend forty, the budget is forty. - Add alerts at several points against it. A notification at
roughly half, one at the budget itself, and one above it. The low threshold is
the useful one: it fires early enough in the month that you can still act. - Add a forecast alert if your provider offers one. Forecast
alerts fire when projected spend will exceed the budget, which catches a runaway
resource days before actual spend reflects it. - Send alerts to an address a person reads. The root account
address is often a mailbox nobody opens. Point alerts at a real inbox, and at a
chat channel if you have one. - Add a second recipient. One person is on holiday the week
it matters. - Check the alert actually arrives. Set a temporary threshold
below your current spend, confirm the notification lands, then set it back. An
untested alert is an assumption.
Alerts lag
Billing data is not real time on most platforms. There is typically a delay
of hours between a charge being incurred and it appearing in the figures your
alert is watching. A sudden, severe cost event can run for some time before the
alert fires, which is an argument for setting the first threshold low rather
than clever.
Thresholds that work
If you have no history, pick a budget by pricing your architecture, then set
the first alert at roughly half of it. You will get a notification in the first
month telling you whether your estimate was right, which is useful information
in itself.
If you do have history, set the budget slightly above a normal month rather
than at the maximum you could tolerate. An alert that only fires during a
genuine emergency is one you have never tested and may not trust.
Resist the urge to raise a threshold the first time it fires. The alert
working as designed is not a reason to disable it. Find out what changed.
What to do when one fires
Work through it in this order, because it moves from most likely to least:
- Check for resources you created and forgot. Test instances, volumes,
snapshots, load balancers and idle managed databases. - Check outbound data transfer. A sudden egress spike means either real
traffic, a misconfigured backup job copying far more than intended, or someone
hotlinking your files. - Check whether a credit or free tier just expired. The usage did not change;
the subsidy did. - Check for anything you did not create. Unexpected instances in a region you
never use is the classic signature of a compromised set of credentials, and it
is the one case where speed genuinely matters.
That last point is worth stating plainly. The largest unexpected cloud bills
are usually not caused by carelessness with your own resources. They are caused
by access keys leaking and being used to run compute at scale. If you find
resources nobody on your team created, rotate credentials before you do anything
else, and read our note on
securing a new account.
One more thing worth doing
Tag or label your resources by project or environment from the start, if your
provider supports it. It costs nothing at creation time and it is the difference
between an invoice that says “compute” and one that tells you which project is
responsible. Retro-fitting tags to an account with fifty resources is a tedious
afternoon; applying them as you go is free.