Which Cloud Provider for Machine Learning Workloads

Machine learning is two workloads that happen to share a name. Training is
bursty, expensive, interruption-tolerant and GPU-hungry. Inference is steady,
latency-sensitive and often modest. They pull provider choice in different
directions, and conflating them produces a bad decision for both.

Training: a batch job with expensive hardware

Training runs for hours or days, uses accelerators, and can usually tolerate
being interrupted if it checkpoints its progress. That last property matters
enormously, because it makes training the ideal candidate for discounted
interruptible capacity, as covered in
on-demand, reserved and spot.

What you need from a provider is GPU availability in a region you can use, at
a price you can tolerate, with enough storage throughput to keep the accelerator
fed. That third point is frequently the bottleneck: an expensive GPU waiting on
slow storage is the most wasteful configuration in this field.

Inference: an ordinary service with unusual hardware needs

Serving a model is a normal web service. It needs predictable latency,
reasonable availability and enough memory to hold the model. Many models serve
perfectly well on CPU, particularly smaller or quantised ones, and assuming a
GPU is required is a common and costly mistake.

Measure first. If CPU inference meets your latency requirement, you have just
removed the hardest constraint from your provider choice and can select on
ordinary criteria instead.

Where the platforms differ

Google Cloud is the
strongest fit when the work is data-centric. BigQuery for the data that feeds
training, Vertex AI for the managed training and serving layer, and GKE for
orchestration make a coherent stack rather than a collection of parts. If your
training data already lives in a warehouse, keeping compute next to it avoids
both transfer costs and a great deal of pipeline work.

AWS offers the broadest range
of accelerator instance types and the widest regional availability of them,
which matters because GPU capacity is genuinely scarce and availability varies
by region and by day.

Azure is the natural choice
for organisations already inside the Microsoft ecosystem, where identity,
governance and existing data estate integration outweigh marginal platform
differences.

IBM Cloud is worth
considering specifically where Watson services fit the problem and the
organisation is already an IBM customer.

Data gravity usually decides it

This is the factor that quietly overrides the others. Training data is large,
and moving it is slow and, because of egress charges, expensive. Whichever
platform already holds your data has a substantial and persistent advantage,
because the alternative is paying to extract it and paying again every time the
dataset is refreshed.

If your data is already somewhere, start there. If it is not yet anywhere,
this is the moment to choose deliberately, because the decision becomes harder
to reverse with every terabyte you accumulate. See
data egress fees explained.

A note on GPU availability

GPU capacity is constrained, and the published price is only relevant if the
instances are actually obtainable. Before building a plan around a specific
accelerator type, check availability in your intended region and be prepared for
quota requests, which are normal and can take time to approve.

This is a real planning input rather than a footnote. A project blocked for a
week on a quota approval is a common experience.

Where the smaller providers fit

The value-focused providers covered on this site are not accelerator
platforms, and for training workloads they are generally not the answer.

They are, however, entirely reasonable for the parts of an ML system that are
not training: data preprocessing, API servers wrapping a model, dashboards, and
CPU inference. A sensible pattern is training on a platform with the hardware
and serving on something cheaper and simpler, provided you account for moving
the model artefact between them, which is small and infrequent.

A practical starting point

1. Establish whether you need a GPU for inference, by measuring rather than
   assuming.
2. Find out where your training data already lives, and weight that heavily.
3. For training, use interruptible capacity and make checkpointing work before
   you rely on it.
4. Check accelerator availability and quota in your intended region before
   committing to a plan.
5. Keep serving infrastructure separable from training infrastructure, so each
   can be bought on its own terms.