Machine learning workflow

Intuition

The part of machine learning that gets taught is choosing and fitting a model. The part that decides whether the work survives is everything wrapped around it: what question you asked, where the data came from, what happens when the world moves, and who notices.

The framing that best predicts how this goes is not "a project" but debt. Building the prediction system is fast; keeping it correct is not. The standard reference on this argues "it is dangerous to think of these quick wins as coming for free" and that "it is common to incur massive ongoing maintenance costs in real-world ML systems" [2, abstract].

The workflow exists to make those costs visible while they are still cheap to change — which mostly means early.

Formal definition

There is no theorem here; the workflow is an engineering discipline, and the most useful formal statement of it is the phase ordering that experienced practice converges on. Google's version has three phases and a precondition, and the precondition is the striking part: Rule 1 is "don't be afraid to launch a product without machine learning", because "machine learning is cool, but it requires data" [1, Rule 1].

The phases that follow are, in order, Your First Pipeline, Feature Engineering, then Slowed Growth, Optimization Refinement, and Complex Models [1, ML Phase I–III]. Note what that ordering asserts: the sophisticated model is the last phase, after the infrastructure and the columns are right. Its Phase I opens with "keep the first model simple and get the infrastructure right" [1, Rule 4].

How it works

Framing. Decide what is being predicted and what will be done with the prediction. The guidance is to "design and implement metrics" before the model exists [1, Rule 2], and to pick "a simple, observable and attributable metric for your first objective" [1, Rule 13].

Data and splits. Everything downstream inherits this. Hold out a test set and keep it closed, because tuning against it leaks: "knowledge about the test set can 'leak' into the model and evaluation metrics no longer report on generalization performance" [3, §3.1].

Features. Once a working pipeline exists, this is where "there is a lot of low-hanging fruit", which is why Google makes it the second phase [1, ML Phase II].

Training. The step everyone pictures, and the one that changes least between projects.

Evaluation. Against the metric chosen in step one, on data chosen in step two, with cross-validation where a single split is too noisy to decide on [3, §3.1].

Serving, and the edge back. The dashed return path is the workflow, not an appendix to it. Production changes the data — through feedback loops, through changes in the outside world, through consumers you did not know you had. The technical-debt analysis names these precisely: "boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, changes in the external world, and a variety of system-level anti-patterns" [2, abstract].

Two rules exist purely to keep the two halves of that loop consistent: save the feature set used at serving time and train from it [1, Rule 29], and measure training/serving skew as a first-class metric [1, Rule 37].

Where it's used

• Any model that ships. A model in a notebook needs the left half; a model making decisions needs the loop.
• Regulated settings — credit, insurance, medicine — where the audit question is about provenance and versioning, not accuracy.
• Recommendation and ranking, where the feedback edge is not a metaphor: the model's output becomes tomorrow's training data, which is the hidden feedback loop by name [2, abstract].
• Team hand-offs. The workflow is also an interface contract — it defines what analysts, engineers and reviewers each own.
• Post-mortems. When a deployed model degrades, the workflow is the list of places to look, in order.

Why & when

Follow the phase order, especially when it feels slow. The temptation is to start at the model, because that part is interesting and legible. Rule 4 gives the reason: the first model "provides the biggest boost to your product, so it doesn't need to be fancy", but "you will run into many more infrastructure issues than you expect" [1, Rule 4].

Prefer the simple, observable objective early. It is easier to improve a metric you can attribute than to argue about a metric you cannot [1, Rule 13].

Test the infrastructure separately from the learning [1, Rule 5]. Most failures are plumbing, and plumbing failures diagnosed as modelling failures cost weeks.

Know when not to do any of this. If a heuristic solves the problem, ship the heuristic [1, Rule 1]. The workflow is expensive and only pays back at a certain scale of consequence.

Worked example

Not a model — a split, which is where more projects go wrong than anywhere else. Two ways of dividing the same customer-visit data, and the gap between what they promise and what they deliver.

import numpy as np

rng = np.random.default_rng(3)
n_cust, per_cust = 300, 6

home = rng.normal(0, 1, (n_cust, 2))                    # where each customer sits
quirk = rng.random(n_cust) < 0.25                       # a quarter behave atypically
label = (home[:, 0] + home[:, 1] > 0) ^ quirk           # generalisable part + quirk

customer = np.repeat(np.arange(n_cust), per_cust)
X = home[customer] + rng.normal(0, 0.05, (n_cust * per_cust, 2))
y = label[customer].astype(int)

def nearest_neighbour(train, test):
    d = ((X[test][:, None, :] - X[train][None, :, :]) ** 2).sum(-1)
    return np.mean(y[train][d.argmin(1)] == y[test])

rows = rng.permutation(len(customer))
random_split = np.zeros(len(customer), bool)
random_split[rows[:360]] = True                         # 20% of rows, scattered

held_out = np.isin(customer, np.arange(240, 300))       # 20% of customers, whole

print(f"random row split      accuracy {nearest_neighbour(~random_split, random_split):.3f}")
print(f"grouped by customer   accuracy {nearest_neighbour(~held_out, held_out):.3f}")

random row split      accuracy 0.808
grouped by customer   accuracy 0.631

Both numbers come from the same data and the same model. The first is measured on visits whose customer the model already trained on — and since each customer's visits sit almost on top of each other, the nearest neighbour is usually that same customer's other visit, quirk included. It is recognition scored as prediction. The second holds out whole customers, which is what deployment actually looks like: new people, whose quirks nobody has seen.

Eighteen points, on a decision made before any modelling started.

The workflow point is not the arithmetic. It is that this decision was made at the data stage, three boxes before anyone looked at a model, and no amount of later work reveals it. Evaluation can only check what the split allowed it to see.

Pitfalls

Optimising a metric nobody chose deliberately. Accuracy on an imbalanced problem is the classic — a model can be excellent by that measure and useless. Pick the metric in the framing stage [1, Rule 2].

Training/serving skew. The feature computed in the notebook and the feature computed in the service drift apart, silently, and the model degrades without any error being raised [1, Rule 37].

Undeclared consumers. Someone builds on your model's output without telling you, and now you cannot change it. Named directly among the debt mechanisms [2, abstract].

Treating deployment as the finish line. The world the training data described keeps moving; "changes in the external world" is on the risk list for that reason [2, abstract]. A model with no monitoring is a model whose failure you will hear about from a user.

Skipping the loop because the first version worked. The first version always works — it was fitted on the world as it was that week.