Feature Engineering That Survives Production

A model in production relies on the numbers going in on Tuesday meaning what they meant during training. Data leakage, freshness changes and schema drift can break that assumption even when the fitted model itself has not changed.

Rule 1: every feature has an as-of time

For each feature, answer one question: at the moment of prediction, was this value available and final?

Three things break this:

• Late-arriving data. Yesterday's sales are 94% complete at 6am and 100% complete at noon. Train on the noon version, score on the 6am version, and the model sees a systematically lower number than it learned on.
• Restatements. Returns, cancellations and corrections rewrite history. A backtest reading today's table sees a version of last March that nobody could have seen last March.
• Slowly changing dimensions. A store's format changed in 2025; the dimension table only holds the current value, so every historical row is labelled with today's format.

The control for all three is the same: point-in-time correct joins. Features are computed against the state of the world as of the prediction timestamp, not the state of the world now. This is necessary if a backtest is to provide a meaningful estimate of production behaviour.

Rule 2: align lags and windows with the forecast origin

For a target four weeks after the forecast origin, a one-week lag defined relative to the target period refers to data three weeks after that origin and is therefore unavailable. A one-week lag defined relative to the forecast origin is available, but it represents data five weeks before the target. Define the reference point explicitly, and ensure every rolling window ends at or before the feature cutoff.

Confusing those reference points is a common source of time-series leakage: a feature exists in a completed historical table but not at the corresponding live scoring time.

Rule 3: handle new entities explicitly

New SKUs, new stores, new customers arrive constantly, and every history-based feature is null for them. Three options, in order of preference:

1. Attribute-based fallback — impute from comparable entities (same category, price band, region).
2. Explicit missing indicator — let the model learn "this is new" as a signal in its own right.
3. Separate cold-start model — when new entities are a meaningful share of volume.

Filling with zero is a risky fourth option because zero can mean either genuine absence or missing history; encode that distinction explicitly.

Rule 4: encode high-cardinality categories carefully

Target encoding of SKU, store or customer is powerful and leaks readily. Encoding computed on the full training set gives every row information about its own target.

Use out-of-fold encoding, computed within the training folds only, with smoothing toward the global mean for low-count categories. And recompute encodings on the same schedule as retraining — a stale encoding map is a slow, silent accuracy drain.

Rule 5: monitor the inputs, not just the outputs

Accuracy metrics tell you something broke, weeks after it broke. Input monitoring tells you what and when. Track per feature, per scoring run:

Upstream schema changes, renamed categories and stale deliveries are common failure modes. These checks can surface the issue before a forecast reaches a planner, provided thresholds are calibrated and alerts are monitored.

Rule 6: one definition, one place

If "active customer" is independently defined in model code, a dashboard and a finance report, the definitions can diverge. A shared, versioned feature definition helps keep training, scoring and reporting reconcilable.

That reconciliation is part of earning operational trust in the output.