Data onboarding

How your data flows to your dashboard

What happens between your nightly export and the plan a planner opens in the morning — including the nights it goes wrong.

9 min read9 sectionsIT leads · Data stewards
Examples describe a configurable workflow, not a contractual commitment. Confirm security, retention, schedules, integrations and automation for your own deployment before relying on them.
On this page
01

The journey, end to end

Seven steps from your export job to the number a planner reads at six in the morning.

You send files. We turn them into a plan. Nothing in between needs anyone at your end to be awake, and every stage is recorded — so the question “where is my data right now?” always has an answer.

  your ERP / warehouse export
            |
            v
  [ your private inbox ] -------- we pick it up, usually within minutes
            |
            v
  [ permanent record ] ---------- your exact bytes, kept before anything reads them
            |
            v
  [ checks ] -------------------- readable > columns > values > references > volume
            |
            +--- held -----------> nothing loaded, we tell you promptly
            |
            v
  [ loaded ] -------------------- rejected rows go to a report you can download
            |
            v
  [ overnight calculation ] ----- forecast, allocation, ordering, alerts
            |
            v
  [ published ] ----------------- the whole day appears at once, or not at all
            |
            +--------------------> your dashboard, 06:00
            |
            +--------------------> approved plans back to your ERP
  1. 1
    Your export runs
    Your ERP, warehouse or file-transfer tool writes the night’s files — one per feed, or several numbered parts for a large one. You choose the schedule; we only need it to finish before your agreed cutoff.
  2. 2
    The files land in your private folder
    Each file is written into the folder we give you at onboarding. Nobody else can see it. Nothing is read while it is still being written: a file becomes visible to us only once the upload has completed in full.
  3. 3
    You tell us the delivery is finished
    After the last data file, you write one small completion file (manifest) listing what you just sent. This is the difference between calculating on your whole delivery and calculating on part of it. Section 03 is entirely about it.
  4. 4
    We pick it up, usually within minutes
    We copy your exact bytes and keep them, unmodified, as the permanent record of what you sent — before anything reads or interprets them. That copy is what answers “what did we actually receive, and when?” months later.
  5. 5
    We check it
    Six checks, in order, from “does this file open?” to “is this volume plausible against your own last five deliveries?”. Nothing is written into your workspace until they pass.
  6. 6
    We load it, then the overnight calculation runs
    Files load in dependency order — products and stores first, then transactions — so a sales file that arrives before its product master never produces thousands of spurious unknown-product errors. Then the night’s work: forecast, status, allocation, DC projection, ordering, transfers, alerts.
  7. 7
    The day is published
    Publication is the last step, and it is all or nothing. Your dashboard moves to the new business date only when the whole night has succeeded. If anything failed you keep yesterday’s complete, coherent numbers rather than a half-built mixture.

A realistic clock

These are the defaults for a European retailer. Every time below is in your local timezone, and every one of them is a setting we agree with you at onboarding rather than something baked into the platform.

Time; What happens
TimeWhat happens
01:30Your export job runs and starts writing files.
by 02:00All files complete, completion file written last. This is your cutoff.
02:00 – 02:45Grace period. Files arriving here still load normally, and you get one reminder rather than a page of them.
03:00Hard cutoff. Whatever has arrived is what tonight runs on. This backstop is always armed, even when a completion file never comes.
when completeThe calculation starts the moment your delivery closes. We do not sit and wait for the cutoff if everything is already in.
03:30Backstop start, if the delivery has not closed by itself.
06:00Published. The dashboard shows the new business date.
06:15Your daily summary: what arrived, what loaded, what needs you.
02

Your own private space

One folder that belongs to you, that no other client can see, list or reach.

When you are onboarded we create a space that belongs only to you. Your files go there and nowhere else. There is no shared folder, no common staging area, and no path by which another client’s credentials can list, read or reach anything of yours.

Write in, but not read back
Your delivery folder accepts files rather than serving them back. Access is scoped to your workspace and to the feeds you have agreed to send, and it reaches nothing else.
Read your own results
A separate results folder works the other way round: you read confirmations, error reports, approved allocations and purchase orders from it. Same space, opposite direction.
Corrections add, they do not overwrite
A correction is a new file, never an edit of an old one, which is what keeps the history honest. Removing data altogether is a separate, deliberate request — the Data security guide sets out how deletion works.

Everything is encrypted on the way to us and while it sits with us, and only your workspace can read your data. Keys, access and retention are covered in Data security.

03

How we know your delivery is complete

The most valuable thing you can send us, and why half a delivery is worse than none.

Here is the failure this section exists to prevent. Your export writes five files. It finishes three and the job dies. Three files sit in your folder, all perfectly valid, all complete in themselves. If we simply calculate on whatever is there, we build tonight’s allocation on two thirds of your stock.

That plan looks exactly like a good plan. Every screen is green and every number is plausible, and it is wrong in a specific, expensive direction: stores whose stock was in the missing file look empty, so the engine ships to them. A plan built on part of your data is worse than no plan, because nothing about it looks wrong.

The completion file

So we do not guess. Once your last data file has finished uploading, you write one small file declaring what you just sent — for each file, its name, its size in bytes, its row count and a checksum. It is written last, and that is the whole trick: its presence is proof that everything before it finished.

{
  "business_date": "2026-08-01",
  "is_complete": true,
  "files": [
    { "feed_type": "inventory_position",
      "key": "inventory_position_20260801.csv.gz",
      "row_count": 498210,  "bytes": 41883902, "sha256": "3f9c1b...", "mode": "full" },
    { "feed_type": "sales_weekly",
      "key": "sales_weekly_20260801.csv.gz",
      "row_count": 1204882, "bytes": 18220417, "sha256": "b71e04...", "mode": "restate" },
    { "feed_type": "product_style_color",
      "key": "product_style_color_20260801.csv",
      "row_count": 51204,   "bytes": 4118220,  "sha256": "9ad2f7...", "mode": "full" }
  ]
}

We verify every number in it against what actually landed. If a checksum or a byte count does not match, we hold that file and name it — we do not discard the rest of your night’s work over one bad upload. If a declared file never arrives, we wait, then close the delivery and tell you exactly which file is missing.

Row count
Required for every transaction feed, with zero tolerance. A truncated export that is still valid CSV is the most common silent corruption in retail feeds, and a declared row count is the only cheap way to catch it.
Checksum and byte count
Proves the bytes we read are the bytes you wrote, so a broken upload is caught before a single row is interpreted.
The is_complete flag
Closes the delivery. If you send a second batch later the same day — a correction drop, say — send a second completion file and we treat it as a continuation, not a contradiction.

If your export tool cannot produce one

Plenty cannot, and that is no reason to leave you unintegrated. Two backstops carry the load, and both are always on:

  • We know what a complete day looks like for you. At onboarding we agree which feeds you send, on what cadence, and how critical each one is. When every expected feed has arrived and nothing new has landed for twenty minutes, the delivery closes on its own.
  • A cutoff timer is always armed. At your hard cutoff the delivery closes with whatever arrived, and the night proceeds on the rules in section 06. A missing completion file can never silently skip a night.
04

What we check before loading anything

Six rungs, in order. A file reaches your dashboard only by passing all of them.

Checks run in order, cheapest first, and all of them run before anything is written into your workspace. A file that fails a rung stops there; it never half-loads while we work out what to do about it.

Rung; What we ask; If the answer is no
RungWhat we askIf the answer is no
1 · ReadableDoes the file open as the type it claims? Is it non-empty, unencrypted, within the size limit, and named in a way we recognise?Held. Nothing is parsed. The message names the problem and the formats we accept.
2 · ColumnsIs there a header row, and does it carry every column the feed contract requires? Order does not matter; capitalisation does not matter.Held, listing exactly which columns are missing. Extra columns you have added are ignored with a note, never an error.
3 · ValuesDates as YYYY-MM-DD. Decimals with a point and no thousands separator. Store and product codes as text, so 00123 stays 00123.Offending rows are rejected individually and reported. Master files, where one bad row corrupts everything downstream, are held whole.
4 · ReferencesDoes every store, DC, style-colour and size in the file exist in your master data?One unknown store holds the file — a whole location silently absent is not a rounding error. A few unknown products is normal churn, so those rows are rejected and reported.
5 · VolumeIs the row count, and the number of distinct stores and products, plausible against the median of your last five accepted deliveries for that feed?More than 40% out in either direction: held, never loaded, until you confirm. Inside that band but still unusual: loaded, flagged amber, with the variance shown against your own trend.
6 · ReconciledAfter loading: do the rows and totals in your workspace equal the rows and totals we parsed out of your file?A mismatch on a critical feed stops publication. Numbers we cannot prove match what you sent do not reach your dashboard.

The column contract for each feed — names, types, the columns that uniquely identify a row, and what each one means — is in File specifications.

05

What happens when something is wrong

Three outcomes, one message, and a report you can hand to whoever owns the export.

There are only three things that can happen to a file. There is no fourth, silent one.

Outcome; When; What you get
OutcomeWhenWhat you get
LoadedEvery row placed, or only cosmetic notes: extra columns, a currency spelled differently, an encoding we transcoded for you.A confirmation with the row count. Notes appear on the status page as information, not as an alarm.
Loaded in partSome rows failed a value or reference check, and the feed allows partial loading — sales, positions, orders.The good rows are in and working. You get a report of every rejected row with its original line number, the failing column, the value and the reason.
HeldThe file is unreadable, missing required columns, short on coverage, or more rows failed than that feed's error budget allows. Also any master file with a bad row.Nothing loaded, so nothing is half-changed. We tell you promptly, with the reason and the fix. Re-send the corrected file and it loads normally.

What an error actually looks like

Messages are written for the person who owns the export, not for us:

Row 4,182 - column store_code: "0115" is not one of your 42 active stores.
  Closest match: "115" - Riverside Mall. 37 other rows carry the same value.
  Fix: correct the store code in your export, or add store 0115 under
       Configuration > Stores if it is a new location.

Row errors are never sent one at a time. They are grouped by column and reason, counted, and the five largest groups are shown with one concrete example each; the rest go into the report. A file with 1.2 million broken rows produces exactly one message — anything else trains you to filter us out.

06

If your data does not arrive

We do not invent numbers. Here is what you see instead, and who we tell.

The tempting thing to do with missing stock data is to carry yesterday’s forward and keep the dashboard moving. We do not do that, and the reason is worth stating precisely.

Yesterday’s on-hand overstates today’s wherever a style sold, and understates it wherever a delivery landed. The overstatement is largest on your fastest-selling style-colours — exactly the ones allocation exists to push out. So a carried-forward plan systematically under-ships your winners and over-ships the slow movers. That is not a slightly noisier plan; it is a plan biased in the wrong direction, and it looks identical on screen to a good one.

What you see instead

Your dashboard keeps serving yesterday’s published figures, unchanged and internally consistent, behind a banner that says why:

Serving: 31 Jul 2026    [!] NOT REFRESHED
inventory_position for 1 Aug has not been received (due 02:00, your time).
Today's allocation and order plans were not refreshed.

The banner is the point. A planner who knows they are looking at yesterday can decide for themselves whether to act. A planner looking at fresh-seeming numbers built on stale stock cannot.

Not every feed is equally critical

Feed; If it is missing tonight; How long we carry it
FeedIf it is missing tonightHow long we carry it
Stock on handThe day is held. Yesterday keeps serving, behind the banner above.Never carried
SalesThe night runs. Completed weeks are unaffected; only the current partial week is short, and every calculation already treats it as partial.6 days
Products, stores, vendorsThe night runs on yesterday’s master data, which is all but identical. New products and stores surface as reference errors on the transaction feeds — a faster detector than the calendar.14 days
Purchase ordersThe night runs. Ordering output is marked as on-order as of an earlier date, so a buyer can see what they are working with.3 days
Prices, promotions, forecast workbooksThe night runs. The forecast module shows which vintage it is using.21 – 30 days

Past the carry limit the old data stops being a reasonable stand-in — a week with no sales signal genuinely breaks the forecast — and that feed is treated like the stock file.

Who we tell, and when

  1. 1
    It shows up immediately, where the work happens
    The feed turns red on the status page and your named data contact is notified. The message says what feed, what date, what we did, and what to do about it.
  2. 2
    It widens as it persists
    A feed that stays missing is escalated through the contacts you nominate at onboarding — planning, IT, and your account lead — and the on-screen banner is widened from administrators to every user in your workspace, because the people making decisions on those screens need to know.
  3. 3
    Prolonged silence gets a person, not a no-reply address
    Past the point where email is doing any good, someone from our side picks up the phone. Outputs that depend on a long-missing feed stop generating rather than generating from stale data, and resuming is an explicit, recorded action by one of your administrators.
  4. 4
    And we tell you when it clears
    The first good delivery resets the count and sends an all-clear to everyone who was told. The exact contacts and thresholds at each stage are agreed with you at onboarding and set out in your contract.
07

Sending the same file twice, and corrections

Re-sending is safe by design. Restating a past week is a normal operation.

Re-sending is safe. If your export job retries, if someone runs it twice, if a network blip leaves you unsure whether a file arrived — send it again. The platform is built on the assumption that this happens.

Situation; What we do
SituationWhat we do
The identical file, twiceRecognised by its content and recorded as a duplicate. Loaded once. Nothing is doubled.
A late first delivery — data for a past date you never sentLoads into the correct week, not into today. We then recompute the affected weeks, so KPIs and forecast accuracy for that period are right.
A correction: sales_weekly_20260725_r1.csv.gzReplaces that week completely, for the weeks and stores present in your file. Nothing outside that scope is touched.
A second correction: _r2Same behaviour. Revision numbers must increase, so the newest correction always wins.
An older correction arriving after a newer oneRecorded and not applied. Out-of-order delivery cannot roll your data backwards.
Thirty days of history in one goRecognised as a backfill: loaded together, then the calculation runs once at the end rather than thirty times.

Why a correction replaces rather than merges

A correction can remove rows as well as change them: a cancelled order line, a returned sale, a store that turns out not to have traded. Merging can add and update, but it can never express a deletion, so a merged correction leaves the removed row sitting there forever, quietly wrong. Instead we delete and reinsert exactly the scope in your file — the specific weeks and the specific stores it contains — in one operation that either completes or does not happen at all. The rest of that week, including every store absent from your correction, is untouched.

One rule underpins all of this: the date in a filename is the date the data is for, not the date you sent it. sales_weekly_20260725_r1.csv.gz delivered on 1 August is a correction to the week of 25 July, and it is handled as one no matter when it lands.

08

What you get back

A confirmation per file, a status page, error reports, and plans flowing back to your ERP.

The flow is not one-way. For every file you send there is something you can read back, and at the end of the day the decisions your planners make flow home to your systems.

A confirmation for every file
Within minutes of a file being processed, a small confirmation appears in your results folder: what we received, when, how many rows we accepted, how many we rejected, and where the reject report is. Your ERP can poll for these instead of guessing from folder state.
The status page
In the app: one tile per feed with its colour, the time it arrived, its row count against your own seven-day trend, and one plain sentence on what a red light costs you today. Planners can see it too — the person who notices that a plan looks wrong should be able to find out why without raising a ticket.
Error reports
A file per delivery listing every rejected row with its original line number, the failing column, the value we saw and the reason. It is written to be handed straight to whoever owns the export, with no translation from us.
The file we received
Download the exact bytes you sent, from the permanent record, straight from the status page. This ends every version of the conversation that begins “we definitely sent that”.
A daily summary
One email each morning: what arrived, what loaded, what was rejected, what needs a decision. Quiet when there is nothing to say — with one exception that cannot wait, a critical feed still missing before tonight’s run.
Your approved plans, back to your ERP
When a planner approves an allocation or a buyer approves an order, the file is written to your results folder in the same shape, and on the same rules, that we ask of you.
<your-workspace>/results/allocations/business_date=2026-08-01/allocation_20260801_001.csv
<your-workspace>/results/purchase-orders/business_date=2026-08-01/po_20260801_001.csv
<your-workspace>/results/transfers/business_date=2026-08-01/transfer_20260801_001.csv
<your-workspace>/results/rejects/sales_weekly/sales_weekly_20260801_rejects.csv
<your-workspace>/results/allocations/business_date=2026-08-01/_manifest_1.json   <- written last
09

Keeping it running

Nightly proof that what you sent equals what we loaded — and nothing to do by hand.

We prove that what you sent equals what we loaded

Every night, after loading, we count back. Not a spot check — a full comparison, on every feed, whether or not anything looks wrong.

Level; What it compares; What it catches
LevelWhat it comparesWhat it catches
TransportThe bytes and checksum we read against the bytes and checksum you wrote.Anything lost or altered between your upload finishing and our reading it.
Loaded vs sentRows accepted against rows now in your workspace, and totals — units on hand, units sold, value — to within a rounding tolerance.Our own mistakes. A merge that collapsed 500 store rows into one, or a filter that quietly dropped a location type, shows up the same night.

The output is not only exceptions. On a clean day you get the positive proof, which is what makes the exception messages worth reading:

Reconciled - 1 Aug 2026
  4,182,903 units on hand matched to your file
  498 stores, 48,112 SKUs, 0 discrepancies

Where we cannot prove that the numbers we loaded match the ones you sent, we say so rather than presenting them as verified. Publishing figures we cannot stand behind is worse than publishing nothing, because you would place a real order against them and neither of us would find out for weeks.

Nobody at your end has to do anything

Once the integration is set up, the entire path on this page is unattended. The export runs, the files land, we detect them, check them, load them, calculate, publish, and write your approved plans back. No one logs in to start it. No one watches a folder. No one clicks a button at three in the morning.

We ask for a human only when the decision is genuinely yours: a held file that needs a corrected export, a restatement older than your window, a feed that has not arrived, or a plan a planner needs to approve. Everything else is our problem, and you will hear it from us before you notice it on a screen.

Nightly checks catch nightly problems. Slow drift is a different shape — a reject rate creeping up over six weeks passes every individual night — so feed quality is also reviewed over a longer window with your team, on a cadence agreed at onboarding.

Want to see this running on your feeds?

We can walk your ERP team through the file contract, the completion signal and the morning after, using one night of your real data.

Book a walkthrough