Process historians do not store every value they receive. They store the values needed to reconstruct the signal within a tolerance you specified, probably a long time ago. This is why decades of history fit on modest hardware, and it is the most commonly overlooked fact when that history is handed to a data science team.

Two filters, not one

Exception happens at the interface, before the data reaches the archive. If a new reading is within the exception deviation of the last one reported, it is not sent at all. Values filtered here were never stored and cannot be recovered.

Compression happens at the archive. It applies a swinging-door algorithm that keeps the values needed to reconstruct the trend within the compression deviation and discards the rest.

The practical consequence: your archive contains an irregular series of points chosen because they were interesting, separated by intervals that carry information of their own. A long gap means the signal was stable. It does not mean the instrument stopped.

Where this goes wrong

• Naive resampling. Pulling values at a fixed interval without interpolating correctly produces a series that reflects when the archive happened to store a point. You end up modelling the compression algorithm rather than the equipment.
• Averaging raw stored values. An unweighted mean over compressed data overweights periods of rapid change, because that is when more points were kept. Time-weighted averaging is almost always what was intended.
• Reading stability into silence. A flat line in a chart may be a genuinely stable process, a failed instrument reporting its last value, or an interface that stopped without buffering. These are very different situations and they look identical unless quality codes are carried through.
• Discarding quality codes. Most export paths drop them. A digital state indicating a bad reading becomes a number, and the number joins the training set.

Why the settings are often wrong anyway

Compression settings were frequently chosen when storage was genuinely expensive, and have been inherited unchanged through two or three hardware generations. On modern infrastructure many are far more aggressive than necessary, discarding resolution that is now cheap to keep.

The opposite also occurs: compression effectively disabled on noisy tags, filling the archive with values that carry no information and slowing every query made against them.

Either way, a migration or an analytics programme is the right moment to review these deliberately, tag class by tag class, with the trade-off written down.

What good practice looks like

• Interpolate onto even intervals explicitly, using the historian’s own interpolation, so windowed aggregation means something
• Carry quality codes through the pipeline rather than dropping them at the first export
• Use time-weighted aggregation unless there is a specific reason not to
• Keep a raw landing layer at full fidelity so the first modelling decision is not irreversible
• Review exception and compression against what the instrument actually produces, not against what was set years ago

None of this is exotic. It is simply the difference between treating operational data as what it is and treating it as rows in a database. More on building pipelines that respect it.