From Historian to Lakehouse: What Operators Are Actually Getting From OT Data in Databricks
Every industrial operator already has the data. A mid-size utility or process plant typically holds a decade or more of compressed history in AVEVA PI System or Aspen InfoPlus.21 — tens of thousands of tags, archived by exception, reliable to the second, and read by a handful of engineers through trend displays. The data was never the problem.
The problem is reach. That history sits where it cannot be joined to maintenance records, lab results, weather, or market prices, and it cannot be queried at the scale where statistical methods start to earn their keep. Ask most plants for five years of vibration data next to the work orders that followed, and you get a CSV export and a week of somebody’s life.
That constraint has quietly come apart over the last eighteen months, and the operators who noticed are publishing numbers that are difficult to argue with.
What actually changed
At AVEVA World 2025, AVEVA and Databricks announced a native integration between AVEVA CONNECT and the Databricks platform, and the mechanism matters more than the announcement did. It is built on Delta Sharing — an open protocol that exposes operational data to the lakehouse without copying it into yet another silo. Data is shared, governed, and read in place.
On the Databricks side, the ingestion and transformation stack consolidated under Lakeflow: Lakeflow Connect for managed ingestion, and Lakeflow pipelines (the product formerly known as Delta Live Tables) for declarative transformation with data-quality expectations built into the pipeline definition rather than bolted on afterwards. Everything lands in Delta Lake and is governed by Unity Catalog.
The practical consequence: the integration work that used to justify a six-figure custom connector and a permanent maintenance burden is now mostly configuration and modelling. The engineering effort moves to where it always belonged — the data model.
The architecture that works
The pattern that survives contact with a real plant is a medallion architecture tuned for time series. Bronze lands raw tag history as-is, full fidelity and replayable, because the first model you build will not be the one you keep. Silver is where the value is created: asset context joined to tags, engineering units normalised, quality codes preserved rather than discarded, and irregular exception-based samples interpolated onto even intervals so that windowed aggregation means something. Gold carries the engineering features the business actually asks for — heat rate, availability, OEE, degradation curves — modelled per asset class so a question asked of one turbine can be asked of all of them.
Two details separate pipelines that scale from pipelines that get rebuilt in year two. The first is honest handling of late and out-of-order data: historians backfill, operators correct bad values, and communications links drop. Watermarking and replay are not optional. The second is physical layout — clustering on asset and timestamp is the difference between a five-year trend that returns in seconds and one that scans the whole table.
Context is the hard part, not volume
Terabytes of time series are unremarkable; Databricks will absorb them without complaint. What defeats most projects is that a tag name is not an asset. PLANT2.U3.BRG.VIB.01 means something to the engineer who configured it and nothing to a model.
The PI Asset Framework hierarchy — elements, templates, attributes, and the relationships between them — is the institutional knowledge that makes the tag interpretable. Carrying that structure into Unity Catalog, so that an analyst can ask for “bearing vibration across all Unit 3 class assets” without knowing a single tag name, is the step that converts a data migration into a data product. Skip it and you have built an expensive archive.
What it is worth
AVEVA has published results from operators running this pattern. SQM, a specialty chemicals producer, reports a 1% increase in nitrate yield, a 10–15% reduction in process variability, and 90% accuracy in its predictive models after unifying historian and lab data and exposing it to Unity Catalog. EDP Renewables replaced twice-daily manual transfers with a hybrid architecture streaming PI System data into the lakehouse, moving to 50 times more frequent updates — every 15 minutes instead of every 12 hours — with faster anomaly response and more accurate energy price adjustment.
Treat vendor-reported figures with the scepticism they deserve. The direction of travel, however, is consistent: a 1% yield improvement on a continuous process is not a dashboard, it is margin.
Where these projects go wrong
- Lifting tags without the asset model. Fast to demo, impossible to scale past one unit.
- Treating OT data as IT data. Exception-based compression, deadbands, and quality codes are engineering decisions. Resample naively and you will model the compression algorithm instead of the equipment.
- Governance as an afterthought. Operational data carries safety, regulatory, and commercial sensitivity. Access control and lineage belong in the design, not the retrofit.
- No path back to the control room. An insight that never reaches an operator is a research project.
Doing this properly
This work sits at an awkward intersection. It needs people who understand PI Asset Framework hierarchies and historian compression behaviour, and the same people need to be fluent in Delta Lake, Unity Catalog governance, and production pipeline design. Those skill sets rarely live in the same team — OT engineers do not usually write Spark, and data engineers do not usually know why a deadband exists.
That intersection is what LAMR Consulting does. We work with power generation, transmission and distribution utilities, and industrial operators on exactly this path: AVEVA PI System and Aspen InfoPlus.21 estates, asset contextualisation, and integration into Databricks that is built to be operated rather than demonstrated. We have delivered the condition-based monitoring, rapid anomaly detection, and asset performance management programmes that this architecture is meant to support.
If you have a historian full of history and no practical way to ask it a question, that is a solvable problem — and a well-understood one. Talk to us about what it would take at your sites.