The pilot impresses the executive team. The project gets funded. Then it sits in staging for eight months. Here is why data infrastructure pilots stall, and what production-first engagement looks like instead.
There is a pattern we see repeatedly across healthcare, agriculture, and manufacturing organisations. The IT team identifies a data infrastructure problem. A vendor or internal team runs a pilot. The pilot looks good. It connects to the right systems, produces a dashboard, and impresses the stakeholders who approve funding. Then it stalls. The pilot sits in a staging environment for months. Eventually it is quietly shelved, and the organisation goes back to manual processes. The problem was not the technology. The problem was that the pilot was never designed to become a production system.
Key takeaways
A data infrastructure pilot typically proves that data can be extracted from source systems and presented in a useful form. This is valuable. It answers the question: can we technically do this?
What it does not prove is whether the system can run reliably in production. Production means the pipeline runs every day without intervention. It means errors are detected, logged, and recoverable. It means PHI is handled according to compliance requirements, not just in the main data path but in error handlers, logs, and alerting. It means someone on your team can operate it without calling the people who built it.
The gap between a pilot and a production system is not a small engineering task. It is typically 60 to 70 percent of the total effort. A pilot that took four weeks to build will take another eight to twelve weeks to harden for production. Organisations that budget for the pilot but not the hardening end up with an impressive demo and no production system.
A pilot dashboard showing data flowing from an EHR system into a warehouse creates confidence that the project is nearly done. Leadership sees the output and assumes the remaining work is polish. In reality, the remaining work is the hard part.
The pilot ran against a sandbox with clean data. Production data has nulls, duplicates, schema variations, and edge cases the sandbox did not contain. The pilot ran for a few hours. Production runs continuously, which means handling rate limits, token expiry, network interruptions, and upstream system maintenance windows. The pilot was operated by the engineer who built it. Production will be operated by someone who did not.
When leadership approves full funding based on a pilot, they are often approving a timeline and budget that reflects the pilot's complexity, not production's. The project starts. The team discovers the production edge cases. The timeline extends. Confidence erodes. The project loses momentum.
A stalled pilot does not just waste the pilot budget. It creates organisational scar tissue. The finance team that was promised automated reporting goes back to spreadsheets. The clinical operations team that saw the dashboard demo goes back to four-day-old reports. The IT team that championed the project loses credibility internally.
The next time someone proposes a data infrastructure investment, the organisation remembers the pilot that never shipped. Approval is harder. Scrutiny is higher. The delay between the original pilot and the eventual production system can be 12 to 24 months, during which the operational cost of broken data infrastructure continues accumulating.
In a hospital network, that accumulation is not abstract. It is analyst hours spent on manual reconciliation, decisions made on stale data, audit risk from uncontrolled PHI flows, and the compounding trust deficit between departments working from different numbers.
A production-first approach does not skip the discovery and assessment phase. It reframes what the deliverable is. Instead of: build a pilot, then figure out production, the framing is: scope and deliver a production system from day one.
This means the assessment phase produces a production architecture document, not a pilot plan. The build phase includes error handling, monitoring, security controls, and documentation alongside the core pipeline, not as a later phase. The deliverable at the end of the engagement is a system your team operates, with runbooks, training, and a support window.
Production-first engagements are not more expensive than pilot-then-production. They are typically less expensive because they avoid the rework, scope creep, and organisational delay that comes from building something twice: once for the demo, once for real.
Ask what the deliverable is at the end of the engagement. If it is a dashboard and a report, that is a pilot. If it is running infrastructure with monitoring, alerting, documentation, and a handover plan, that is a production engagement.
Ask about error handling. If the vendor has not thought about what happens when the EHR API rate-limits the pipeline, when a source system schema changes, or when an OAuth token expires mid-job, they are scoping for the happy path. Production is mostly about the unhappy paths.
Ask about handover. Who operates the system after the engagement ends? If the answer is vague, the vendor may be scoping for a managed service relationship rather than a build-and-handover. There is nothing wrong with managed services if that is what you want, but you should know which model you are buying.
In summary
The pilot-to-production gap is not a technical problem. It is a scoping problem. Organisations that scope for pilots get pilots. Organisations that scope for production systems get production systems. The difference is not in the technology. It is in whether the engagement is designed around a demo or around a working system that your team will operate. If you have been through a pilot that stalled, the path forward is not another pilot. It is an engagement scoped for production from day one.
Talk to us. We will scope an engagement before any work begins.