Inez Hogarth, Founder and Managing Director
The Shape of a Pipeline Over Time
Data pipelines tend to grow in place.
A small number of sources, a handful of transformations, and a reporting layer is usually enough to get started. Over time, more data is added, more use cases emerge, and more people begin to rely on the outputs. What was once easy to follow becomes harder to reason about, and changes take longer to make.
At that point, attention shifts. The focus moves away from building new capability and towards making sure the existing system behaves as expected. Questions around data quality, timeliness, and ownership become more visible. Confidence in outputs starts to vary depending on who is using them and for what purpose.
This pattern is not unusual. It reflects how data pipelines evolve in practice.
In practical terms, a data pipeline is a system that moves and transforms data from source systems into formats that can be used for reporting, analytics, or machine learning. As that system grows, the way it is structured becomes more important than the tools used to build it.
Where Complexity Starts to Show
As pipelines expand, the number of dependencies increases. Data is pulled from multiple systems, transformed across different stages, and made available to a growing set of consumers. Those dependencies rarely sit in isolation. They begin to overlap.
One of the more subtle challenges sits within the transformation layer. This is where business logic is applied, often across multiple steps, and where small inconsistencies can have a disproportionate impact. When logic is spread across different tools or written in slightly different ways, it becomes difficult to trace how a particular output has been derived.
Responsibility can also become less clear. Different parts of the pipeline are often owned by different roles, and while each component may function as expected, the overall reliability of the system does not sit clearly with any one person or team. Issues that span multiple stages take longer to identify and resolve.
Visibility is another pressure point. Pipelines do not always fail in obvious ways. Data can arrive later than expected, be partially processed, or reflect changes in upstream systems. Without a clear view of how data flows through the pipeline, these issues tend to surface at the point of use rather than at the point of failure.
A common scenario is an overnight pipeline feeding executive reporting. An upstream change or delay results in incomplete data reaching dashboards the following morning. The issue is only discovered when the data is reviewed, by which point decisions may already be based on it.
What Holds Up as Pipelines Scale
Pipelines that remain workable over time tend to have a different feel to them.
There is a clearer separation between stages, making it easier to understand how data moves from raw inputs to curated outputs. Patterns are applied consistently, so that new additions follow an established approach rather than introducing variation. Ownership is more explicit, particularly in relation to reliability and data quality. Monitoring is treated as part of the system, providing a clearer view of how data behaves as it moves through each stage.
These qualities are not tied to a specific toolset. They tend to emerge from deliberate design decisions and a willingness to introduce structure as complexity grows. In many cases, this forms part of broader investment in data infrastructure, where the focus is on creating foundations that can support analytics and AI at scale.
In practice, the challenge is often less about choosing the right tools and more about introducing enough structure to make existing pipelines understandable again.
Choosing Between Batch and Streaming
The choice between batch and streaming approaches is usually driven by the nature of the problem rather than a desire to adopt a particular technology.
Batch processing remains appropriate in many environments, particularly where data does not need to be acted on immediately and workloads are predictable. Streaming becomes relevant where the timing of data has a direct impact on decisions or outcomes.
In both cases, the requirement is the same. The pipeline needs to behave predictably, even as data volumes and usage increase.
Pipelines, AI, and MLOps
As organisations invest more heavily in AI, the role of the data pipeline becomes more pronounced. Training data needs to be consistent with production data, and the processes that prepare it need to be repeatable. Small discrepancies in how data is handled can lead to unexpected behaviour in models.
This is where data engineering begins to overlap with MLOps. The concern is not just about building models, but ensuring that the data feeding those models is reliable and consistent across environments.
Industry research consistently highlights the impact of this challenge. A significant proportion of AI initiatives fail to progress beyond pilot stage due to data quality and operational issues.
Working With Pipelines That Already Exist
In most situations, pipelines are already in place. The task is to make them more manageable without disrupting what they currently support.
This often involves introducing clearer structure, simplifying areas where complexity has built up, and improving visibility into how the system behaves. Progress tends to be incremental, with each change making the next one easier to implement.
This type of work is closely tied to broader data strategy.
A Practical Delivery Perspective
At Data Understood, work typically begins with understanding how a pipeline has evolved and where pressure is starting to build. The aim is not to replace what exists, but to introduce patterns that allow it to scale more comfortably.
That might involve reshaping parts of the transformation layer, clarifying ownership, or putting in place monitoring that provides a more immediate view of how data is moving through the system. Over time, those changes reduce the effort required to maintain the pipeline and increase confidence in the outputs it produces.
This work often connects directly to analytics and data product development.
Final Thoughts
Data pipelines are often treated as a technical concern, but their impact is felt more widely. When they are stable and predictable, teams can move quickly and with confidence. When they are not, the friction is experienced across the organisation.
Introducing structure, ownership, and visibility is not always urgent, but it tends to become necessary as the demands on the system continue to grow.
