Where does your engineering budget really go?
In March, Fivetran published a study based on a survey of 500 senior data and technology leaders at companies with more than 5,000 employees. One number caught our attention: 53% of engineering capacity goes to maintaining and troubleshooting pipelines. And nearly 97% of those leaders say pipeline failures have slowed their analytics or AI programs.
Take it with a grain of salt. Fivetran sells managed pipelines, and its conclusion unsurprisingly points that way. It’s also a survey: leaders estimated, nobody timed anything. dbt Labs doesn’t give a percentage in its report published in April, but a majority of its respondents say they spend most of their time maintaining or organizing datasets.
Even as a rough order of magnitude, half of a skilled team is a lot. What is that half doing? And does it really have to?
On an integration project for a high-volume platform, the business rule fit in a few lines, but connecting the two ERPs took three weeks: formats, authentication, retries, pagination. Along the same lines, a pipeline ran for several days producing wrong data without raising a single error. The sender was emitting JSON arrays, while the receiver expected JSON Lines. Neither tool was at fault, and nobody was watching the junction.
Part of it is the job
Not everything in that 53% is waste. A supplier changes its API, a source system renames a field, a regulation changes a retention period: you have to follow. That’s what a data team is for, and no architecture will make it go away.
The rest gets talked about a lot less outside the team. Retries when a call fails, timeouts set once and never revisited, format conversions between two internal tools, credentials passed around by hand, error handling that swallows a failure in one place and lets it resurface three stages later, and all those adapters written so that what one tool produces gets accepted by the next.
Teams call it glue. It’s written by whoever is closest to the ticket, often at the end of a sprint. It’s rarely reviewed, because it looks secondary. It’s tested even less, because testing it means running two systems at once. And yet plenty of incidents start right there: a failure between two components belongs to neither of them, and nobody is really watching it.
Where the glue comes from
Nobody decided to write all this glue. It’s there because our tools share no common idea of what a connection is.
Take a typical data stack: an ingestion tool, an orchestrator, a transformation tool, a warehouse, often a message broker, a quality tool, a catalog. Each one does its job very well. But none of them shares with the others a definition of what “passing data to the next one” means. So every junction needs a contract, and someone writes that contract in code, with no types, no verification, and often no record of it.
That’s also why the usual levers don’t work well. Hiring, adding process, or consolidating tools just means writing the same glue with more people, more reviews, or in a different format. And the coding assistant? It writes it faster. Except typing the code was never the expensive part. The expensive part is knowing whether the junction holds.
Same problem in software delivery
If you’re on the software side rather than data, you already have a cousin of this problem at home. Run this in your main repository:
git log --oneline | grep -ciE "fix.*\bci\b"
That number is the commits made to repair the build pipeline, not the product. Each one is a well-known round trip: push, wait a few minutes, it breaks, fix, push again. Because nobody can really run a CI pipeline exactly the same way on their own machine.
CI configuration started as a list of commands. Then came conditions, loops, templates, variables… It turned into a programming language, but with no types, no local execution, and no way to know a change is wrong before paying to run it.
What can actually move the number
There aren’t a dozen ways around it: the connection between two components has to become something the system understands, instead of something the team reinvents at every junction.
In practice, handing data from one stage to the next needs a declared shape, and something needs to check that both sides agree before anything runs (rather than at three in the morning, once they’ve drifted apart). Retries, error propagation, and flow control need to be handled by the layer underneath, not by whoever wrote a given junction. And a stage that has to move to another machine should be able to move without being rewritten.
We’re not going to pretend to be neutral: this is exactly what we build at Mélodium. Like Fivetran, we have a stake in this diagnosis. That’s why what follows is a measurement you can make without us, and without them.
Measure your own number in an afternoon
Take your most important pipeline or integration, and the last twenty changes merged into it (or last quarter’s tickets, if that’s easier). Ask two people to sort them independently, then compare. Wherever they disagree, there’s already something to learn.
Three categories:
- What the business asked for. A new source, a new metric, a new rule, a request from outside the team.
- What the outside world imposed. A supplier changed its API, a source system changed its schema, a regulation changed.
- What your own tools imposed. Two of your tools didn’t get along, a retry or timeout was readjusted, a format was converted between two internal components, a credential was handled by hand, or a failure happened between two stages rather than inside one.
The first category is investment. The second is the price of existing in the real world, and you manage it with suppliers and contracts. The third is the glue we’re talking about here. If it weighs more than the first, your architecture is taking up more of your team than your roadmap is.
To put it in money: share of the third category × number of engineers × fully loaded annual cost. An example, with numbers made up for the occasion: with eight engineers at a fully loaded cost of €90,000 and a third category at 35%, that’s €252,000 a year spent on the junctions between your own tools. Plug in yours.
Three decisions, nothing to buy
Whatever your result, there are three things to do that don’t need any vendor.
First, measure again. Every quarter, same classification, with the third category shown in the same review as cloud spend. Cloud bills get tracked down to the cent, while this number almost never gets measured at all.
Next, count junctions in every tooling decision. Every product added to the stack creates connections someone will have to maintain. “How many more junctions, and who looks after them?” belongs in every architecture review and every purchase, right next to the license price.
Finally, go after the worst junction rather than the whole platform. Take the one that generated the most third-category changes and the most incidents, and treat it as a real project, with an owner and a budget, not as background noise you put up with.
53% is Fivetran’s estimate. For you it might be less, or a lot more. If you run the exercise, we’d really like to know how it turns out.
Sources
- Fivetran, The Enterprise Data Infrastructure Benchmark Report 2026, published March 26, 2026. Survey of 500 senior data and technology leaders at enterprises with more than 5,000 employees, conducted in Q4 2025. Press release , report .
- dbt Labs, 2026 State of Analytics Engineering Report, published April 10, 2026. Report .