Why don’t these numbers match?

“Why don’t these numbers match?”
That question usually signals the start of a data engineering project.
Sales, marketing, and finance all have dashboards, but they don’t align.
New questions take too long to answer.
Confidence in data varies across teams.

So the business decides to invest in a proper data platform.

Before pipelines or tools are chosen, a short needs assessment helps answer:
• Which metrics drive decisions?
• Who uses them, and how often?
• How fresh does the data need to be?
• Where is consistency critical?

This clarity determines both team structure and automation strategy.

Once needs are clear, the data work typically includes activities such as:
• Discover and profile data sources
• Define schemas and metric definitions
• Build ingestion and transformations
• Validate and monitor data quality

Now it’s possible to decide what AI can accelerate, and where human ownership is required.

𝘋𝘢𝘵𝘢 𝘱𝘳𝘰𝘧𝘪𝘭𝘪𝘯𝘨 & 𝘥𝘪𝘴𝘤𝘰𝘷𝘦𝘳𝘺

AI: analyse datasets, surface patterns, and quality issues
(Tools such as LLMs, Monte Carlo, Bigeye)
Human role: Data Engineer / Data Engineering Lead validates findings and decides what matters

𝘚𝘤𝘩𝘦𝘮𝘢 & 𝘥𝘢𝘵𝘢 𝘮𝘰𝘥𝘦𝘭 𝘥𝘦𝘴𝘪𝘨𝘯

AI: propose schemas and relationships
(Tools such as LLMs, dbt with AI-assisted models)
Human role: Analytics / BI Specialist defines business meaning and metric logic

𝘋𝘢𝘵𝘢 𝘵𝘳𝘢𝘯𝘴𝘧𝘰𝘳𝘮𝘢𝘵𝘪𝘰𝘯𝘴

AI: generate and refactor SQL
(Tools such as dbt + AI, Cursor, GitHub Copilot)
Human role: AI-Assisted Data Engineer reviews, tests, and approves transformations

𝘔𝘰𝘯𝘪𝘵𝘰𝘳𝘪𝘯𝘨 & 𝘥𝘢𝘵𝘢 𝘲𝘶𝘢𝘭𝘪𝘵𝘺

AI: detect anomalies, data drift, and pipeline issues
(Tools such as Monte Carlo, Databand)
Human role: Data Engineering Lead / SME defines what “drift” actually means for the business, assesses impact on decisions, and prioritises fixes

𝘋𝘰𝘤𝘶𝘮𝘦𝘯𝘵𝘢𝘵𝘪𝘰𝘯 & 𝘨𝘰𝘷𝘦𝘳𝘯𝘢𝘯𝘤𝘦

AI: generate and maintain technical documentation
(Tools such as LLMs, dbt docs)
Human role: Data Governance / Compliance role ensures consistency, security, and auditability

Starting with needs, not tools, leads to:
• Clear ownership
• Faster delivery
• Effective use of AI
• Data teams the business can rely on

This is the approach we apply at iForce Connect when designing data engineering teams and delivery models. Let's connect.

AI speeds up data work.
Humans define meaning and take responsibility.
The key is deciding who does what before the project starts.