Spark pipeline – Definity raises $12M Series A to move monitoring inside Spark runs—so teams can stop bad data before it reaches agentic AI systems.
For data teams, a pipeline failure isn’t just a broken job—it can become the “bad input” that derails downstream AI decisions.
Definity. a Chicago-based data pipeline operations startup. is betting that the next leap in reliability won’t come from watching systems from the outside.. Instead. it embeds agents inside Spark and DBT execution so they can detect and prevent problems while pipelines are still running—before stale or incorrect data propagates into agentic AI systems that depend on clean inputs.
The company announced a $12 million Series A financing led by GreatPoint Ventures. with participation from Dynatrace and existing investors StageOne Ventures and Hyde Park Venture Partners.. Definity’s premise is straightforward: if you discover a pipeline issue after the job has completed. the damage—wasted compute. delayed workflows. and downstream contamination—may already be done.. In a world where analytics pipelines increasingly feed AI-driven processes, “after-the-fact” observability can be too slow to protect business outcomes.
That urgency is driving a shift in how enterprises think about pipeline operations.. Traditional monitoring tools typically collect metrics after a job finishes, then alert engineers to what went wrong.. Those signals can be useful for audits and postmortems. but they struggle to offer the kind of prevention that agentic workloads increasingly require.. Definity’s approach aims to close that gap by moving the intelligence point from monitoring dashboards to the execution layer itself.
Technically, the mechanism is built around inline instrumentation.. Definity installs a JVM agent directly into the pipeline execution layer via a single line of code. running below the platform layer and pulling execution data as Spark runs.. Rather than relying only on aggregate metrics. the agent captures execution behavior and resource pressure in real time—signals such as memory pressure. data skew. shuffle patterns. and infrastructure utilization.. It can also infer lineage between pipelines and tables dynamically, without requiring a predefined data catalog.
More importantly, Definity positions the agent as an intervention layer, not just a camera.. During a run. it can modify resource allocation mid-flight. stop a job before bad data spreads. or preempt a pipeline based on upstream conditions.. The logic is designed for the moments that matter most: where a pipeline is about to waste time or generate incorrect outputs that downstream systems will later treat as reliable.
A useful way to frame the difference is “real time where it counts.” Definity’s monitoring is real time for detection and prevention while jobs are executing.. Root-cause analysis and optimization recommendations. by contrast. can run on demand when an engineer queries the assistant—because the system already has the execution context assembled.. That design choice tries to balance fast safeguards with a workflow that still fits how engineers troubleshoot today.
For enterprises with strict data residency requirements, the company says the agent sends only metadata externally and supports fully on-premises deployment, reducing the risk that security constraints become a blocker.
One of the clearest motivations for this kind of in-execution intelligence comes from cost and capacity constraints. not only from failure avoidance.. Nexxen. an ad tech platform running large-scale Spark workloads on premises. reportedly used Definity to reduce the accumulating costs of inefficiency in an environment without elastic cloud capacity.. According to Definity and Nexxen’s data engineering leadership. the team identified 33% of optimization opportunities in the first week and reduced engineering effort on troubleshooting and optimization by 70%.. They also describe freeing infrastructure capacity to support growth without additional hardware.
That human impact matters because it changes what time looks like for data teams.. When monitoring is reactive. engineers spend hours tracing symptoms across distributed jobs. often under pressure to restore service for business-critical workloads.. Proactive. continuous optimization shifts the effort from constant firefighting to prioritizing roadmap work—an operational difference that can be hard to quantify until it’s felt.
The broader takeaway for enterprise data organizations is that pipeline ops is increasingly an AI infrastructure problem.. Pipelines that used to support analytics are now part of the supply chain for agentic AI systems—where “silent” failures. latency issues. and stale outputs can directly affect what models do next.. Under those conditions, reliability becomes not just a technical metric but a delivery constraint for AI initiatives.
In that sense, Definity’s bet is less about a single product feature and more about architectural timing.. As pipelines become more autonomous and AI-dependent, the window for intervention shrinks.. Solutions that can act during execution—rather than merely report after the fact—may increasingly determine whether teams treat reliability as an engineering discipline or as an ever-present tax.
If the market moves toward “in-execution intelligence” as a standard. expect more startups and platforms to compete on how quickly they can detect failure modes and how safely they can prevent bad outputs—while keeping overhead low and integration friction manageable.. For teams evaluating this category. the key question won’t be whether monitoring exists. but whether the system can translate observability into real operational control at the right moment.