The Contract Document That Did Not Hold
A data architect at a logistics company showed me a contract document from 2023 that her team had written between the order management team (producer) and the analytics team (consumer). The document specified field types, refresh cadence, null-handling, and a 30-day notice requirement for breaking changes. Twenty pages, signed by both sides, filed in Confluence.
By mid-2024 the contract had been violated nine times in measurable ways. Once with notice, eight without. Each violation produced a downstream incident. Each incident triggered a meeting where everyone agreed contracts mattered. Nothing structural changed. The document stayed in Confluence. The violations continued.
She called the experience "the contract paradox." Teams that say they value contracts the loudest are often the teams whose contracts hold the least, because the contracts exist as documents rather than as infrastructure. The reverse is also true. Teams whose contracts hold reliably usually do not talk about contracts much because the enforcement is automatic and unremarkable.
The pattern is consistent enough to be worth diagnosing. Data contracts hold when infrastructure enforces them. Documents that ask producers to honor commitments do not hold because nobody operates against documents under deadline pressure.
What Real Enforcement Looks Like
Three enforcement patterns produce contracts that actually hold. Each one addresses a specific failure mode that document-based contracts cannot address.
The first pattern is producer-side schema gates. The contract is expressed as a schema (Avro, Protobuf, JSON Schema, or a typed table contract). Producer changes that violate the schema fail at build time, not at runtime. A pull request that breaks the contract cannot be merged.
The mechanics matter. Schema registries (Confluent Schema Registry, Apicurio, Glue Schema Registry) hold the contract. CI integration checks producer changes against the registered schema. Compatibility rules (backward, forward, full) define what changes are allowed without consumer breakage. Changes that need consumer coordination get flagged before they ship.
Without producer-side schema gates, every producer change is a roll of the dice. With them, breaking changes require deliberate effort and consumer notification.
The second pattern is contract validation at the boundary. As data crosses from producer to consumer, the validation runs. Records that violate the contract get routed to dead-letter handling. Records that conform proceed normally. The consumer never sees malformed data.
The mechanics are pipeline-level. Stream processing platforms (Kafka with Schema Registry, Pulsar, Kinesis) support this through serializers that reject incompatible records. Batch pipelines support it through validation stages early in the pipeline. The pattern is engineering work that has to be designed in.
Without boundary validation, consumer code accumulates defensive logic to handle every possible producer mistake. The defensive logic is fragile and incomplete. With boundary validation, consumer code can trust its inputs.
The third pattern is observability that surfaces contract violations operationally. Schema registry tracks compatibility events. Validation pipelines emit metrics. Dashboards show violation rates over time. Alerts fire when violations cross thresholds.
The mechanics include integration with the existing observability stack. Datadog, Grafana, or the data observability platforms (Monte Carlo, Bigeye) surface contract-related signals alongside other operational metrics. The data engineering team sees contract health as part of platform health.
Without observability, contract violations get noticed by consumer incidents rather than by producer-side signals. The detection lag matters because violations propagate through downstream systems before discovery.
We analysed 100 CTOs
This report shows what actually predicts delivery success and what CTOs discover too late.
The Three Patterns in Practice
The patterns work together. Producer-side gates prevent most violations. Boundary validation catches the violations that bypass gates. Observability surfaces both the gate hits and the violations for ongoing management.
A team operating all three has data contracts that genuinely hold. Producer teams know what they can change unilaterally and what requires coordination. Consumer teams trust the data they receive. The platform team has visibility into contract health across the organization.
A team operating one or two patterns has partial coverage. The gaps emerge predictably. Without producer-side gates, the team relies on validation to catch issues, which is later than catching them at build time. Without boundary validation, the team relies on gates that producers can sometimes bypass. Without observability, the team operates blind to contract drift.
Most teams in 2026 operate one of the three. Producer-side gates have become standard through schema registries. Boundary validation is sometimes present, sometimes not. Observability is rare. The combination of all three is what distinguishes mature contract enforcement from theoretical contract enforcement.
The Cultural Component That Has to Exist
Technical enforcement is necessary and not sufficient. The cultural pattern that has to accompany the technical enforcement is producer accountability.
Producer teams have to understand what contracts they hold and care about honoring them. The understanding comes from making contracts visible. The caring comes from making violations have consequences.
The visibility usually requires data product framing. Producer teams treat the data they emit as products their consumers depend on. The framing changes how producers think about their work. The data is not exhaust from operational systems; it is a deliberate output that downstream teams build against.
The consequences require organizational backing. Contract violations get tracked. Patterns of repeated violations escalate. Producer teams get recognized for maintaining clean contracts and held accountable for breaking them. The recognition and accountability cannot be theatrical; they have to affect actual decisions.
Without the cultural pattern, the technical enforcement gets routed around. Producer teams find ways to ship breaking changes despite the gates, treating contract enforcement as friction rather than as discipline. With the cultural pattern, the technical enforcement reinforces what producers already value.
What Contracts Should Cover
Contracts that are too permissive provide no protection. Contracts that are too restrictive prevent legitimate evolution. Calibrating contract scope matters.
A useful contract covers schema (field names, types, nullability), semantics (what the fields mean, business rules, validation constraints), freshness (how often the data updates, latency expectations), and operational properties (SLA for availability, breaking change notice periods).
Contracts do not typically cover data values directly. The contract specifies that order amounts are positive decimals, not that any specific order amount has a specific value. The latter would over-constrain the producer.
Contracts also typically do not cover internal implementation. Producers can change how they generate the data as long as the output continues to conform to the contract. The contract is the interface, not the implementation.
This calibration takes practice. Initial contracts often over-constrain or under-specify. Iteration over months produces contracts that fit the actual needs of producer and consumer teams.
What This Costs
Building contract enforcement infrastructure typically requires one to two quarters of focused platform engineering work plus ongoing 10-15 percent of one engineer's capacity for sustained operation. The work spans schema registry deployment, CI integration, pipeline validation, and observability.
The alternative cost is the cost of contract violations that escape to consumers. For most enterprises, the violations cost more in incident response, data correction, and consumer trust than the enforcement infrastructure costs to operate.
The cultural work is harder to cost directly. The investment is in process design, training, and organizational accountability. The returns compound over time as producer teams internalize the contract discipline.
Why the Best CTOs Don't Hire, They Audit
Inside a one-quarter overhead audit that pulled a five-person data team back from 67% firefighting.
What Logiciel Does Here
Logiciel works with data engineering and platform teams establishing contract enforcement for data flowing between producer and consumer teams. The work is typically structured around assessment of current contract state followed by sequenced buildout of the three patterns.
The Building a Data Platform That AI Teams Actually Want to Use framework covers the broader platform context that contracts fit within. The Data Observability: Why Dashboards Lie framework covers the observability discipline that contract enforcement depends on.
A 30-minute working session is enough to assess your current contract enforcement against the three patterns.
Frequently Asked Questions
What is the right tooling for schema registry?
Confluent Schema Registry for Kafka-centric platforms. AWS Glue Schema Registry for AWS-centric platforms. Apicurio as an open-source alternative. The choice depends on existing platform fit. The registry is necessary; the specific tool matters less than its consistent use.
How do I handle legacy systems that resist contracts?
Through wrapping. The legacy system's output gets wrapped in a contract-compliant boundary that downstream consumers see. The wrapping is engineering work but isolates the legacy system from contract requirements while still providing contract guarantees to consumers.
What about contracts for unstructured data?
Harder to specify but possible. Contracts for documents, images, or other unstructured content cover metadata schemas, file format guarantees, processing freshness, and the semantic categories the data falls into. The contract is more about the data's properties than its detailed structure.
How do I prevent contract enforcement from becoming a bottleneck?
By distributing ownership. Producer teams own their contracts. The platform team provides infrastructure. Consumer teams negotiate evolution with producers. Central enforcement that gates every change becomes a bottleneck; distributed enforcement that runs automatically does not.
How does this work for data that has no clear single producer?
Through stewardship. Even when multiple sources contribute to data, one team is designated as steward and holds the contract. The stewardship pattern is common for shared concepts (customer, product, order) that span multiple operational systems. The steward team consolidates and serves the data with contract guarantees. Sources: - Confluent Schema Registry documentation, 2024 - Monte Carlo, "State of Data Quality 2024"
