Data downtime is the period when data is missing, wrong, or stale. It is the state where a dataset cannot be trusted for decision-making because it is incomplete, incorrect, or outdated. The term was coined by Barr Moses at Monte Carlo Data to describe a problem that was being conflated with system downtime, even though they are fundamentally different. System downtime is infrastructure breaking. Data downtime is the data itself being broken.
Data downtime differs from system downtime because your infrastructure can be completely healthy while your data is corrupted. A pipeline can complete successfully every hour and still deliver null values due to a schema change that no one caught. A database can accept connections fine while returning stale records because an upstream source stopped updating. A warehouse can run queries without error while those queries return incorrect results because a transformation applied wrong business logic. The system works. The data doesn't.
The numbers behind downtime are significant. Splunk's global enterprise survey found that close to 66% of organisations report each hour of downtime costs more than $150,000. Fivetran's 2026 Benchmark puts the average impact of a single pipeline failure in a large enterprise at $1.4M, with organisations averaging 60+ hours of pipeline downtime per month. Monte Carlo and Wakefield Research found that 68% of organisations take four or more hours just to detect a data incident — before resolution even begins — and once detected, incidents take an average of 15+ hours to fix.
The impact of data downtime flows downstream quickly. Analytics teams build reports on bad data. Dashboards show false trends. ML models train on incorrect records. Sales teams make forecasts on stale information. Fraud detection systems operate blind. The longer data downtime persists undetected, the more damage cascades through the organization.
Reducing data downtime requires both prevention and detection. Prevention means building robust transforms and monitoring upstream sources so issues don't happen. Detection means running data quality tests automatically and alerting before downstream consumers see the bad data. Most teams are weak at detection, which is why data downtime often goes unnoticed for hours or days.
Data downtime occurs when data is missing, wrong, or stale, distinct from system downtime which is infrastructure unavailability, and requires completely different detection and response approaches.
Common root causes include schema changes from upstream systems, pipeline execution failures, incorrect transformation logic, missing source data, and stale upstream loads that remain undetected.
Detecting data downtime requires automated data quality testing after each pipeline run, not relying on system logs or manual checks that scale poorly and arrive too late.
MTTR (mean time to recovery) measures the average time between when data downtime starts and when it is resolved, and shorter MTTR reduces downstream impact and prevents bad data from propagating.
Reducing data downtime requires investment in observability tools that profile data, detect anomalies, track test results, and correlate issues across datasets to find root causes quickly.
Success is measured by tracking hours of downtime per month, average MTTR, and the percentage of incidents detected by automated monitoring rather than consumer complaints.
System downtime happens at the infrastructure layer. Your Airflow scheduler crashes. Your data warehouse is unreachable. Your cloud storage goes offline. These are visibility issues that affect everyone immediately and show up in monitoring dashboards. System downtime is binary: either the system is working or it is not.
Data downtime happens at the data layer. The system is running fine. Pipelines complete successfully. Queries execute without error. But the data itself is broken. A schema change in a source system means your pipeline inserts null values. You don't realize it because the pipeline job status is green. Hours later, dashboards are showing wrong numbers.
Many organizations conflate these because they use similar tools. System monitoring and data quality monitoring look different. System monitoring answers "Is the warehouse available?" Data monitoring answers "Is the data in the warehouse correct?" You need both. Some teams over-invest in system reliability while completely ignoring data quality, creating a false sense of security.
Schema changes from upstream systems are common. A source system adds a column as required instead of nullable. Your transform doesn't handle it. Pipeline fails silently or inserts nulls. You don't know until someone complains. Source systems rarely announce breaking changes, so you need to detect schema drifts automatically.
Pipeline failures from infrastructure problems also cause downtime. A job timeout exceeds expected duration. Memory limits are hit. A dependency service goes down. The job fails to complete. Instead of alerting immediately, teams discover the failure the next morning when they check logs. The data is missing for hours.
Incorrect transform logic is insidious because it succeeds silently. A developer rewrote the logic for calculating revenue and made a mistake. The pipeline runs fine. Numbers are slightly wrong in every row. You don't catch it because tests didn't cover that edge case. Bad data propagates downstream for weeks.
Missing or stale data from upstream sources also causes downtime. An external API stops updating. You don't notice and keep loading old data. Downstream consumers see stale records without realizing they are stale. A source data export fails but you continue using yesterday's snapshot without realizing it is old.
Automated detection is non-negotiable. Manual spot-checks don't scale. You cannot inspect every dataset every hour and hope to catch issues before they reach production. Use data quality tools to define tests that run automatically in your pipeline after each transformation and load.
Common tests include null rate validation (column should have less than 0.1% nulls), row count drift (counts should not change by more than 10% unexpectedly), cardinality checks (distinct value count should be within expected range), schema compliance (columns should match expected types), and range validation (values should fall within expected bounds).
Beyond hand-written tests, use data profiling and anomaly detection to catch unexpected behavior. Establish baselines for each dataset. If freshness suddenly increases or row counts drop, alert immediately. Some tools use machine learning to detect anomalies without requiring you to specify exact thresholds. This catches issues that would slip past manual tests.
MTTR is mean time to recovery. It measures the average time between when data downtime starts and when it is resolved. If a data quality issue begins at 2am but is not detected until 8am when an alert finally fires, your MTTR is at least 6 hours. That is 6 hours of bad data potentially affecting dashboards and decisions.
MTTR includes both detection time and resolution time. Fast detection means your observability is good and alerts trigger quickly. Fast resolution means you have clear incident procedures and your team can diagnose and fix problems efficiently. Fast resolution also assumes you can quickly identify the root cause using data lineage, so you know whether to rollback, reprocess, or fix the source.
High-performing teams aim for MTTR under 15 minutes. They detect issues within 5 minutes and resolve them within 10. This requires investment in alerting, monitoring, and incident response infrastructure. It also requires runbooks for common failure modes so engineers don't waste time guessing. Tracking MTTR over time shows whether your investments in observability and response processes are working.
Data quality platforms like Great Expectations and Soda let you embed tests in your data pipeline. After each transformation and load, tests run automatically. If thresholds are exceeded, the pipeline stops before loading bad data. Test results are logged so you have historical records. This prevents bad data from reaching production most of the time.
Data observability platforms like Monte Carlo, Bigeye, and Databand go deeper. They profile your data continuously, detect anomalies without requiring hand-written tests, and correlate issues across datasets. If upstream freshness delays, they automatically detect and alert. If cardinality changes unexpectedly, they flag it. If a downstream dataset is affected by an upstream change, they trace the impact automatically.
Data catalogs and lineage tools help you understand data dependencies. When an incident occurs, lineage helps you trace backward to find the root cause. Did a transformation fail? Did an upstream source change? Did a join key disappear? Lineage answers these questions quickly, reducing diagnosis time and MTTR.
The first challenge is visibility. Many organizations don't even measure data downtime because they lack observability. Without measurement, you cannot manage improvement. You might have hours of undetected downtime per month and not realize it. The first step is instrumenting your pipelines to detect and track downtime so you understand your baseline.
The second challenge is alert fatigue. If you define thresholds too aggressively, you get false positives. Alerts fire constantly. Teams ignore them because most are not real issues. You need to calibrate thresholds carefully based on actual data distributions, not arbitrary rules. This requires statistical analysis and experimentation. Many teams either over-alert (causing fatigue) or under-alert (missing real issues).
The third challenge is organizational alignment. Data quality is not a single team's problem. If upstream systems change without warning, downstream teams cannot prevent downtime. If downstream teams don't report problems, upstream teams don't know their data is breaking consumers. This requires establishing data contracts between teams, sharing alerts across teams, and creating accountability for data quality improvements.
Finally, responding to data downtime quickly requires both technical capability and organizational processes. You need tools to detect issues. You also need a clear incident response procedure: who investigates when an alert fires? How are teams notified? How do you decide whether to rollback or reprocess? Without processes, fast detection doesn't lead to fast resolution.
System downtime occurs when infrastructure is unavailable: your database goes offline, your data warehouse rejects connections, or your API is down. Data downtime occurs when the system is running but the data is broken: the pipeline succeeded but inserted null values due to a schema change, upstream data became stale, or a transformation applied wrong business logic.
Your system can be 99.9% available while experiencing frequent data downtime. A pipeline might complete every hour reliably but gradually accumulate silent errors that corruption all the data. Dashboards work fine but show wrong numbers. This is the dangerous scenario because infrastructure monitoring looks healthy.
Data downtime breaks the business layer. System downtime breaks the infrastructure layer. Both matter, but they require different detection and response approaches. You need system monitoring for availability. You need data quality monitoring for correctness.
Common causes include schema changes from upstream systems, pipeline failures from timeout or resource limits, incorrect transforms that apply wrong logic, missing data from source API changes, stale data from delayed upstream loads, and referential integrity violations when join keys don't exist.
Many incidents involve multiple causes: a schema changed at 2am, no one caught it until 8am when alerts finally triggered, and by then bad data had been loaded and consumed downstream for hours. The longer you don't detect data downtime, the more damage occurs. This is why observability matters more than just logging pipeline completion.
Some root causes are technical failures that show in logs. Others are silent: data quality degradation that code considers valid but which violates business rules. These require explicit quality checks, not just error log inspection.
Use data quality tools to run tests after every pipeline run: check for null rates, cardinality changes, schema compliance, and business logic validity. Set thresholds for each metric so you alert when values drift out of expected ranges. Use data profiling to establish baselines for each table, then detect anomalies automatically.
Monitor freshness by comparing arrival time to expected refresh time. Log all tests and anomalies so you have historical records and can identify patterns. The key is not waiting for consumer complaints. Detect data downtime in your pipeline before it reaches dashboards or analytical tools.
Combine hand-written tests for known risks with anomaly detection for unexpected issues. Hand-written tests are precise but labor-intensive. Anomaly detection is automated but requires tuning. Use both together for comprehensive coverage.
MTTR is mean time to recovery: the average time between when data downtime starts and when it is resolved. A data quality issue that starts at 2am but is not detected until 8am because alerts didn't trigger has a 6-hour MTTR. Fast MTTR is critical because bad data spreads downstream.
Stale data affects dashboards immediately. Incorrect data corrupts analytics and reporting. Long MTTR means more users see the bad data and make wrong decisions. Reducing MTTR requires faster detection (observability) and faster response (clear incident procedures). Some teams aim for under 15 minutes from detection to resolution.
MTTR includes both the time before detection and the time after. Improving either reduces total MTTR. Fast detection requires good alerting. Fast resolution requires clear incident procedures and runbooks so teams know what to do when an alert fires.
Log every incident with metadata: when it started, when it was detected, root cause, how long it lasted, and which datasets were affected. Track the severity by measuring downstream impact. A schema change that affected a rarely-used report is less severe than one affecting your main dashboard.
Use incident logs to identify systemic patterns. If the same upstream source keeps changing schema, you know you need better contracts with that team or more robust transforms. Track metrics like total hours of data downtime per month, average MTTR, and percentage of downtime detected within 15 minutes. Use these metrics to justify investment in observability.
Share incident logs and lessons learned with your team. Use them to update runbooks. Use them to identify which monitoring gaps allowed incidents to go undetected. This transforms each incident into an opportunity to improve your observability and response processes.
Data latency is the expected delay between when something happens in the source and when it appears in the warehouse. If your pipeline runs every hour, there is inherent one-hour latency. That is normal and acceptable. Data downtime is when the actual latency exceeds your SLA. If your SLA is one hour but data has not refreshed in three hours, that is data downtime.
Latency is predictable and engineered. You design pipelines with specific latency characteristics. Downtime is unexpected and broken. You can have tight latency (15-minute refresh) and still have downtime if that 15-minute window breaks. You can have loose latency (daily refresh) and have low downtime if your daily refresh is reliable.
Understanding the difference helps you set realistic SLAs and build appropriate monitoring. Do not conflate expected latency with downtime. Monitor for downtime (actual latency exceeding SLA) separately from monitoring normal latency patterns.
Build observability into your pipeline architecture: test data quality at every transformation step, not just at the end. Validate schemas before loading. Check that row counts do not drift unexpectedly. Alert on stale upstream sources. Use version control for transform logic so you can quickly revert bad changes.
Build circuit breakers: if a pipeline detects severe data quality issues, stop loading rather than propagating bad data. Establish clear incident response: when an alert fires, who investigates? How are teams notified? Document runbooks for common failure modes so resolution is faster. Test your monitoring itself. An alert that never fires is useless. Test that it triggers correctly when data actually breaks.
Invest in upstream reliability. If source systems change schema frequently, create contracts with those teams. If upstream loads are inconsistent, build better monitoring or redundant sources. Prevention is cheaper than detection and response.
Data quality platforms like Great Expectations, Soda, and Databand let you define tests and run them automatically in your pipelines. They track test results over time and alert when metrics drift. Data observability tools like Bigeye, Databand, and Monte Carlo go further: they profile your data, detect anomalies automatically without requiring hand-written tests, and correlate issues across multiple datasets to find root causes.
Data catalogs help track data lineage so you can see which datasets feed into which reports. When an incident occurs, lineage helps you trace backward to the root cause. Data governance tools help define and enforce data standards so teams understand what quality is expected.
The investment in these tools pays for itself quickly by reducing MTTR and preventing bad data from reaching consumers. The cost of a single major data quality incident that goes undetected for days often exceeds the annual cost of observability tooling.
Prevention through testing and validation catches issues before they happen. Detection through observability catches issues quickly after they happen. You need both. Prevent schema changes from breaking your pipeline by using defensive transforms that handle optional columns. Detect unexpected schema changes by running schema compliance tests.
Prevent stale data by monitoring upstream sources and alerting if they have not updated. Detect staleness by comparing arrival time to expected time. Good data teams invest heavily in prevention so fewer incidents happen. They also invest in detection and response so when something does break, it is caught and fixed in minutes, not hours.
Prevention and detection are complementary, not alternatives. As you mature, you improve both. The result is fewer incidents (prevention) and shorter response times when incidents do occur (detection).
Include the incident date, start time, detection time, resolution time, root cause, affected datasets, downstream impact, and preventative actions taken. Be specific about impact: which reports were wrong, which decisions were made on stale data, how many users were affected.
Include what alarms should have fired but did not, so you can improve monitoring. Use incident reports to educate the team about what happened and what they should watch for next time. Share them with stakeholders so they understand the importance of data quality investment. Use them to justify spending on observability and infrastructure improvements.
Template your incident reports so information is consistent and comparable over time. Track trends in incident types, root causes, and MTTR. Use trends to identify which monitoring improvements or preventative measures would have the highest impact.
Track hours of downtime per month. Initially this number will be high because you are measuring things you were not measuring before. But you should see it decline as you fix root causes and improve detection. Track MTTR: the time between when downtime starts and when it is resolved. Fast detection means short MTTR.
Track alert effectiveness: what percentage of data downtime was detected by your automated alerts vs discovered by complaints? Higher is better. Track prevention effectiveness: how many potential data quality issues did you catch in testing before they reached production? This requires instrumentation but pays off by reducing incidents.
Set targets for each metric and review them monthly. Celebrate progress. Share metrics with leadership so they understand the value of data quality investment. If downtime hours are declining, MTTR is improving, and alert effectiveness is increasing, your program is working.
Have a clear incident response plan. When an alert fires, notify the on-call data engineer immediately. They triage the incident: how severe is it? Which datasets are affected? Does it need escalation? If severe, page out the team. Investigate systematically using data lineage to trace the problem.
Was it an upstream source delay, a failed transformation, or a schema mismatch? Identify the root cause. If the data can be fixed easily, roll back the pipeline or reprocess the data. If it cannot be fixed easily, communicate the impact to downstream teams so they know not to rely on that data.
Post-incident, document what happened, why alerts did not catch it sooner, and how to prevent recurrence. Follow up on action items from the incident review. This transforms each incident into a learning opportunity and prevents similar issues from recurring.
Data downtime and data quality are related but not identical. Data downtime is the state: data is missing, wrong, or stale. Data quality is the discipline: the practices, tools, and processes that keep data downtime low. You maintain data quality through testing, monitoring, and incident response.
The result is that data downtime is rare and brief. A data team with poor quality practices might have days of undetected data downtime per month. A team with good quality practices might have hours or minutes. As you mature, you measure both to understand your starting point and track improvement over time.
Data quality is what you build. Data downtime is what happens when quality fails. Invest in quality practices to keep downtime minimal. Invest in observability to detect downtime quickly when it does occur.
