Data Downtime.

System downtime occurs when infrastructure is unavailable: your database goes offline, your data warehouse rejects connections, or your API is down. Data downtime occurs when the system is running but the data is broken: the pipeline succeeded but inserted null values due to a schema change, upstream data became stale, or a transformation applied wrong business logic.

Your system can be 99.9% available while experiencing frequent data downtime. A pipeline might complete every hour reliably but gradually accumulate silent errors that corruption all the data. Dashboards work fine but show wrong numbers. This is the dangerous scenario because infrastructure monitoring looks healthy.

Data downtime breaks the business layer. System downtime breaks the infrastructure layer. Both matter, but they require different detection and response approaches. You need system monitoring for availability. You need data quality monitoring for correctness.

Common causes include schema changes from upstream systems, pipeline failures from timeout or resource limits, incorrect transforms that apply wrong logic, missing data from source API changes, stale data from delayed upstream loads, and referential integrity violations when join keys don't exist.

Many incidents involve multiple causes: a schema changed at 2am, no one caught it until 8am when alerts finally triggered, and by then bad data had been loaded and consumed downstream for hours. The longer you don't detect data downtime, the more damage occurs. This is why observability matters more than just logging pipeline completion.

Some root causes are technical failures that show in logs. Others are silent: data quality degradation that code considers valid but which violates business rules. These require explicit quality checks, not just error log inspection.

Use data quality tools to run tests after every pipeline run: check for null rates, cardinality changes, schema compliance, and business logic validity. Set thresholds for each metric so you alert when values drift out of expected ranges. Use data profiling to establish baselines for each table, then detect anomalies automatically.

Monitor freshness by comparing arrival time to expected refresh time. Log all tests and anomalies so you have historical records and can identify patterns. The key is not waiting for consumer complaints. Detect data downtime in your pipeline before it reaches dashboards or analytical tools.

Combine hand-written tests for known risks with anomaly detection for unexpected issues. Hand-written tests are precise but labor-intensive. Anomaly detection is automated but requires tuning. Use both together for comprehensive coverage.

MTTR is mean time to recovery: the average time between when data downtime starts and when it is resolved. A data quality issue that starts at 2am but is not detected until 8am because alerts didn't trigger has a 6-hour MTTR. Fast MTTR is critical because bad data spreads downstream.

Stale data affects dashboards immediately. Incorrect data corrupts analytics and reporting. Long MTTR means more users see the bad data and make wrong decisions. Reducing MTTR requires faster detection (observability) and faster response (clear incident procedures). Some teams aim for under 15 minutes from detection to resolution.

MTTR includes both the time before detection and the time after. Improving either reduces total MTTR. Fast detection requires good alerting. Fast resolution requires clear incident procedures and runbooks so teams know what to do when an alert fires.

Log every incident with metadata: when it started, when it was detected, root cause, how long it lasted, and which datasets were affected. Track the severity by measuring downstream impact. A schema change that affected a rarely-used report is less severe than one affecting your main dashboard.

Use incident logs to identify systemic patterns. If the same upstream source keeps changing schema, you know you need better contracts with that team or more robust transforms. Track metrics like total hours of data downtime per month, average MTTR, and percentage of downtime detected within 15 minutes. Use these metrics to justify investment in observability.

Share incident logs and lessons learned with your team. Use them to update runbooks. Use them to identify which monitoring gaps allowed incidents to go undetected. This transforms each incident into an opportunity to improve your observability and response processes.

Data latency is the expected delay between when something happens in the source and when it appears in the warehouse. If your pipeline runs every hour, there is inherent one-hour latency. That is normal and acceptable. Data downtime is when the actual latency exceeds your SLA. If your SLA is one hour but data has not refreshed in three hours, that is data downtime.

Latency is predictable and engineered. You design pipelines with specific latency characteristics. Downtime is unexpected and broken. You can have tight latency (15-minute refresh) and still have downtime if that 15-minute window breaks. You can have loose latency (daily refresh) and have low downtime if your daily refresh is reliable.

Understanding the difference helps you set realistic SLAs and build appropriate monitoring. Do not conflate expected latency with downtime. Monitor for downtime (actual latency exceeding SLA) separately from monitoring normal latency patterns.

Build observability into your pipeline architecture: test data quality at every transformation step, not just at the end. Validate schemas before loading. Check that row counts do not drift unexpectedly. Alert on stale upstream sources. Use version control for transform logic so you can quickly revert bad changes.

Build circuit breakers: if a pipeline detects severe data quality issues, stop loading rather than propagating bad data. Establish clear incident response: when an alert fires, who investigates? How are teams notified? Document runbooks for common failure modes so resolution is faster. Test your monitoring itself. An alert that never fires is useless. Test that it triggers correctly when data actually breaks.

Invest in upstream reliability. If source systems change schema frequently, create contracts with those teams. If upstream loads are inconsistent, build better monitoring or redundant sources. Prevention is cheaper than detection and response.

Data quality platforms like Great Expectations, Soda, and Databand let you define tests and run them automatically in your pipelines. They track test results over time and alert when metrics drift. Data observability tools like Bigeye, Databand, and Monte Carlo go further: they profile your data, detect anomalies automatically without requiring hand-written tests, and correlate issues across multiple datasets to find root causes.

Data catalogs help track data lineage so you can see which datasets feed into which reports. When an incident occurs, lineage helps you trace backward to the root cause. Data governance tools help define and enforce data standards so teams understand what quality is expected.

The investment in these tools pays for itself quickly by reducing MTTR and preventing bad data from reaching consumers. The cost of a single major data quality incident that goes undetected for days often exceeds the annual cost of observability tooling.

Prevention through testing and validation catches issues before they happen. Detection through observability catches issues quickly after they happen. You need both. Prevent schema changes from breaking your pipeline by using defensive transforms that handle optional columns. Detect unexpected schema changes by running schema compliance tests.

Prevent stale data by monitoring upstream sources and alerting if they have not updated. Detect staleness by comparing arrival time to expected time. Good data teams invest heavily in prevention so fewer incidents happen. They also invest in detection and response so when something does break, it is caught and fixed in minutes, not hours.

Prevention and detection are complementary, not alternatives. As you mature, you improve both. The result is fewer incidents (prevention) and shorter response times when incidents do occur (detection).

Include the incident date, start time, detection time, resolution time, root cause, affected datasets, downstream impact, and preventative actions taken. Be specific about impact: which reports were wrong, which decisions were made on stale data, how many users were affected.

Include what alarms should have fired but did not, so you can improve monitoring. Use incident reports to educate the team about what happened and what they should watch for next time. Share them with stakeholders so they understand the importance of data quality investment. Use them to justify spending on observability and infrastructure improvements.

Template your incident reports so information is consistent and comparable over time. Track trends in incident types, root causes, and MTTR. Use trends to identify which monitoring improvements or preventative measures would have the highest impact.

Track hours of downtime per month. Initially this number will be high because you are measuring things you were not measuring before. But you should see it decline as you fix root causes and improve detection. Track MTTR: the time between when downtime starts and when it is resolved. Fast detection means short MTTR.

Track alert effectiveness: what percentage of data downtime was detected by your automated alerts vs discovered by complaints? Higher is better. Track prevention effectiveness: how many potential data quality issues did you catch in testing before they reached production? This requires instrumentation but pays off by reducing incidents.

Set targets for each metric and review them monthly. Celebrate progress. Share metrics with leadership so they understand the value of data quality investment. If downtime hours are declining, MTTR is improving, and alert effectiveness is increasing, your program is working.

Have a clear incident response plan. When an alert fires, notify the on-call data engineer immediately. They triage the incident: how severe is it? Which datasets are affected? Does it need escalation? If severe, page out the team. Investigate systematically using data lineage to trace the problem.

Was it an upstream source delay, a failed transformation, or a schema mismatch? Identify the root cause. If the data can be fixed easily, roll back the pipeline or reprocess the data. If it cannot be fixed easily, communicate the impact to downstream teams so they know not to rely on that data.

Post-incident, document what happened, why alerts did not catch it sooner, and how to prevent recurrence. Follow up on action items from the incident review. This transforms each incident into a learning opportunity and prevents similar issues from recurring.

Data downtime and data quality are related but not identical. Data downtime is the state: data is missing, wrong, or stale. Data quality is the discipline: the practices, tools, and processes that keep data downtime low. You maintain data quality through testing, monitoring, and incident response.

The result is that data downtime is rare and brief. A data team with poor quality practices might have days of undetected data downtime per month. A team with good quality practices might have hours or minutes. As you mature, you measure both to understand your starting point and track improvement over time.

Data quality is what you build. Data downtime is what happens when quality fails. Invest in quality practices to keep downtime minimal. Invest in observability to detect downtime quickly when it does occur.