The Outage Nobody Recovered Cleanly From
A SaaS company experienced a 14-hour partial outage in 2024. The primary database recovered within RTO targets. The application servers recovered within RTO targets. The AI features did not recover for an additional 22 hours because nobody had documented how to rebuild the embedding indexes, restore the prompt management state, or revalidate the model performance after the regional failover.
The post-incident review found the disaster recovery plan was written in 2022 and did not contemplate AI workloads. The DR plan was technically followed. The business was offline anyway.
Uptime Institute's 2024 outage analysis found 60 percent of large outages now cost more than $100K, with cloud-scale outages reaching tens of millions (Uptime Institute, "Annual Outage Analysis 2024"). AI workloads complicate the math because model state, embeddings, and prompt versions have different recovery profiles than traditional application state.
If your DR plan was written before 2024, it probably does not handle AI workloads correctly. The math worth doing is what RPO and RTO actually mean for each component of your modern stack.
Coined Frame: The Component-Specific Math
Traditional DR planning uses a single RPO (recovery point objective, how much data loss is acceptable) and a single RTO (recovery time objective, how long until service is restored) per system. Modern stacks with AI workloads need component-specific targets.
Component 1 - Application state. Customer-facing application data, user sessions, transactional state. RPO measured in seconds to minutes for most modern applications. RTO measured in minutes to hours. Well-understood DR patterns apply.
Component 2 - Database state. Primary data store, source of truth. RPO measured in seconds with synchronous replication or minutes with asynchronous. RTO depends on failover automation. Mature patterns exist.
Component 3 - Model and prompt state. Which model version is in production, which prompt versions are deployed, which retrieval pipeline configurations are active. RPO can usually be zero (these are versioned artifacts that exist outside the production system). RTO depends on automation; manual restoration can take hours.
Component 4 - Embedding and vector state. Indexes that may have been built over months. Full rebuild can take days. RPO is the question: can we afford to lose recent embeddings, or do we need point-in-time recovery on the vector store. RTO is heavily dependent on whether the vector store has built-in DR capability.
Component 5 - Evaluation and observability state. Eval results, production traces, drift metrics, audit logs. RPO matters for compliance. RTO often less critical operationally but compliance-critical.
Component-specific RPO and RTO targets produce DR plans that actually recover AI-integrated systems. Single-target plans usually miss components 3 through 5 entirely.
What RPO and RTO Cost
The DR cost curve is exponential. Tighter targets cost much more than looser ones.
RPO = 0 (no data loss). Requires synchronous replication. Doubles or triples infrastructure cost for affected components. Justified for workloads where any data loss is unacceptable (financial transactions, healthcare records).
RPO < 1 minute. Requires near-synchronous replication. Adds 20-50 percent to infrastructure cost for affected components. Justified for many production workloads.
RPO < 1 hour. Requires asynchronous replication with low replication lag. Adds 10-25 percent to cost. Acceptable for most enterprise workloads.
RPO < 24 hours. Daily backup and restore. Adds 5-15 percent to cost. Acceptable for batch and analytical workloads.
RTO < 5 minutes. Requires hot standby. Doubles infrastructure for affected components. Justified for revenue-critical customer-facing workloads.
RTO < 1 hour. Warm standby with automation. Adds 30-60 percent to cost.
RTO < 24 hours. Cold standby or rebuild from backup. Adds 10-25 percent to cost.
The honest exercise is identifying which workloads genuinely require which tier. Most enterprises over-protect non-critical workloads and under-protect critical ones because the decisions were made by infrastructure team alone rather than with business input on actual cost of downtime.
The AI Workload Twist
Four AI workload patterns have specific DR considerations.
Embedding indexes. Rebuilding a vector store from raw documents can take days at scale. The DR strategy is either point-in-time replication of the vector store (which most vector databases now support) or maintaining a warm standby vector store that is kept in sync. The choice depends on RTO target.
Prompt versions and configurations. These are usually versioned in source control and tooling like LangSmith or Langfuse. The DR strategy is ensuring the version control and tooling are themselves resilient. Lost prompt versions are rare but high-impact when they happen.
Model provider dependencies. Outages at OpenAI, Anthropic, Google, or AWS Bedrock affect your AI features regardless of your infrastructure DR. The DR strategy here is multi-provider fallback designed into the application layer.
Training and fine-tuning artifacts. For workloads with custom fine-tuned models, the training artifacts need DR coverage. Storage in cold tiers with retention sufficient to retrain if needed.
Each pattern has standard DR approaches in 2026. The teams that have updated their DR plans incorporate them. The teams that have not are vulnerable to the 22-hour-extra outage in the story above.
The Testing Discipline
DR plans that are not tested do not work. The teams that recover successfully from real incidents are the teams that practice DR regularly.
The cadence that works for most enterprises is quarterly DR exercises for major components, annual full DR exercises for entire systems. The exercises are not optional and not skipped. Each exercise produces a written report on what worked, what failed, and what changes the plan needs.
AI workload DR testing has to specifically include the AI components. Testing whether the application servers fail over correctly without testing whether the model state, embeddings, and prompts recover correctly produces partial coverage. The full system needs to be exercised.
What This Costs
Building modern DR coverage for AI-integrated systems typically adds 15-30 percent to infrastructure cost for the workloads that get coverage. The investment in DR automation, runbooks, and testing typically requires one senior engineer for one quarter for initial setup, plus 10-20 percent ongoing for testing and maintenance.
The alternative cost is the cost of outages that take longer to recover from than they should. For most enterprises, this is much larger than the DR investment.
What Logiciel Does Here
Logiciel works with engineering teams modernizing their DR posture for AI-integrated systems. The work is structured around component-specific RPO/RTO assessment and the AI-specific patterns that traditional DR plans have not yet incorporated.
The Continuous Intelligence Reliability framework covers the broader reliability posture that DR fits within. The Cloud Security Architecture framework covers the security implications of DR architecture decisions.
A 30-minute working session is enough to identify the most exposed AI workload DR gaps.
Frequently Asked Questions
What is the right RPO for a typical SaaS application in 2026?
Customer-facing components typically need RPO under 5 minutes. Administrative and analytical components can accept hours. The specific answer depends on cost of data loss to customers and business.
Do I need a separate DR cloud or region?
For most enterprises, a different region of the same cloud provides sufficient resilience. Multi-cloud DR is justified for very high availability requirements or specific regulatory needs. Most failures affect single regions, not entire cloud providers.
How often should I test DR?
Quarterly for major components, annually for full system. The cadence that catches problems before they catch you during a real incident.
How do I handle DR for third-party AI dependencies?
Multi-provider fallback in the application layer. If the primary model provider has an outage, the application routes to a secondary provider with eval-validated comparable behavior. The architecture has to support this from initial design.
What is the right organizational owner for DR?
Engineering operations owns execution. Product owns RPO/RTO target setting based on business impact. Security owns compliance requirements. Joint ownership with clear lanes works better than single-owner DR programs. Sources: - Uptime Institute, "Annual Outage Analysis 2024" - Flexera, "2024 State of the Cloud Report"
