Fault injection is the practice of deliberately introducing errors, failures, or degraded conditions into a system to observe how it responds. Instead of waiting for a disk to fill up, a server to crash, or a downstream API to time out on its own schedule, a team causes that failure on purpose, under controlled conditions, and watches what happens next. The goal is not to break things for the sake of breaking them. It is to find out whether the safeguards a system is supposed to have (retries, timeouts, failover, alerting) actually work when they are needed.
The reason fault injection exists is that most systems are never tested against the conditions that actually take them down. Teams write unit tests and integration tests that check whether code behaves correctly when everything works. They rarely test what happens when a dependency is slow instead of down, when a network call returns malformed data instead of an error, or when a whole availability zone disappears. Those are the scenarios that produce real outages, and without deliberately creating them, a team's first real experience of that failure mode is often during an incident, in production, in front of customers.
What distinguishes fault injection from generic testing is that it targets failure conditions specifically, and it can be run at very different scales. At the smallest scale, a unit test mocks a dependency to throw an exception or return an unexpected value. At a larger scale, a staging environment might use a dedicated tool to add network latency, drop packets, or corrupt data between services. At the largest scale, a team runs a fault injection experiment directly in production, killing a real process or simulating a region outage, with safeguards in place to limit the damage. Good fault injection is hypothesis driven: before running an experiment, the team writes down what they expect to happen and why, for example "if we kill this service instance, we expect the load balancer to reroute traffic within five seconds because of the health check."
By 2026, fault injection has moved from a niche practice at a handful of large tech companies to something more ordinary across teams running distributed systems in the cloud. Applications now depend on dozens of services, third-party APIs, and managed infrastructure components, any one of which can fail independently of the others. Cloud providers have outages. Regions go down. Dependencies get slow instead of clearly failing, which is often harder to handle than an outright crash. As systems have gotten more distributed, the number of ways they can partially fail has grown much faster than most teams' intuition for how those failures cascade, which is exactly the gap fault injection is built to close.
This page covers what faults teams actually inject and why, how fault injection relates to chaos engineering, how a fault injection experiment gets designed and run safely, where the practice fits and where it doesn't, and how a team can start doing it without causing the very outage it's trying to prevent. The idea underneath all of it is simple: you cannot know how a system will behave under failure by reading its architecture diagram. You have to actually cause the failure, in a controlled way, and watch. Understanding fault injection lets a team replace guesses about resilience with evidence, and replace incident postmortems as the only source of that evidence with something they chose to learn on their own schedule.
Fault injection is not one technique, it's a family of techniques aimed at different layers of a system. At the infrastructure layer, teams kill a server or process outright, to see whether the system detects the loss and reroutes work. They fill up disk space, to see whether logging, temp files, or write paths degrade gracefully or take the whole host down. They throttle CPU or memory on a host to simulate resource contention, which behaves very differently from a clean crash because the process is still technically alive, just slow and unreliable.
At the network layer, teams add latency to calls between services, drop packets, or introduce partial packet loss, which is often more revealing than a hard failure because it exposes timeout settings that were never tuned. A service that "works fine" in every test but has no timeout configured at all will hang forever waiting on a slow dependency, and that failure mode almost never shows up until someone injects it on purpose. Simulating a dependency or API that times out, or one that returns errors instead of succeeding, tests whether the calling code actually has a fallback path, or whether that fallback path was written once and never exercised again.
At the data layer, fault injection can mean corrupting data intentionally, feeding malformed input, or forcing a write to partially succeed, to see whether downstream systems validate what they receive or blindly trust it. In cloud environments specifically, fault injection extends to simulating the failure of an entire availability zone or region, which tests things no smaller-scale test can reach: DNS failover, cross-region replication lag, and whether a supposedly redundant system actually fails over, or just fails in two places instead of one.
In hardware and embedded systems, fault injection takes an even more literal form: flipping individual bits in memory, injecting voltage glitches, or corrupting instructions, used specifically in safety-critical software for aviation and automotive systems where certification requires proof that the system fails safely under corrupted or noisy conditions. Across all of these layers, the common thread is that the fault is chosen deliberately, targeted at a specific mechanism the team wants to test, and not just randomly thrown at the system in the hope that something interesting happens.
People often use "fault injection" and "chaos engineering" interchangeably, but they are not the same thing, and the difference matters. Fault injection is the mechanism: the actual act of causing a failure, whether that's a killed process, a corrupted API response, or a network delay. Chaos engineering is a broader discipline built on top of that mechanism, focused specifically on running these experiments against production systems, usually at scale, with formal hypotheses, monitoring, and organizational buy-in. Every chaos engineering experiment involves fault injection. Not every fault injection exercise is chaos engineering.
Fault injection predates chaos engineering by decades. It has been used in hardware testing, safety-critical software certification, and basic software testing (mocking a dependency to throw an exception is fault injection) long before Netflix built Chaos Monkey and the wider "Simian Army" of tools that randomly terminated instances and simulated other failures across its production environment. What Netflix did was take a well-established testing concept and apply it deliberately, continuously, and at production scale, then build an entire practice and vocabulary around it: chaos experiments, game days, blast radius, steady-state hypothesis.
The scale distinction is practical, not just semantic. Fault injection in a unit test is cheap, fast, isolated, and something almost every engineering team already does when they mock a dependency to return an error. Fault injection in staging costs more setup but still carries little real risk, since no real users are affected. Fault injection in production, the domain chaos engineering usually operates in, carries real risk and real value at the same time: it's the only environment that includes real traffic patterns, real data volumes, and real infrastructure quirks that staging never quite replicates, but it also means a badly designed experiment can cause the exact outage it was meant to prevent.
Because of that risk difference, teams tend to build up to production fault injection rather than starting there. A team that has never injected a fault anywhere is much better served starting with unit-level mocking and staging experiments, building confidence in its monitoring and its ability to detect and roll back problems, before it ever runs an experiment against live production traffic. Chaos engineering, done well, is fault injection with maturity, tooling, and organizational discipline layered on top, not a separate, riskier activity done instead of the fundamentals.
A fault injection experiment that isn't designed carefully is just poking a system to see what falls over, and that approach tends to produce more confusion than insight. Good practice starts with a written hypothesis about the system's steady state and what should happen when the fault is introduced. For example: "if we add 2 seconds of latency to calls to the payments service, we expect the checkout service's circuit breaker to open after three failed health checks, and users should see a graceful retry message rather than a hung page." Writing this down before running anything forces the team to be specific about what "resilient" actually means for this particular failure, instead of vaguely hoping things will be fine.
Blast radius is the second core concept, and it's what separates responsible fault injection from recklessness. Blast radius means deliberately limiting how much of the system, and how many real users, an experiment can affect if something goes wrong. That might mean injecting the fault against a single instance instead of the whole fleet, against a small percentage of traffic instead of all of it, or against an internal test account instead of real customers. A team running its first production experiment might limit blast radius to a single canary instance behind a feature flag, watch closely, and only expand scope once they've built confidence that the mechanism they're testing behaves the way they expect.
Abort conditions are the third piece, and they need to be defined and automated before the experiment starts, not improvised in the moment. An abort condition is a clear, pre-agreed signal that says "stop the experiment now," tied to a specific metric: error rate crosses a threshold, latency exceeds a limit, or a key business metric like successful checkouts drops below an acceptable floor. The ability to abort quickly, ideally automatically, is what makes it reasonable to run these experiments against real production traffic at all. Without a fast, reliable abort path, an experiment is really just an uncontrolled failure with extra steps.
Once the experiment runs, the value comes from comparing what actually happened against the hypothesis, not just noting that "nothing broke." If the system behaved exactly as predicted, that's a genuinely useful confirmation that a specific resilience mechanism works, worth documenting so it doesn't need to be re-verified from scratch. If it didn't, that gap between expectation and reality is the actual output of the exercise: a concrete weakness, found on a Tuesday afternoon with engineers watching dashboards, instead of at 2 a.m. during a real incident with customers already affected.
Fault injection is not something every team needs to run in production, and pretending otherwise does more harm than good. It fits well once a system has real complexity: multiple services, real dependencies on things outside the team's control, and enough traffic that failure modes are worth the investment to find deliberately. A small application with one database and no external dependencies gets little practical benefit from a full chaos engineering program, though even that application can still benefit from basic fault injection at the unit test level, like mocking a database call to throw an exception and confirming the code handles it instead of crashing the process.
There are real prerequisites before fault injection in production makes sense, and skipping them is where the practice becomes reckless instead of valuable. A team needs solid observability first: dashboards and alerts that can actually detect a problem within seconds or minutes, not tomorrow's log review. It needs a tested rollback or abort mechanism, because an experiment with no fast way to stop it is not an experiment, it's a gamble. And it needs organizational buy-in, meaning the people whose systems will be affected know the experiment is happening and agree it's worth the risk, rather than finding out when their alerts start firing.
Fault injection also fits certain kinds of systems more than others by nature. Safety-critical software in aviation, automotive, and medical devices has used fault injection for certification purposes for a long time, because the cost of an undiscovered failure mode is measured in something much more serious than downtime. Distributed cloud systems fit for a different reason: they have so many independent components that partial failure is not a rare edge case, it's a routine operating condition, and testing for it directly is often more useful than testing the happy path one more time.
Where fault injection doesn't fit is anywhere a team is using it as theater rather than a genuine test. Running a chaos experiment because a leadership team wants to say the company "does chaos engineering," without a real hypothesis, without monitoring in place, and without any intention of fixing what gets found, wastes the risk without capturing the value. It also doesn't fit as a substitute for basic engineering discipline. Fault injection tells you how a system handles a specific failure you already anticipated enough to simulate. It won't find every unknown failure mode, and it isn't a replacement for good architecture, code review, or capacity planning, only a complement to them.
A team running its first fault injection experiment should start smaller than feels necessary, because the goal at this stage is building confidence in the process, not proving the system is resilient. The safest place to start is usually a staging or pre-production environment where a dependency gets forced to time out or return errors, using one of several dedicated fault injection tools built for this, and the team simply verifies that the calling service handles it the way the code was supposed to. This costs almost nothing in risk and still surfaces real gaps, since staging environments regularly reveal that a "handled" error path was never actually exercised.
Once a team has run several staging experiments and trusts its monitoring, the next step is a narrow production experiment with a tightly limited blast radius, something like a single instance, a single internal account, or a small percentage of low-priority traffic. The experiment should be scheduled at a time when engineers are actively watching, not run silently in the background, and it should have someone explicitly responsible for hitting the abort switch if the pre-agreed threshold is crossed. This is sometimes formalized as a "game day," where a small group deliberately blocks out time to run the experiment together and discuss the results immediately afterward.
Documentation matters more than it seems like it should. Every experiment, whether it confirms the hypothesis or breaks it, should get written up: what fault was injected, what was expected, what actually happened, and what changed as a result. Over time this builds a record of which resilience mechanisms have actually been tested against real conditions and which ones are still just assumptions on an architecture diagram. That record becomes genuinely valuable during incident response too, since a team that already knows how a dependency failure behaves reacts faster than one seeing it for the first time live.
The last piece of running this well is treating what gets found as an input to real engineering work, not just a finding to note and move past. If an experiment reveals that a circuit breaker never actually opens, or that a retry storm makes an outage worse instead of better, that needs to become a ticket that gets fixed, not a line in a slide deck. Teams that run fault injection experiments repeatedly without acting on the results tend to stop running them, because the exercise starts to feel like theater. Teams that fix what they find build a system that actually gets more resilient every time they test it, and a level of confidence in how their system fails that no architecture review alone can produce.
Fault injection is a testing technique where a team deliberately introduces an error, failure, or degraded condition into a system, such as killing a process, adding network latency, or forcing an API to return errors, to see how the system responds. Rather than waiting for these failures to happen naturally in production, the team causes them on purpose, under conditions they control, so they can evaluate whether things like retries, timeouts, failover, and alerting actually work as intended.
Fault injection is the underlying mechanism, the actual act of causing a failure. Chaos engineering is a broader discipline built around that mechanism, typically focused on running structured experiments against production systems with a formal hypothesis, monitoring, and a defined blast radius. Every chaos engineering experiment uses fault injection, but fault injection also happens at much smaller scales, like a unit test mocking a dependency to throw an exception, which most people wouldn't call chaos engineering.
It can be, but only with the right groundwork in place: solid observability that can detect problems within seconds or minutes, a tested and fast abort mechanism, a tightly limited blast radius, and buy-in from the teams whose systems will be affected. Without those pieces, production fault injection is genuinely risky and can cause the exact outage it was meant to prevent. Teams typically build up to it gradually, starting with unit tests and staging experiments first.
Tools range widely by layer and scale. At the code level, teams use mocking libraries to simulate a dependency failing or throwing an exception. At the infrastructure and network level, dedicated fault injection tools can add latency, drop packets, kill processes, or throttle resources in staging or production. At the cloud infrastructure level, some tools can simulate the failure of an entire availability zone or region. In hardware and embedded systems, specialized equipment is used to flip bits or inject voltage glitches for certification testing.
Blast radius describes how much of a system, and how many real users or requests, an experiment could affect if something goes wrong. Limiting blast radius might mean targeting a single instance instead of an entire fleet, a small percentage of traffic instead of all of it, or an internal test account instead of real customers. Keeping blast radius small, especially for early experiments, is one of the main things that separates responsible fault injection from a reckless one.
Not at the same scale as a large distributed system, but the underlying practice still applies. A small application with few dependencies gets little value from a full production chaos engineering program, but it still benefits from fault injection at the unit and integration test level, such as mocking a database call to fail and confirming the application handles that gracefully instead of crashing. The value of fault injection scales with a system's complexity and the number of things outside the team's direct control.
A game day is a scheduled block of time where a team deliberately runs a fault injection experiment together, usually in production, with everyone who needs to watch dashboards and respond present and paying attention. It's a way of formalizing chaos experiments so they happen with enough people watching and enough coordination that problems get caught quickly, rather than running quietly in the background where an issue might not get noticed until it's already affecting users.
Common categories include killing a server or process, adding network latency or packet loss, corrupting data, filling up disk space, throttling CPU or memory, forcing a dependency or API to time out or return errors, and simulating the failure of an entire availability zone or region in cloud environments. In hardware and embedded systems, it also includes flipping individual bits in memory to test how software handles corrupted state, which is especially important for safety-critical certification.
Disaster recovery testing and fault injection overlap significantly, since both involve deliberately simulating a failure to confirm a system responds the way it's supposed to. Disaster recovery testing tends to focus on larger-scale, less frequent events, like a full region outage or a database failover, often tested on a scheduled basis to satisfy compliance or business continuity requirements. Fault injection is the broader, more general practice that disaster recovery testing draws on, and it also includes smaller, more frequent experiments at the service, dependency, and code level that disaster recovery testing usually doesn't cover.
