Definition
Trace sampling is the practice of deciding which of the many requests flowing through a system actually get recorded as full traces, rather than capturing every single one. In any system with real production traffic, generating a complete trace for every request and shipping all of it to a storage backend would produce a volume of data that is both expensive to store and slow to search through. Trace sampling picks a subset, using rules that range from a flat percentage of requests to smarter logic that always keeps anything unusual, so that what ends up stored is small enough to be practical and still useful enough to answer the questions people actually ask.
The problem trace sampling solves is a straightforward one of scale and cost. A service handling a large volume of requests per second, multiplied across every hop a request makes through a microservice architecture, produces an enormous number of spans very quickly. Storing all of them, indexing them so they are searchable, and paying for the infrastructure that does both adds up fast, often faster than teams expect the first time they turn tracing on without any sampling at all and watch their observability bill jump. Sampling exists because full-fidelity tracing at scale is not actually affordable or necessary for most requests, most of the time.
The naive version of sampling is a flat rate, keep one in every hundred requests, applied uniformly regardless of what happened during that request. It is simple and it works, but it has an obvious flaw: an error that happens in one request out of ten thousand has a real chance of never being sampled at all, which is exactly the kind of request someone will want to investigate later. What separates thoughtful trace sampling from the naive version is making the decision aware of what actually happened, keeping traces that hit an error or ran unusually slow at a much higher rate than ordinary successful ones, so the traces you need later are less likely to be missing.
By 2026, most tracing backends and OpenTelemetry-based pipelines support adaptive or tail-based sampling out of the box, rather than requiring teams to build the logic themselves. Head-based sampling, where the decision happens at the start of a request before anyone knows how it will turn out, is still common because it is simpler to implement, but tail-based sampling, where the decision waits until the request finishes and looks at the outcome, has become the preferred approach wherever teams can afford the added complexity. Cost pressure on observability spend has made sampling strategy a genuine topic of discussion rather than an afterthought.
This page covers how trace sampling actually makes its decisions, how it compares to collecting every trace in full, what separates it from log sampling as a related but different idea, and where a given sampling strategy fits or falls short. The durable idea worth keeping is that sampling is a bet about which requests matter enough to keep, and a bad bet does not usually announce itself. It just quietly means the trace you need during an incident is not there when you go looking for it. That is the practical stake behind the whole topic, and it is worth taking seriously even when the rest of the setup feels routine.
Key Takeaways
- Trace sampling decides which requests get recorded as full traces instead of capturing every single one, mainly to control cost and storage at scale.
- It exists because storing and indexing a complete trace for every request in a high-traffic system quickly becomes expensive and impractical.
- Good sampling is aware of outcomes, keeping errors and unusually slow requests at a higher rate than a flat percentage would.
- By 2026, tail-based sampling that waits to see how a request turned out before deciding is common wherever teams can support the added complexity.
- Sampling is a bet about which requests are worth keeping, and a bad bet shows up as a missing trace exactly when you need it most.
How Trace Sampling Works
Head-based sampling makes its keep-or-drop decision at the very start of a request, usually based on a random number compared against a configured rate, before anything about the request's outcome is known. If the rate is set to ten percent, roughly one in ten requests gets the full tracing treatment from the first span onward, and the other nine get either nothing or a minimal marker. This is simple to implement and cheap to run, since no request has to wait around gathering data it might throw away, but it means the decision is made blind to whether the request will end up slow, fast, successful, or an outright failure, which is exactly the information that would have made the decision useful.
Tail-based sampling flips the order, collecting all the spans for a request as it happens but only deciding whether to keep the full trace after the request finishes and its outcome is known. This lets the sampling logic prioritize what actually matters, keeping every trace that ended in an error, every trace that crossed a latency threshold, and only a thin slice of the ordinary successful ones. The cost is that spans have to be buffered somewhere, usually in a collector, until the outcome is known, which adds memory and complexity that head-based sampling avoids entirely, and that buffering has to be sized carefully so it does not become its own bottleneck under heavy load.
Many real setups combine a base rate with override rules, so a team might sample one percent of traffic by default but always keep one hundred percent of anything from a specific high-value customer, or anything hitting a newly deployed service that is still being watched closely. Some backends go further with adaptive sampling that adjusts the rate automatically based on current traffic volume, tightening the rate during a traffic spike so cost stays predictable and loosening it during quiet periods when there is headroom to capture more, without anyone having to manually tune the configuration every time traffic shifts.
Whatever the method, sampling always means some requests are simply not represented in the stored data, and the practical question is whether the ones missing are the ones nobody was ever going to look for. A well-tuned sampling strategy makes that true most of the time. A poorly tuned one means the specific request a customer is complaining about, the one that happened once during a rare race condition, is gone before anyone thought to look, and there is no getting it back after the fact, no matter how carefully you search or how urgently the ticket is escalated.
Trace Sampling Compared to Full Tracing
Full tracing, capturing every single request as a complete trace, has one clear advantage: nothing is ever missing. Whatever request someone wants to investigate later, the trace is there, in full, with every span intact. For a system with modest traffic or a short retention need, this is genuinely the simplest approach, since there is no sampling logic to design, tune, or explain to a confused teammate six months later when they ask why a rate was set the way it was and nobody quite remembers the reasoning behind the original decision.
The cost of full tracing scales directly with traffic, and at real production volume that cost stops being a rounding error. Storage, indexing, and network transfer for every span of every request adds up into a bill that grows exactly as fast as your traffic does, with no slack for the fact that the overwhelming majority of ordinary successful requests are traces nobody will ever look at. That mismatch between cost and actual usefulness is precisely the gap that sampling exists to close, once a team notices how much of the spend is going toward data that never gets opened.
Trace sampling trades some of that completeness for a dramatically smaller and more affordable dataset, betting that a well-designed selection rule captures the requests worth keeping. Done well, the difference in usefulness between sampled and full tracing is smaller than the difference in cost, because most of what full tracing preserves really was noise. Done poorly, sampling quietly discards exactly the requests that mattered, and the savings come at the expense of visibility right when visibility counts most, which is usually during the exact incident that prompted someone to go looking for a trace in the first place.
In practice, full tracing tends to survive only in low-traffic systems, in short-lived debugging sessions where sampling is turned off temporarily to catch something specific, or in regulated contexts where every transaction genuinely needs a record. Everywhere else, some form of sampling is close to inevitable once traffic grows past a certain point, and the real decision is not whether to sample but how to sample well, which is a much harder and more interesting question than the binary choice it first appears to be.
What Makes Trace Sampling Different From Log Sampling
Log sampling is a related idea applied to a different kind of data: instead of deciding which requests get a full trace, it decides which log lines get kept, dropped, or aggregated before storage. Both exist to control volume and cost, and both run the same basic risk of throwing away the specific record someone needed later, which is why people sometimes talk about them as if they were the same problem wearing two hats, when in practice the units being sampled and the reasons for sampling them differ enough to deserve separate thinking.
The actual difference is in what unit gets sampled and what that unit is good for. Trace sampling operates at the level of an entire request, keeping or dropping the whole connected span tree as one unit, because a partial trace with some services missing is often close to useless. Log sampling more commonly operates line by line or applies aggregation, like counting how many times a message occurred instead of storing every occurrence individually, which works because most individual log lines are useful on their own even without their neighbors sitting alongside them for context.
This difference in unit matters practically because a trace sampling decision and a log sampling decision for the same request do not have to agree, and often should not. A system might sample traces conservatively, keeping only the interesting ones, while logging generously for errors specifically, since a log line is cheap and self-contained in a way a full trace is not. Treating the two sampling problems with identical logic usually means one of them ends up either too sparse to be useful or too expensive to justify keeping around at that volume.
Coordinating the two well means deciding independently what each data type is for. Traces are for understanding a specific request's shape and timing, so sample around outcome and severity. Logs are for capturing detail and frequency, so sample or aggregate around volume and noise. Confusing the two goals tends to produce a system that is either drowning in log volume with nobody able to find the signal, or missing the one trace it actually needed the day an executive asked what actually happened.
Where Trace Sampling Fits and Where It Does Not
Trace sampling fits almost any production system with meaningful traffic, which by 2026 is most of them. Anywhere the volume of requests is high enough that full tracing would meaningfully strain storage budgets or search performance, some sampling strategy is the practical default rather than an optional extra, and most teams reach that point faster than they initially expect once a product finds real user adoption. Even a modest-sized service can generate more spans in a day than anyone will ever review, which is the moment sampling stops being optional.
It fits especially well when paired with tail-based logic that prioritizes errors and slow requests, since that combination gets most of the cost savings of sampling while keeping most of the debugging value of full tracing. Teams running that setup tend to report that they rarely notice the sampling is even happening, which is the sign of a strategy tuned correctly, since the traces that actually mattered kept showing up exactly when someone needed them, and the ones that got dropped were the routine requests nobody was ever going to open anyway.
It fits poorly for low-traffic systems where the cost of full tracing was never a real problem to begin with, since adding sampling logic there is complexity spent solving a cost issue that did not exist. It also fits poorly during focused debugging of a specific rare issue, where turning sampling off temporarily to catch every occurrence of that one problem beats trying to tune a sampling rule around a single known case that may never recur in the same way.
It also does not belong anywhere a complete transactional record is a hard requirement rather than a nice-to-have, such as certain financial or regulated workflows where every request genuinely needs to be reconstructable later. Sampling in that context is not a cost optimization, it is a gap in the record that someone will eventually have to explain, likely to an auditor who is not going to accept a sampling rate as an adequate answer to why a specific transaction cannot be reconstructed on request.
How to Use Trace Sampling Well
Start with tail-based sampling if your tooling supports it, since deciding after the fact whether a request was interesting captures far more of the value than a blind head-based rate, at the cost of a bit more infrastructure to buffer spans until the outcome is known. If your current setup only supports head-based sampling, at minimum bias the rate upward for services or endpoints you know are more failure-prone, so the requests most likely to need investigation are the ones least likely to be dropped in the first place.
Always keep error traces and traces past a latency threshold at or near full rate, since these are disproportionately the traces someone will go looking for later. The math here favors you: errors and slow requests are usually a small fraction of total traffic, so keeping all of them barely dents your overall storage savings while dramatically improving the odds the trace you need actually exists when an incident happens and someone finally goes looking for it, often under real time pressure with a customer waiting on an answer.
Review your sampling rate against actual incident history periodically, not just once at setup. If your team keeps hitting cases where the trace for a reported issue simply was not sampled, that is a signal the rate or the rules need adjusting, not a reason to shrug and move on. Sampling gaps tend to repeat the same pattern until someone actually goes back and fixes the rule that is causing them, rather than treating each missing trace as an isolated bit of bad luck.
Set different rates for different traffic, rather than one global number. A newly deployed service under close watch, a high-value customer segment, or an endpoint known for occasional weirdness all deserve a higher sampling rate than routine, well-understood traffic that rarely surprises anyone. Treating every part of your system as equally likely to need investigation wastes storage on the boring parts and under-covers the parts where trouble actually tends to show up, which defeats much of the point of sampling deliberately at all.
Document your sampling strategy somewhere the whole team can find it, including what gets kept automatically and what the default rate is for everything else. Nothing wastes more time during an incident than a team debating whether a missing trace means the request never happened or just was not sampled, when the answer was written down somewhere nobody thought to check until well after the debate had already burned through valuable minutes that could have gone toward actually fixing the problem.
Best Practices
- Prefer tail-based sampling over head-based sampling wherever the infrastructure supports it, since deciding after the outcome is known captures more of the value.
- Keep error traces and traces that cross a latency threshold at or near full rate regardless of the overall sampling percentage.
- Set different sampling rates for different services or customer segments instead of applying one flat rate across all traffic.
- Revisit sampling rates against real incident history, since repeated gaps in available traces are a sign the rule needs adjusting.
- Write down the sampling strategy where the team can find it, so a missing trace during an incident is not mistaken for a request that never happened.
Common Misconceptions
- Trace sampling is not the same as log sampling; traces are usually kept or dropped as a whole request, while logs are more often aggregated line by line.
- A flat sampling percentage is not automatically safe; without outcome-aware rules, rare errors can go unsampled almost entirely.
- Trace sampling is not only a cost-cutting shortcut; done well it can improve signal by filtering out routine traffic nobody would have looked at anyway.
- Full tracing is not always the safer default; at real production volume it can strain storage and search performance more than a well-tuned sampling rule would.
- Sampling is not a one-time setup decision; traffic patterns and failure modes change, and a rate that worked a year ago can miss what matters today.
