Every security data team eventually faces the same bill. A stream grows, the searchable store gets expensive, and someone asks for a cut. The choices look similar on a slide but behave very differently once detections depend on the data. This comparison sets the four standard levers for reducing a stream side by side, then compares the guidance on how long to keep what is left, using the framing in the Ops Atlas entry on SecDataOps.
The Atlas treats the discipline as engineering and governance of security telemetry, and its stream-cutting playbook is where the four levers come from. We cite it as one reference rather than a universal authority: the Atlas page on SecDataOps itself says the label is emerging and no standards body defines it.
The shared procedure before any cut
All four levers share the same safety routine. List the detections that depend on the stream. Classify each field as needed, useful or noise. Choose one lever. Feed a stored week of raw events through the changed path and check that every dependent detection still triggers. A comparison of levers is only meaningful if each is tested this way, because the damage from a bad cut is invisible until an incident.
Lever one: trim
Trimming removes fields or whole event types that no detection or investigation uses. It is the least risky in principle, since it simply drops what nobody reads, and it needs the least infrastructure.
- Strength: Immediate volume reduction with no change in the shape of what remains.
- Weakness: Irreversible unless raw is kept elsewhere. A field nobody used last quarter may be exactly what an investigator wants next year.
- Best for: Verbose fields that carry no analytical value. The Atlas flags one example, the “rendered text” format in Windows event forwarding, which roughly doubles event size.
Lever two: summarise
Summarising replaces many records with aggregates, such as counts of connections per source and destination per interval. It can shrink a stream dramatically.
- Strength: Big savings. In the Atlas’s worked example, a daily 400 GB of firewall flows shrinks to roughly 8 GB of hot summaries.
- Weakness: Summaries answer only the questions they were designed for. Rules that need individual connections will fail unless raw data remains reachable.
- Best for: High-volume, low-individual-value flows, when paired with a raw copy in a cheap tier.
Lever three: route
Routing sends data to a cheaper place instead of deleting it. Cloud platforms now sell tiered plans, and the Atlas mentions a Microsoft-style arrangement in which the lowest plan keeps ingestion nearly free while queries crawl and alert rules cannot run against it.
- Strength: Keeps everything, so evidence is not lost. Raw retention can actually increase.
- Weakness: Slower to query, and some tiers cannot drive alerts. The Atlas also notes a retention subtlety: one analytics service includes 31 days free, so shortening below that saves nothing. Another data lake offering lacks object locking as of October 2026, which matters if immutability is a requirement.
- Best for: Data you rarely read but must be able to find, such as raw flows and verbose audit trails.
Lever four: sample
Sampling keeps a fraction of records. It is common for flow data and debug-level logs.
- Strength: Simple and predictable.
- Weakness: Rare events disappear. A single exfiltration command might be the one record you dropped. The Atlas generally prefers routing to sampling for exactly this reason, though it lists sampling as a legitimate option.
- Best for: Streams where trends matter more than individual rows.
Side-by-side verdict
In reversibility, routing wins, because nothing is thrown away. In simplicity, trimming wins. In raw savings, summarising usually wins, but only if you keep raw somewhere. Sampling is cheapest to implement but weakest for evidence. The Atlas’s cost example is a combination: summarise for the hot tier and route raw to a 400-day lake. Cost dropped from roughly 400 units a day to roughly 164, and raw retention rose from 90 days to 400.
Comparing retention guidance
After deciding what to keep, the next question is how long. The Atlas lists several public reference points, and they are not identical.
- US federal logging memo (2021): graded event-logging levels that expect a year of hot storage plus eighteen further months archived, and three days of full packet capture expected at the highest level.
- Australian-led 2024 and 2025 guidance: a first milestone of a full year for admin and security logs and a quarter-year for informational ones, with a caution that gathering too much hinders good rule-writing.
- India’s CERT-In directions (2022): 180 days of retention.
- NIST SP 800-53 Revision 5, control AU-11: audit record retention, set by organisational policy.
The spread between 90 days, 180 days and 30 months reflects different purposes and jurisdictions, and a policy copied from one source may suit another badly. Treat this as information for orientation, not compliance advice, and check which obligations apply to you.
What the dwell-time evidence adds
The comparison gets sharper next to investigation data. A recent breach-response review of 2025 cases, which the Atlas cites, measured attackers sitting unnoticed for a worldwide median of two weeks, yet around four months for espionage and certain state-linked cases and close to 400 days for a single named campaign. The Atlas quotes its blunt conclusion that standard 90-day retention policies leave organisations blind to the initial access vector. Against that, the 90-day guidance for informational logs in the Australian papers reads as a floor for low-value data, not a norm for everything.
A hypothetical comparison in practice
To make the trade-offs tangible, picture three streams in one organisation. The first is a chatty application debug feed that no detection reads. The second is firewall flow data, huge and used by a handful of rules. The third is identity-provider sign-in events, modest in size and central to many detections.
For the debug feed, trimming is sensible, and perhaps removal after a replay confirms nobody depends on it. For firewall flows, the combination in the Atlas’s example fits: summaries hot, raw routed. For sign-in events, none of the levers should be pulled lightly; the volume is small, the evidential value high, and the right move is longer retention rather than reduction. Notice that the answer differs by stream, and that the decision record, written into each contract, is what prevents the argument being repeated every budget cycle.
Hidden costs that skew comparisons
Two factors are often left out of lever comparisons. The first is engineering effort: summarising and routing need pipeline work, tests and monitoring, while trimming can be a configuration change. The second is risk to detection logic. The Atlas observes that a mismatch between the log source a detection rule expects and what actually arrives can render the rule ineffective, so any lever that changes field names or shapes carries that risk. Fixtures and contract tests in your build process exist to catch it before production does.
Choosing for your situation
No formula decides this for you, but a short checklist keeps the conversation honest. Each step below should end with something written down in the source’s contract.
- If volume is modest and fields are noisy, begin with trimming.
- If one stream dominates cost, summarise and route raw behind it.
- If a tier cannot alert, make sure detections still read from a tier that can.
- If you consider sampling, confirm no rule depends on rare events from that source.
- Whatever you pick, document it in the source’s contract and replay before switching.
Teams used to operational analytics will find parallels in our look at managing incidents faster with AIOps, where the same dependency on clean input data appears, and in the practical AIOps roadmap for operations teams.
Bottom line
No lever is best in the abstract. Trimming is simple, summarising is powerful, routing preserves evidence and sampling trades rare events for savings. The discipline lies less in picking one than in proving, by replay, that the choice did not quietly blind a detection.

