Run the lab before you argue. That is the whole premise of this piece: every claim made about AIOps for Security — every accuracy figure, every automation percentage, every benchmark score — can be checked against arithmetic small enough to fit in a notebook, using data your own estate already produces. The lab below takes two analysts about three weeks of part-time work, needs no special tooling whatsoever, and at the end you will know more about your own alert queue than most vendor slide decks know about theirs.

Before you start: what AIOps for Security claims, and what only you can verify

The label covers anomaly detection, alert ranking, LLM-drafted investigation, plus the two parts vendors mention least — evaluation and autonomy grading. Its public claims come in three shapes: accuracy numbers produced on the vendor’s data, benchmark scores produced on shared suites, and automation promises produced on nothing in particular. The first kind transfers badly between estates, because attack rarity differs everywhere, and a number that dazzles at one prevalence collapses at another. The second kind currently sits far below brochure language — agents completed 25.2% of compliance-and-security scenarios in one independent suite, and landed between roughly a fifth and a half on malware and threat-intel work in another, against near-zero random baselines. The third kind is unfalsifiable until you run the lab, so run the lab.

The seven steps of the base-rate lab

1. Export the dispositions. Pull three months of alert decisions with their outcomes attached. Without labelled history nothing downstream is possible, and this single export usually costs the most organisational effort of the whole exercise — expect arguments about who owns the data, and win them politely.
2. Choose two categories. One high-volume, one high-risk, and no others for this first pass. Refuse blended reporting from the first minute: one real deployment’s cloud-IAM category carried 42.3% misses, all hidden behind a calm 9.1% phishing line, and only a category split exposed it.
3. Compute the base rate, then translate error rates into workload. With roughly fifty genuinely malicious items hiding in fifty million daily events, prevalence sits near one in a million. Now test candidate error rates against it: a detector erring once in ten thousand delivers precision near 0.9% — around 112 queue items reviewed per genuine catch — while an error rate of one in a million lifts precision to 47.4%. Two lines of arithmetic, one hiring plan, and a permanent answer to the question of what “better detection” is actually worth.
4. Blind-label a holdout. Have analysts label a recent sample with every model verdict hidden. Whoever labels must never see what the candidate tool concluded, and the tool’s operators must never see the labels before scoring. Any shortcut here corrupts everything after it, quietly and completely.
5. Compute the miss bound, not the miss rate. A clean run over n cases bounds the true miss rate near three-over-n; a run with one observed miss and a 2% ceiling needs 236 samples to prove anything, and two observed misses need 313. One published triage study makes the difference vivid: 2.8% raw misses stretched to a near-8.6% worst case on seventy-one positives, demolishing a 2% gate on statistics alone.
6. Assign the gate per action class. Recommendation earns its place with a floor of 200 labelled alerts including 30-plus true positives. Unaided closure needs misses inside 2% across 150-plus true positives, plus canaries in the queue, a blind holdout and one-click reopening. Reversible actions need false actions below 1% across 300-plus opportunities, rehearsed rollback and a throttle on volume. Irreversible actions are refused, categorically and permanently.
7. Install canaries and the withdrawal rule. Fifty planted cases per quarter, indistinguishable from live traffic, with standing instructions that three or more wrongly closed pulls the autonomy tier until results recover. Then diary the next lab date, because attacker behaviour keeps moving and every proof carries an expiry, whether or not anyone bothers to write it down.

Reading your own results without fooling yourself

Three readings in particular trip up first-timers. A high precision figure means nothing until the prevalence it was computed on travels with it. A spotless quarter proves less than it feels like it does, for the sample-size reasons in step five. And a category that never fails deserves suspicion rather than pride — models trained on thin data fail in exactly the categories nobody watches, which is why step two insists on the split. If a candidate agent also reads attacker-controlled text through outbound channels, add a fourth reading: treat everything it fetches as potential instructions, because prompt injection has no complete input-side fix and the mitigations that hold are deterministic guards and least privilege. The UK’s NCSC made the point formally in late 2025 — injection is not an ordinary input-validation bug — and teams that internalise the warning design smaller, less exposed agents from the start.

Four lab mistakes that invalidate everything downstream

The research record is specific about how these experiments fail, and each failure mode has bitten published work you may have cited. Watch for all four in your own lab:

• Biased sampling. A review of thirty high-profile security ML papers recorded sampling defects in nine of every ten — if your holdout is drawn only from Tuesdays, or only from one sender domain, your result describes your sample, not your queue, and it will betray you the week the traffic pattern shifts.
• Label peeking. Test information leaked backwards into training or tuning in about three-quarters of the same papers. The wall between labellers and the model’s outputs exists to stop exactly this, and every shortcut through it shows up later as a score you cannot reproduce.
• Category pooling. Covered in step two, worth repeating: blended rates hide the failing category, and the failing category is usually the one that hurts.
• Trusting a flawless score. Scores close to a flawless 0.99 F1 appeared throughout 2019-2022 detector research, and fell apart as soon as evaluation bias was squeezed out. In this field, perfection is a symptom.

From lab to gate: AIOps for Security in production

The lab earns a decision; the gate keeps that decision honest afterwards. Production means the register stays current — tier, evidence, dates, owner per action class — and the monthly per-category table keeps publishing even when it is boring, especially when it is boring, because boredom is precisely when categories quietly begin to fail. The modelling background overlaps pleasantly with work on quantum approaches to machine learning, and the agent-side caution maps onto what AI agent architectures already teach about scoping tools and permissions; the cryptographic habit of never trusting unverified input, familiar from quantum-safe communication basics, is the same reflex that keeps an investigating agent out of an attacker’s script, and adopting it early costs nothing. The complete reference for everything above — thresholds, sampling arithmetic, worked evaluations, the failure catalogue — is the Ops Atlas page devoted to AIOps for Security. Three weeks of lab work buys you the only opinion about your alert queue that really counts, and it is the one opinion written in your own numbers rather than somebody’s marketing quarter.