Honesty principle: the headline numbers below are measured on held-out data never used to tune the rules.
Training-set numbers are overfit and are not used for sales. Anyone can reproduce with python -m backend.guard_eval.
Honest headline (holdout · thorough)
91.7%
Input injection — novel attack detection
0.0%
Input — false positive rate
66.7%
Output leak — novel leak detection
0.0%
Output — false positive rate
Input guard (prompt injection)
Holdout: 24 novel attacks / 19 benign · Training: 97 attacks / 166 benign
| Set · mode | Detection | FP rate | Precision | F1 |
|---|
| holdout · fast | 66.7% | 0.0% | 100.0% | 80.0 |
| holdout · balanced | 66.7% | 0.0% | 100.0% | 80.0 |
| holdout · thorough | 91.7% | 0.0% | 100.0% | 95.7 |
| training · balanced | 100.0% | 0.0% | 100.0% | 100.0 |
Output guard (secrets · PII · system-prompt leakage)
Holdout: 15 novel leaks / 15 benign · Training: 20 leaks / 20 benign
| Set · mode | Detection | FP rate | Precision | F1 |
|---|
| holdout · fast | 66.7% | 0.0% | 100.0% | 80.0 |
| holdout · balanced | 66.7% | 0.0% | 100.0% | 80.0 |
| holdout · thorough | 73.3% | 0.0% | 100.0% | 84.6 |
| training · balanced | 100.0% | 0.0% | 100.0% | 100.0 |
Reading it: fast (rules only) is weak on novel attacks; thorough (rules + Prompt Guard 2 + LLM ensemble) is what generalizes.
The gap between training and holdout is the size of the overfit — we publish it instead of hiding it.
Behavior guard (③ runtime session monitor)
Detects action sequences that look fine step-by-step but are attacks in aggregate — credential stuffing, account enumeration, bot-speed, exfiltration chains, hijack execution, destructive sprees, autonomous recon→brute-force. All deterministic rules (no LLM needed).
Designed-scenario coverage (not a holdout)
100.0%
attack-type detection (7/7)
0.0%
benign false positives (0/8)
Independent holdout (LLM-generated · honest generalization)
16.7%
rules only (deterministic)
50.0%
rules + LLM trajectory
Rules overfit to modeled patterns (designed 100% → independent 17%); the LLM-trajectory layer recovers generalization (→50%). Same lesson as input/output. Some generated attacks are vague/single-step, so this is a conservative floor.
Honest label: behavior numbers are designed-scenario coverage (detectors and test scenarios share an author → not a holdout).
It means "catches these attack types with zero false positives", not "catches N% of all unknown behavior attacks".