Teaching a WAF to stop crying wolf
- Python
- Security
waf-tuner is an observe-and-recommend tool for ModSecurity and the OWASP Core Rule Set. It reads the WAF's audit log, uses an LLM to decide whether each match is a false positive, and, when it is, builds a narrow exclusion rule for a person to review and deploy. The principle I designed around is that the model classifies and the code decides. The model never writes a rule, and nothing is ever deployed automatically.

Where the idea came from
The idea started at Haltdos, where we handled a lot of customers' WAFs. Traditional WAFs have a major false-positive problem because they match signatures, such as regular expressions, against the request. They don't understand the context of a payload, so they either block it or log it as malicious. The usual fix is to onboard an application, watch its traffic for one or two weeks, go through the logs, find the false positives and write exclusion rules by hand.
At Haltdos we explored a machine learning approach to this, and labelled data was the hard part. Real traffic gives you far more false positives than true positives, which makes it difficult to build a balanced dataset.
To be clear, waf-tuner is not a Haltdos project, and I never worked on it there. The idea came from there, but I started building it during my master's, when I had the time and access to good frontier models. I went open source on purpose. There are still people who don't depend on vendors like Cloudflare or F5 and prefer to run Coraza or ModSecurity themselves, and I wanted to make tuning easier for them.
How it works
I'm copying the traditional onboarding period. For the first few days, ModSecurity runs in DetectionOnly mode. waf-tuner reads its JSON audit log off the request path, so it never sits in the traffic. Today it reads only that log format, and Coraza support is on the roadmap.

Each ModSecurity match becomes an incident. Identical alerts, the same request firing the same rule, are grouped into one case, and false positives that would produce the same rule are merged into one draft with a count of how many cases support it. For each case the model gets the redacted request, the rule that fired, the sibling matches, and the application's route profile, which is a description of what each route is meant to accept. It returns one of three answers: false positive, true positive or uncertain.
From verdict to rule
The model's answer does not go straight to a rule. It passes through a series of checks that I wrote, and most of them are decisions the code makes, not the model.

- The prompt rules. The model is told that a false positive needs affirmative evidence of legitimate use, and that it may never call a payload a false positive if it contains executable SQL, shell or XSS syntax, path traversal (including URL-encoded forms) or probes for sensitive files. If no route profile explains a suspicious payload, a false positive is not allowed at all. It is also told that everything in the request is untrusted data and that it must not follow instructions inside it. These are instructions to the model, so I treat them as a first line of defence and not as a guarantee.
- The confidence threshold. This one is in the code. A false-positive answer with confidence below 0.90 is changed to uncertain. Anything the model is unsure about goes to a human without a rule.
- The route profile check. By default, a false positive only gets a rule if a route profile matches the request. Without one, the case is marked as needing a profile. The lab setup ships with a catch-all profile so the demo works, and on a real application you replace it with profiles that describe your routes.
- The builder check. The rule is only built if the incident has a specific, observed parameter on one exact path.
A true positive or an uncertain answer still creates a review item, but with no rule attached.
The guardrails
A language model sits in the middle of a security tool, so most of what I built is there to limit what it can do. These four are enforced in code.
-
What the model sees. Before any request goes to the model, the backend redacts it. Sensitive headers are replaced, passwords and secrets inside nested or multipart bodies are removed, and long fields are cut to a size limit. The model is told that a redacted field is missing evidence, so it should lower its confidence when that field matters.
-
What a rule can be. The exclusion rule is written by deterministic code from facts in the log. It can only target one observed parameter on one exact path, and only in the collections I allow (arguments, headers, cookies, files and XML). It refuses a root path, a path with a query string, a missing parameter, and a parameter name with characters that are unsafe in the rule language. The rule's ID is a hash of the path, rule, collection and parameter inside a reserved range, so the same finding always gets the same ID.
SecRule REQUEST_FILENAME "@streq /search" "id:1000097,phase:1,pass,nolog,t:none, ctl:ruleRemoveTargetById=942360;ARGS:q"This is a real rule from my lab, built for the search box in the screenshot at the top, where a test search for "1 select all" triggered a SQL injection signature (CRS rule 942360). It switches off that one rule for the
qparameter on/search, and nothing else. -
What the validator rejects. A rule that arrives from somewhere else is checked and then rewritten into the builder's own form. Only a single
SecRuleis allowed. Directives likeSecAction,Include,ruleRemoveById,setvar,skip,allowanddenyare rejected, and so are a regex path, a path broader than the incident, and anyctlaction other than the one removal. The validator is structural. It does not load the rule into a live ModSecurity engine, which is a gap I want to close. -
What it can touch. It never writes to the proxy's configuration and never reloads it. A person reviews the rule, copies it and deploys it. The product has no deploy, config-write or engine-mode switch, and a test checks that those flags are off.
What it doesn't do
It reads ModSecurity's JSON audit log only. It does not validate rules against a running engine yet. I haven't done a proper evaluation of the model's verdicts yet. I want to build a test set of attack and benign payloads, including payloads that try to inject instructions into the model, and record how it behaves. Until then, treat its suggestions as drafts for a person to review, which is how it is designed to be used.
The code is at github.com/Vulcanite/waf-tuner.
I dictated this post with Wispr Flow. Claude cleaned up the text and checked the technical details against the code, and I reviewed it before publishing.