AI wrote this software. An AI agent went looking for the holes.

Wintertrace is written with heavy AI assistance. Not as a side note — as the method. Most of the code in this repository was drafted by a model, reviewed by one person, and shipped.

That is worth saying out loud rather than burying, because it changes what you should reasonably want from the project. Code written that way is fluent. It compiles, it reads well, it has tests. What it does not automatically have is the paranoid streak of somebody who has spent fifteen years watching applications get broken into. The plausible-looking authorisation check that never actually runs is exactly the kind of thing a language model writes without hesitating.

So on 3 August 2026 the software was handed to an adversarial agent and told to break it. Twice, from two different angles. This article is the full record: what the tool is, how it was pointed at the code, everything it reported, everything it got wrong, and the things it walked straight past that a person found a fortnight later by installing the release on ordinary shared hosting.

Everything below is fixed in version 1.2.0, released today.

What Strix is

Strix is an open-source penetration-testing agent (Apache-2.0). It runs inside a Docker sandbox, drives a real browser and an intercepting proxy, and works the target the way a human tester would: map the surface, form a hypothesis, try to prove it, discard it when the proof fails.

The part that matters is that it is not one agent. A root agent reads the target, then spawns specialists and hands each one a narrow brief. The source-code run created sixteen of them. Their names are a fair description of how the work was divided:

• Discovery Coordinator, Static Triage Lead, Runtime Prep Lead
• Installer Surface Lead → Installer Route Validation Agent → Installer Route Reporting Agent
• Module Surface Lead → Module API Auth Default Validation Agent
• Storage Lead → Storage Auth Validation Agent → Storage Auth Reporting Agent
• Disabled Module Boot Validation Agent → two separate reporters
• Composer Dependency CVE Lead

Note the shape: a lead agent finds a lead, a separate validation agent has to reproduce it, and only then does a reporting agent write it up. A hypothesis that cannot be reproduced is dropped. That separation is the whole reason the output is usable, and it is also where the tool’s one visible weakness shows up — more on that below.

Each agent loads skill files for the vulnerability class it is chasing: idor, ssrf, rce, path_traversal_lfi_rfi, header_injection, business_logic, broken_function_level_authorization, insecure_file_uploads, dependency_cve_scanning, plus Semgrep and a source-aware static-analysis skill for the white-box run, and xss, sql_injection, csrf and mass_assignment for the run against the live instance. Thirty tools were available per agent; the ceiling was 500 turns.

The model underneath was GPT-5.4. Sandbox image ghcr.io/usestrix/strix-sandbox:1.1.0.

Two runs, two vantage points

Run one — white-box, against the source. Target: the unpacked 1.1.9 release directory. Standard scan mode. Strix read the routes, the middleware and the trust boundaries, ran static analysis and secret scanning over the tree, then stood the Laravel application up locally so it could prove its findings over real HTTP instead of asserting them from the code. It ran from 16:37 to 18:09 UTC — one hour thirty-one minutes.

The reported constraint is worth repeating, because it is the sort of thing a vendor report usually omits: the delivered artefact had an inconsistent install state, with a populated database but a missing installer lock file. Strix said so, worked around it, and flagged that it had done so.

Run two — authenticated black-box, against a live instance. Target: a throwaway demo instance of the kind any visitor can spin up, which deletes itself after 48 hours. Deep scan mode. Strix was given one admin login and explicit authorisation to be destructive — create, mutate and delete records, trigger workflows, whatever it took. No source code this time. Twelve agents, from 18:37 to 19:10 UTC — thirty-two minutes.

It created its own low-privilege users and seeded its own business records when the demo dataset turned out too sparse to exercise the interesting workflows. That is a small thing, but it is the difference between a scanner and a tester.

What it cost

The numbers, because “we ran an AI security test” means nothing without them:

	Source run	Demo run	Total
Duration	1 h 31 min	32 min	2 h 3 min
Model requests	608	462	1,070
Input tokens	43,137,738	29,433,310	72,571,048
— of which cached	41,720,576	28,466,688	70,187,264
Output tokens	205,405	169,257	374,662
— of which reasoning	99,192	61,627	160,819
Total tokens	43,343,143	29,602,567	72,945,710

Seventy-three million tokens for two hours of work. The figure that makes it affordable is the second row: 96.7% of the input was served from cache. An agent loop re-sends its accumulated context on every turn, so the raw input count balloons while the genuinely new material stays small. Output — the tokens that represent actual decisions — is a rounding error at 0.5% of the total.

Both runs went through a consumer subscription rather than metered API billing.

What it found

Thirteen vulnerability records across the two runs. After removing a duplicate and collapsing six advisories in one dependency into one dependency problem, that is seven distinct findings.

Setup steps stayed reachable after installation (High). On a fully installed system, parts of the setup wizard were still callable without logging in. An unauthenticated request could rewrite the outgoing mail configuration and trigger a scheduled-task run. This was the most serious finding: externally reachable, no credentials required, and it changes how the application behaves. The middleware exempted the mail, cron and completion routes from the install-state check before checking whether the application was installed.

Privacy-notice text could carry active content (High, CVSS 8.1). Privacy notices are written in Markdown and rendered in four places. The renderer accepted link targets that browsers treat as script, so a javascript: link survived saving, came back through the preview endpoint, and was rendered as a live anchor on the driver onboarding screen. An administrator account was needed to plant it — but administrators are exactly who a phishing attempt goes after.

A disabled module kept running (Medium). Switching a module off was only remembered for the current request. After the next restart the module booted again, registered its routes again, and carried on — while the admin interface still showed it as disabled. Strix reported this twice through two independent agents, which is the duplicate.

Job photos were reachable by address alone (Low). Uploaded photos sat in a publicly served folder. Anyone who knew or guessed a path got the file with no authentication — including annotated images carrying address and location detail. Low severity only because you have to know the path first.

A manual job could be booked onto a non-driver (Low). The check consulted a field that had not been maintained since roles were reworked and that defaulted to “driver” for everybody. Parameter tampering could therefore record jobs and shifts against accounts that are not drivers, polluting dispatch and reporting data.

The weather-service contact address accepted line breaks (Low). That value is sent as part of an outgoing HTTP header, where line breaks have structural meaning. As a side effect, an address that had picked up a line break made weather retrieval fail silently.

Six published CVEs in the PDF library (Low to Medium). The lockfile pinned dompdf/dompdf v3.1.5. Strix confirmed six advisories against that version from dependency data and — the part that makes it a finding rather than noise — traced the package into the live PDF rendering path rather than dismissing it as a transitive dependency.

What it got wrong

Three things, and they are as instructive as the findings.

It reported the same bug twice. Two agents chased the disabled-module problem independently and both filed a report. The final report caught this itself and said the two should be treated as one root cause — but the raw output has thirteen records where there are seven problems, and anyone quoting the record count would have been quoting a number that means nothing.

Six CVEs is one problem. Counting each dompdf advisory separately inflates the finding count sixfold. One composer update closes all six.

It ruled out something that was real. A module API authentication default looked wrong during triage, so a validation agent re-tested it over real HTTP — and could not confirm it. The report says so plainly: “A suspected module API default-auth weakness did not hold up under independent real-HTTP validation and was not treated as a finding.” The right decision by the tool’s own rules; an unproven hypothesis should be dropped.

It was fixed in 1.2.0 anyway, because reading the code afterwards made the problem obvious: the browser-session shortcut into the module API, meant for administrators testing a module, admitted every logged-in account including drivers. The validation agent’s local conditions simply did not surface it.

That is the honest shape of this tool. Its discipline about proof is what makes the confirmed findings trustworthy, and the same discipline is what let a real issue slip out of the report.

What it did not find

Nothing at all about deployment.

Both runs were pointed at a working application — one unpacked in a prepared directory, one already running on a demo host. Neither installed the software the way an operator does. So neither noticed:

• The driver app was never actually available offline. The service worker was served from a subdirectory, and a service worker can only take charge of addresses below where it is served from — so it took charge of nothing a driver ever visits. Every caching rule for driver pages, fonts and module screens had been dead code since it was written.
• A fresh installation on a real domain refused every request, the setup wizard included. The host-header check added in 1.1.6 trusted only localhost while no configuration file existed yet. There was no way in at all.
• The release package shipped a database file. A fresh installation connected to it, found an account, and concluded it was already installed.
• The driver dashboard was also served at /dashboard, an address that looked identical but carried none of the offline machinery. Anything a driver did there without network was lost without a trace.

Those four were found by one person installing 1.2.0 on ordinary shared hosting and using it. No test suite caught them; no agent was looking. Two hours of adversarial AI is worth a great deal and it is not worth an afternoon on a real host.

What 1.2.0 changes

All seven findings are fixed, plus the one Strix ruled out. Each has a regression test that fails if the behaviour returns.

Setup routes now refuse to work once the system is installed, and “installed” is decided consistently everywhere — a finished installation or an existing user account, so deleting the lock file changes nothing. All four Markdown paths go through one renderer that permits ordinary link schemes only. Module enable state is read from the database at boot, and a module without a database entry does not boot at all.

Photos moved off public storage entirely. They are stored outside the web root and delivered through a checked route: administrators see everything, drivers see the jobs they drove themselves, customers see their own jobs and only while photo sharing is enabled for them. Links are signed and expire after 24 hours, so a forwarded address stops working. Files already on disk migrate automatically in the background and stay available throughout.

The manual-job check now consults the real role assignment. Line breaks are stripped from the weather contact address on save and again before the request. The module API session shortcut is administrators only; access tokens are unaffected.

dompdf is on 3.1.6, closing all six advisories, and the HTTP library was updated alongside it. composer audit is clean. Beyond that, PDF generation no longer fetches remote content at all — nothing in the reports needs it, since photos are embedded directly, the logo is read from disk and the GPS track is drawn inline. It was switched on regardless, which is what made several of those advisories applicable in the first place.

The modules had to move too

Locking photos down broke things downstream, which is the honest cost of a fix like this. Three modules shipped their own updates:

• Telegram 1.0.5 — the module used to hand Telegram a photo URL, and Telegram’s servers fetched it themselves, session-less. Under the new default-deny rule that request is correctly refused, so photo messages stopped arriving. The module now uploads the image bytes directly. Install it together with 1.2.0. Text and location messages were never affected.
• Documents 1.0.10 — registers its own storage prefix with the core’s new media registry. Until a module does that, its files are served to nobody and migrated for nobody; documents uploaded before the core update stay on the public path.
• Warehouse 1.3.0 — adopts the core’s new formatting service, so stock quantities, currency amounts and timestamps follow the installation’s region settings instead of German defaults, along with driver offline capture and an interface pass.

If you build your own modules, the module documentation covers the registry and the three breaking changes 1.2.0 introduces.

What this does and does not prove

It proves that two hours of adversarial testing found seven real problems in software that had passed its own test suite, and that all seven are now closed with tests behind them.

It does not prove the software is secure. A penetration test finds what it looks for in the time it has. This one looked hard at authorisation, injection, business logic and dependencies, and it did not look at deployment at all — which is precisely where the next four problems turned out to be. Nobody certified anything, and no such claim is being made here.

What it does say about method is narrower and more useful: code written with AI assistance needs an adversarial pass that the same process did not produce, and the pass is now cheap enough that there is no excuse for skipping it. Two hours, seventy-three million tokens, seven findings. The full source is public, so you can check the fixes rather than take this article’s word for them.

The update is available in-app, or as a download for a fresh installation. Nothing needs configuring afterwards — but open the driver app once with network available, so the service worker can finally install itself.