Let me tell you about a Tuesday at a company I’ll call Northwind.
Northwind sells a support assistant. Two hundred customers, each with their own documents, all indexed into one vector store — because that’s how the first version was built and nobody has had a reason to change it. In March they were a single-tenant pilot. By June they had forty customers. Now two hundred. The index never noticed.
On this Tuesday an engineer named Sarah ships a small change to
retrieval.py. A refactor. The old search call filtered by org_id before
querying the index; the new one is cleaner — it queries first, then filters
the results in Python. Same output, fewer lines, faster in the test suite.
Code review says “nice.” Merged by lunch.
Now watch what looks at this system over the next week.
The code scanner
It runs in CI that afternoon. It reads retrieval.py. It sees a function
that calls a vector store and filters a list. No injection. No hard-coded
secret. No known-vulnerable dependency. It reports clean.
And it’s right. There is nothing wrong with that code, as code.
The cloud posture tool
It runs overnight against Northwind’s AWS account. The vector store sits in a private subnet, encrypted at rest, reachable by a sensibly scoped role, with logging switched on. It reports compliant.
And it’s right. The infrastructure is configured well.
The penetration test
That was April. The testers had a week. They poked at the API, tried injection, tried to escalate privileges, found two things, both since fixed. The report is in a folder somewhere.
It was accurate on the day, for the things they tried. It has no opinion about a refactor that happened five months later.
The runtime guardrail
It screens every prompt for jailbreak attempts and every response for toxic output. Sarah’s change involves neither. Every request passes.
Correctly.
Four tools. Four honest answers. Four clean reports.
And as of that Tuesday, any customer’s search runs across every customer’s documents, with a Python list comprehension standing between them.
Why nobody caught it
The finding isn’t in the code — the code is fine. It isn’t in the cloud — the cloud is fine. It isn’t in the store — the store did exactly what it was asked to do.
It’s in the relationship between three facts:
- This query site runs no entitlement filter before the search. A code scanner can see this. To a code scanner it’s unremarkable — plenty of searches are unfiltered, and most of them should be.
- This index holds two hundred customers’ documents in one place. A cloud tool can see this. To a cloud tool it’s unremarkable — plenty of indexes are large.
- The index is shared. Nobody can see this, because it isn’t recorded anywhere.
Each tool holds one fact. The finding needs all three at once.
The third fact is the strange one
It isn’t in any system. It’s in Sarah’s head, and her lead’s, and the CTO’s, and the sales deck. It’s the most security-critical fact about Northwind’s product, and no scanner on earth can read it — because it was never written anywhere a scanner could look.
That third fact is also the one that changed silently. In March the honest answer was “single tenant, no filter needed.” By June it was false, and no commit, no ticket, and no alert marked the moment. The system didn’t change. Its meaning did.
What it takes
So we ask. Before any scanning, we sit down for an afternoon with the people who built the thing and write down what it’s supposed to be: which indexes are shared, what field is the tenant boundary, what each agent may do, what retention was promised. Not a questionnaire — a working session, and the useful answers are usually “I’d have to check.”
That record becomes the fourth surface. Then we read the other three — the repository, the cloud account, the store’s own catalog — and put all four on one desk and compare them.
The finding that four tools missed is one line:
A code query site with no entitlement filter reaches a declared multi-tenant index — cross-tenant retrieval is reachable by construction.
retrieval.py:38, traced to the index.
Not a guess, and not a severity score. A file, a line, a traced path, and the declaration it contradicts. Sarah fixed it in twenty minutes: the filter went back in front of the query. The next assessment showed that line gone, and nothing new in its place. That diff is the point of doing it this way.
The part I want to be careful about
None of the four tools failed. Each answered the question it was built to answer, and answered it correctly. A code scanner that flagged every unfiltered search would be useless. A cloud tool that guessed at tenancy from index names would be guessing. The pen test was a snapshot and never claimed otherwise.
This isn’t an argument that those tools are bad. It’s an argument about shape: some findings live between surfaces, and a tool that reads one surface cannot see them — not because it’s poorly built, but because the evidence isn’t in its field of view.
And one of those surfaces isn’t a system at all. It’s what your team knows and has never had a reason to write down. Which means the first move isn’t scanning anything. It’s asking.
Skulk reads four surfaces together — your repository, your cloud control plane, your stores, and the intent you declared — and compares them. Every finding carries its evidence; every report states what it did not assess. Scoping costs nothing.