A security policy does not prove that a KSC process works in practice
I have a vulnerability management policy. I have a scanner that checks every day whether any CVE found on my machines has made it into the catalogue of vulnerabilities known to be exploited in attacks. The scanner ran, the policy applied, and the answer I was getting was not true.
The check compares my vulnerabilities against a copy of the KEV catalogue, the list maintained by the US agency CISA, which holds vulnerabilities with evidence of use in real attacks. In the 19 September measurement the catalogue held 1716 entries. My copy held 390, because the synchronisation had never started. So the answer "none of the CVEs on your fleet is in this catalogue" was produced by comparing against not quite a quarter of the list, and it sounded exactly as confident as the truth would have sounded.
With the catalogue filled in, the answer came back as seven unique CVEs across 95 instances on five machines. All seven occur on three internet-facing machines, and two of them, both in kernel packages, also on two internal machines. I published the full calculation in a piece about what is actually sitting on nine ordinary servers.
Nothing turned red. As far as the mechanism was concerned everything was fine: question asked, answer returned, status green.
Why this is an audit problem, not just mine
If somebody had asked me back then whether I check vulnerabilities for active exploitation, I would have said "yes, automatically, every day". The answer would have been true. I had a policy for it, I had a screenshot, and I had logs showing the check had run.
Only the second question reveals the difference: against how many entries of the catalogue did that check run.
Article 10(2) of the Polish KSC Act lists normative and operational documentation as the two parts of security documentation, and Article 10(4) describes the latter as records evidencing that activities required by the normative documentation were performed. A policy is one of those normative documents and it describes intent. Operational records are there to show execution.
My case adds a third layer that the Act does not name but that is worth writing down: an operational record can be green while covering a fraction of the intended scope. The mechanism ran correctly and covered a quarter of the catalogue, and neither fact says anything about the other.
The rule and its evidence
Hence the practical check I set myself after this. Where a requirement has a measurable scope, I verify two things separately: that the activity ran, and how far it reached.
Take the sentence "all systems are scanned regularly". The evidence of execution is the scan history; the evidence of scope is that history compared against the inventory. A scan history cannot tell you what is missing from it.
If the policy says "critical vulnerabilities have a deadline", the record should show the detection date, the deadline set, and the date from which the entry stopped appearing in successful scans. That last one is not yet confirmation of a fix: a "fixed" status is an operator decision and needs separate change evidence.
If the policy allows exceptions, the evidence is a decision by an authorised person with a justification, compensating controls and an expiry date.
Management oversight of risk is evidenced by dated reports, review minutes and recorded decisions. A responsibility table shows who was assigned the role, not that the oversight took place.
If the policy says you rely on an external source of knowledge, a catalogue of exploited vulnerabilities or a vendor database, the evidence is not the integration itself. The evidence is how many entries of that source you actually hold and when it last updated.
Do not create evidence as separate busywork
The cheapest model is the one where the trail appears while the work is done. The next successful scan records the date from which an entry stopped being detected. Risk acceptance requires an owner and a review date. A scan import records the machine, the time and the result. A report is a view over that data rather than a separate document written by hand.
Two things such a trail will not give you on its own, and it is better to know that up front. The first is coverage: a scanner knows nothing about machines it was never pointed at, so the list of machines has to be compared against an inventory kept somewhere else. The second is confirmation of a fix: an entry disappearing from a scan only means the scanner no longer sees it, which is not the same as a fix, because the package may have been uninstalled or the scanner may have changed how it identifies it.
Secvalis supports this model for the technical part of vulnerability management. It does not cover the whole ISMS, it does not confirm that an inventory is complete, and it does not turn records into proof of compliance.
If you would like to see what an export for a chosen period looks like, open the Secvalis demo.
Sources
Weekly CVE digest
One email a week with newly published vulnerabilities worth knowing about. No account needed.
This digest covers new vulnerabilities in general, not your servers. If you want to know which of them actually run in your infrastructure, that is what Secvalis does: it scans your machines and reports only what concerns them.