Control maturity assessment vs compliance: what the maturity score does not measure
Geert WarmenbolApril 29, 2026
Key takeaways
Control maturity assessments score documentation quality. The gap between a Level 3 maturity score and Level 3 control effectiveness is exactly the space where undetected breaches occur.
A fintech client achieved Level 3 maturity on access control based on documented RBAC policies. Vulnox scanning found Kafka service accounts with no scoping, allowing unauthorized microservices to read live transaction data. The auditor validated the policy. The policy did not cover service accounts.
In serverless environments, IAM role boundaries between Lambda functions are the network segmentation equivalent. Maturity assessments built around VPC and subnet concepts miss this entirely, producing Level 3 scores in environments with no lateral movement controls.
35% of vulnerabilities in Vulnox assessments require authenticated scanning to detect. Evidence-only maturity reviews cannot find them because evidence review does not execute against the system (Vulnox assessment data, 2024).
The most common maturity assessment failure is treating Level 3 Defined as an outcome rather than a precondition. Consistent documentation is necessary for Level 3. It is not sufficient evidence that the controls are working.
TL;DR
Control maturity assessments measure how well an organization documents and processes its security controls. They do not measure whether those controls produce the outcomes they were designed for. The gap between those two things is where the most expensive findings live. Organizations that reach Level 3 maturity on paper and have never tested whether their Level 3 controls actually stop anything are carrying a risk that their maturity score does not reflect.
The Level 3 that let attackers read transaction data
A fintech company came to Vulnox having recently completed a control maturity assessment. Access control had scored Level 3: Defined. The evidence package was solid. RBAC policies were documented, role assignments were mapped to job functions, the access review process ran quarterly. The auditor accepted the policy documentation as evidence of implementation. During the Vulnox assessment, scanning against the Kafka message bus found service accounts with no scoping. Unauthorized microservices could consume sensitive transaction data directly. The RBAC policy existed. The service accounts the Kafka deployment used had never been brought into the RBAC model. The policy governed the systems the team knew about when they wrote it.
The maturity score was accurate for the controls it measured. The controls it measured were not the controls that mattered. That is not a scoring error. It is a structural limitation of evidence-based maturity review: the assessment validates documentation quality against a defined evidence set. It cannot validate coverage of systems that were never included in the evidence submission.
What maturity levels measure and what they do not
Most control maturity frameworks use a five or six-level scale built around documentation consistency and process repeatability. Level 1 is ad hoc. Level 2 has documentation but inconsistent application. Level 3 is consistently documented and applied. Level 4 adds measurement and monitoring. Level 5 adds continuous improvement loops. The progression is logical for organizational processes. It maps poorly onto the question of whether a control stops attacks, because that question requires execution testing, not documentation review. A logging configuration that is consistently documented, consistently deployed, and consistently ignored because alert volume is too high to action scores Level 3. The threat detection function it was supposed to create does not exist.
Example
The serverless environment problem is the clearest current example of where maturity frameworks break. Controls for network segmentation were designed around VPCs, subnets, and security groups. Lambda functions in the same AWS account sharing an IAM role have no network segmentation equivalent between them. An attacker compromising one Lambda function and using its IAM role to access other functions in the same account is bypassing lateral movement controls that maturity frameworks do not have evaluation criteria for. The organization scores their network segmentation controls as Level 3 or Level 4. The IAM role boundary problem goes unscored because the framework evidence set does not include it.
S3 bucket versioning creates a specific data retention and disposal gap that maturity assessments miss at the documentation level. Organizations implement deletion policies and score Level 3 on data disposal controls. AWS S3 versioning retains previous object versions after the current version is deleted. An attacker with account access recovers prior versions of files the organization considers deleted. The control was documented. The implementation detail that breaks it is not in scope for a documentation review. It requires testing the deletion behavior against the actual storage configuration.
What the assessments found behind the maturity scores
Assessment base: Vulnox control maturity assessment and gap analysis data, 2024, across fintech, SaaS, and healthcare environments.
Service account coverage absent from documented access control policies
In fintech and SaaS environments, access control policies consistently covered human user accounts and role assignments. Service accounts used by automated pipelines, message buses, and background jobs were documented separately, managed by engineering teams under delivery pressure, and not included in the quarterly access review process the maturity assessment used as evidence. The policies were real. The coverage gap was systematic. Every organization in this pattern had the same explanation: the access control process was built when the environment had simpler infrastructure and service accounts were provisioned later without updating the governance model.
A maturity assessment that scores access control based on the RBAC policy and quarterly review evidence is scoring the policy and the review. It is not scoring whether those controls apply to every credential with access to the environment. The gap between the policy scope and the actual credential inventory is where privilege escalation paths accumulate invisibly. (Vulnox assessment data, 2024)
Incident response maturity scores disconnected from operational response capability
Healthcare and financial services organizations consistently scored Level 4 on incident response maturity based on documented IR procedures, defined roles, and completed tabletop exercises. In operational testing, the median time between a simulated critical alert and documented initial response was significantly longer than the IR procedure specified. In several assessments, the escalation contact listed in the IR procedure had left the organization. The procedure had not been updated. The maturity score reflected the documentation state. The operational capability reflected the reality.
Level 4 maturity on incident response requires measurement and monitoring of the control effectiveness, not just consistent documentation. An IR procedure that has never been timed against a realistic alert scenario is a Level 3 control with Level 4 documentation. The distinction surfaces during the incident, not during the assessment.
CloudWatch log configurations that satisfy backup requirements and miss the intent
AWS CloudWatch logs were consistently used as evidence for backup and recovery control maturity in cloud-native environments. In several assessments, the logs were not immutable and were not stored separately from the production environment, which most maturity frameworks require for backup evidence to be valid. The control was documented as CloudWatch logging, which is correct as a description. The implementation did not satisfy the backup requirement the evidence was submitted for. The gap was between what the team understood the control to cover and what the framework required it to cover.
This is a framework translation error rather than a negligence gap, but the outcome is the same: a maturity score that overstates control coverage. The fix is not complex. The problem is that nobody validated the evidence against the requirement before submitting it.
Framework-based vs in-house maturity assessment
Framework-based maturity assessment
Provides standardized evidence requirements and industry-recognized scoring that supports external communication: customer security questionnaires, regulatory inquiries, investor due diligence. The evidence set is defined by the framework, which means the assessment is reproducible and benchmarkable. The limitation is that standardized evidence requirements are designed for average environments, not specific threat models. A framework built around enterprise perimeter architecture will underweight IAM boundary controls in serverless environments regardless of how significant that gap is for the specific organization.
Choose framework-based when external credibility is a primary requirement. Accept that the maturity score reflects framework-defined evidence quality, not environment-specific control coverage. Supplement with hands-on testing against the specific technology stack to find what the evidence set misses.
In-house maturity assessment
Allows evaluation criteria to match the actual environment and threat model. Can include IAM role boundary testing for serverless environments, Kafka service account scoping, and API credential lifecycle management, none of which appear in standard framework evidence sets. The limitation is that in-house assessments lack external credibility and benchmarking reference points. Organizations cannot demonstrate to customers or regulators that their Level 3 means the same thing as a recognized framework's Level 3.
Choose in-house when the environment is sufficiently differentiated that framework evidence requirements miss significant control areas. Recognize that the output cannot substitute for framework-based maturity in external communication contexts. Most mature organizations need both: a framework-based score for external use and an in-house testing program that validates what the framework evidence set cannot reach.
Higher maturity scores can delay remediation of real gaps
Common belief
Reaching a higher maturity level means an organization is more secure. The goal of a control maturity program is to move scores upward over time. Progress is measured in level advancement.
What we found
In re-assessments of organizations that had improved maturity scores since a prior assessment, the most common pattern was documentation improvement rather than control coverage expansion. The score advanced. The finding that the prior assessment surfaced but did not fully remediate was still present. It was now sitting under a better-documented control that passed the evidence review.
The failure mode is specific to organizations that optimize for maturity score improvement rather than control effectiveness improvement. Documentation polish, process formalization, and evidence quality can all advance a maturity score without changing what the control actually does. An organization that moves from Level 2 to Level 3 on access control by formalizing their RBAC documentation without auditing service account coverage has a better score and the same risk. Worse: the improved score can reduce the urgency of the hands-on review that would find the service account gap, because the team and its stakeholders now have a documented reason to believe the control is working.
Where maturity assessment methodology is heading
Within 24 months, at least one major maturity framework will add mandatory execution testing requirements for Level 4 and above, specifically requiring authenticated scanning or behavioral testing rather than documentation review alone, following a publicized breach at an organization with publicly disclosed high maturity scores.
The gap between documentation-based maturity scoring and control effectiveness has been documented in academic and practitioner literature for years. Framework bodies resist adding execution requirements because they increase assessment cost and complexity, reducing adoption. The political pressure to change comes from a high-profile incident where the breached organization had a recognized high maturity score. That incident has not yet happened at sufficient visibility to force framework revision. When it does, the revision will follow within 12 to 18 months. The signal: a breach disclosure where the post-incident review explicitly names the maturity assessment methodology as having missed the exploited control gap.
Confidence: mediumNo major framework revision adding execution testing requirements by 2028, or no publicized breach with a direct maturity assessment credibility failure within the prediction window.IAM boundary controls for serverless and container environments will emerge as a standalone maturity domain in at least two major frameworks within three years, as organizations increasingly discover that existing network segmentation evidence criteria are structurally inapplicable to their actual architecture.
The Lambda IAM role boundary problem and its equivalents in container environments are already well-documented in security research. Framework bodies lag practitioner knowledge by two to four years. The serverless architecture adoption curve means the volume of organizations encountering this gap in maturity assessments is growing. The signal: framework working groups publishing supplemental guidance on cloud-native control evidence requirements, followed by formal framework revision.
Confidence: highNo standalone cloud-native IAM boundary domain added to NIST CSF, CIS Controls, or ISO 27001 by 2028.
The maturity model problem nobody wants to name
Control maturity assessments are useful for process management and organizational communication. They are not reliable measures of security posture. The conflation of the two is the source of most of the expensive findings that follow a maturity assessment with a high score. Organizations that reach Level 3 or Level 4 on a documented framework and treat that score as evidence of security effectiveness have confused the measurement for the thing being measured. The maturity model measures how well the organization manages its security program documentation and processes. It does not measure whether an attacker who gets into the environment will encounter controls that stop them. Those are different questions. A mature security program is a necessary precondition for consistent security. It is not sufficient evidence of it. The organizations that understand this distinction run maturity assessments to manage their programs and run hands-on testing to validate whether those programs work. The organizations that do not conflate them and are surprised when the breach happens to an organization with a documented Level 4 program. The counterargument is that hands-on testing is expensive, time-consuming, and disruptive to production environments, and that for organizations with limited security budgets, a well-managed Level 3 program is a better investment than episodic penetration testing. That argument is correct about the resource constraint. It is not a defense of treating the maturity score as a security posture indicator.
Counterargument
The reasonable objection is that hands-on execution testing is not accessible to most mid-market organizations on a continuous basis, and that well-managed maturity frameworks produce genuine security improvements even if they do not validate control effectiveness directly.
One test to run against the last maturity assessment
Pull the access control section of the last maturity assessment. Find the evidence submitted for service account management or privileged account governance. Check whether the evidence covers every credential type in the current environment: human user accounts, service accounts used by automated pipelines, API keys provisioned by development teams, and credentials used by CI/CD systems. If the evidence does not explicitly address service account coverage, the maturity score is accurate for the controls it measured and silent on the controls it did not. That silence is where the Kafka service account gap lives. Closing it does not require a new assessment. It requires running the access review process against the actual credential inventory rather than the credential inventory that was current when the process was designed.
Further Reading
Gap Analysis
framework gap analysisVulnerability Assessment
comprehensive vulnerability assessmentNIST Cybersecurity Framework 2.0
NIST Cybersecurity FrameworkNIST Vulnerability Assessment Definition
NIST's vulnerability assessment definitionHow to Perform Gap Analysis
perform gap analysis effectively
Frequently Asked Questions
What is the difference between a control maturity assessment and a compliance audit?
A compliance audit verifies that required controls exist and are documented. A control maturity assessment evaluates how well those controls actually function: whether they are consistently applied, operationally tested, and producing the outcomes they were designed for. Organizations can pass a compliance audit at Level 3 maturity and still have critical controls that fail in production. The audit checked the policy. The maturity assessment is supposed to check the implementation. Most maturity assessments check the documentation of the implementation instead.
What do control maturity assessments miss that hands-on testing finds?
Service account scoping, IAM role boundaries in serverless environments, and logging configurations that satisfy documentation requirements but produce no actionable signal. Maturity assessments score against evidence submissions: policies, configuration screenshots, procedure documents. They do not test whether the control produces the intended outcome under realistic conditions. In Vulnox assessments, 35% of vulnerabilities requiring authenticated scanning are missed by evidence-only review approaches.
How many maturity levels are there in standard control maturity frameworks?
Most frameworks use a five or six-level scale. The most common is: Level 1 Initial (ad hoc, undocumented), Level 2 Developing (some documentation, inconsistent application), Level 3 Defined (documented and consistently applied), Level 4 Managed (measured and monitored), Level 5 Optimizing (continuous improvement with feedback loops). The critical gap is between Level 3 and Level 4: organizations that document controls consistently but have never measured whether those controls produce intended outcomes are sitting at Level 3 on paper and Level 2 in practice.
When should an organization choose a framework-based maturity model versus an in-house assessment?
Framework-based models when external credibility matters: customer security questionnaires, regulatory requirements, or investor due diligence. In-house when the organization has a specific technology stack or threat model that generic frameworks underweight. The practical failure mode of framework-based maturity is that it standardizes evidence requirements against the framework, not against actual threats. The practical failure mode of in-house maturity is inconsistency and the inability to benchmark against peers. Neither is categorically better. Both produce the same blind spot: maturity scores that reflect documentation quality rather than control effectiveness.
What is the most common gap between stated control maturity and actual security posture?
The gap between access control policy documentation and access control implementation coverage. Organizations document RBAC policies, achieve Level 3 or Level 4 maturity scores on access control, and have service accounts, legacy API keys, and automated pipeline credentials that the RBAC policy never reached. The policy exists. The policy does not describe what the policy actually governs. Vulnox assessments find this pattern consistently in fintech and SaaS environments where infrastructure was provisioned incrementally and the access control inventory was never updated to match.
How do I use a control maturity assessment to make a budget case to the board?
Map maturity gaps to specific failure scenarios with named financial consequences: the average cost of a data breach in your industry, the regulatory fine range for the compliance obligation the gap affects, the estimated attacker dwell time the detection gap enables. Boards respond to quantified risk scenarios, not maturity level numbers. A gap from Level 2 to Level 3 on incident detection means nothing. The same gap framed as the monitoring configuration that would have failed to detect the Okta-style credential theft your industry saw last year means something.
Related Articles
GovRAMP Moderate authorization: why the Significant Change Request process catches providers off guard
GovRAMP Moderate is the first tier where you need a government sponsor, annual 3PAO reassessment, and a Significant Change Request process that can pause normal product releases for months. Most providers who stall post-authorization were not prepared for what maintaining Moderate status actually costs operationally.
GovRAMP Low+ authorization: the impact level that punishes providers who get the CUI boundary wrong
GovRAMP Low+ is where providers handling limited Controlled Unclassified Information land — or discover they should not be there. The defining failure is not a missing control. It is a CUI boundary that was drawn before anyone asked what data the government actually sends through the system.
GovRAMP High authorization: why FIPS-validated crypto and personnel security controls catch providers off guard
GovRAMP High is where cloud providers discover that having strong encryption is not the same as having FIPS 140-2 validated encryption — and that distinction alone has derailed authorizations from vendors who passed every other control family. The architectural constraints at High are qualitatively different from every lower tier.
Ready to Secure Your Digital Assets?
Get a comprehensive vulnerability assessment for your website today.