DevSecOps gap analysis: where compliance programs stop and real security begins
Geert WarmenbolApril 29, 2026
Key takeaways
Compliance audits verify that DevSecOps controls exist. They do not verify that those controls are correctly configured. In Vulnox assessments, SAST misconfiguration — tools excluding file types, running on wrong scope, or set to report-only — is the single most common gap in organizations with passing compliance reviews.
Ephemeral cloud resources are structurally outside most compliance asset inventory frameworks. Lambda functions, serverless containers, and spot instances may never appear in audit logs if they terminate before inventory collection runs — and they routinely carry the same vulnerable libraries as persistent systems.
RBAC compliance passes when roles exist and are assigned. It fails when roles have been expanded through accumulated exceptions. The compliance control is present. The access control intent has been abandoned. These are different things and standard audits do not distinguish them.
OWASP SAMM and NIST 800-160 address different organizational needs. SAMM is built for developer-led DevSecOps implementation. NIST 800-160 is built for formal systems engineering programs. Applying the wrong framework to your organizational context produces documentation overhead without security improvement.
Security testing in CI/CD pipelines only works if the scope is calibrated to the stage. Comprehensive scanning on every commit creates latency and noise that developers route around. Fast-scoped checks on commits, comprehensive scanning on release paths, is the operational model that sustains coverage.
Organizations performing a framework gap analysis of their DevSecOps programs typically find that current practices satisfy framework documentation requirements while missing the control intent. The gap is not in what is documented. It is in whether the documented controls are correctly implemented and functioning.
TL;DR
A compliance audit of your DevSecOps program tells you whether the right controls are documented. A gap analysis tells you whether they work. The distance between those two answers is where attackers operate. This article covers the specific misconfiguration patterns that survive compliance reviews in CI/CD pipelines, the structural gaps that frameworks cannot address in ephemeral environments, and the practical difference between a SAST tool that runs and a SAST tool that catches anything.
The SAST tool was running. It was checking the wrong files.
A fintech running a Node.js trading platform had passed their compliance audit with a clean result on code review controls. SAST integrated in CI/CD, passing on every build, findings reported to the security team. Three months later, a Vulnox assessment found prototype pollution and XSS vulnerabilities throughout their front-end codebase. The SAST tool had been configured at initial deployment to exclude JavaScript files — a decision made during setup to reduce scan time on a slow pipeline. The exclusion was never revisited. The compliance audit reviewed whether SAST was present and producing output. It did not review what the tool was configured to analyze. The tool had been running correctly against the wrong scope for the better part of a year.
The finding was not that static analysis fails. It was that the configuration decision made at deployment time — which files to scan — was never treated as a security-relevant decision. It was treated as a performance optimization. No one documented it as a scope limitation. The audit saw a functioning SAST stage. The gap was invisible until someone asked what the tool was actually checking.
Where compliance frameworks stop short in DevSecOps environments
Compliance frameworks for DevSecOps address the existence of security controls across the SDLC. They require code review, vulnerability scanning, access controls, and incident response capability. What they do not address is configuration correctness — whether the controls are implemented in a way that achieves the control intent, or just achieves the compliance documentation requirement.
The gap exists because frameworks are written to be assessable by auditors who review documentation and evidence artifacts. An auditor can verify that a SAST tool is present in the pipeline, that it produces output, and that findings are tracked. An auditor cannot efficiently verify that the tool's configuration covers the actual attack surface of the application. That requires technical knowledge of the specific codebase and tool configuration, which is outside the scope of most compliance assessments.
The result is a class of gaps that are structurally invisible to compliance audits: correct tools, wrong configuration. Security controls that exist on paper, that produce output that looks correct, and that are systematically failing to check the things that matter.
Example
DAST tools have the same problem on the other side of the pipeline. A DAST scanner configured against a staging environment with seeded test data will produce findings on that data set. If the production environment has different application paths, different data structures, or different authentication flows than the staging environment, the scan coverage does not transfer. The compliance evidence shows DAST scanning in the pipeline. The coverage gap is in what the scanner actually reached.
The tool configuration problem compounds across the pipeline because each stage's configuration decisions are made independently and rarely reviewed against each other. SAST excludes a file type. DAST covers staging but not production paths. SCA scans direct dependencies but not transitive ones because the tool default excludes them. Each individual decision was made for a legitimate operational reason. The cumulative effect is a pipeline that produces compliance evidence and leaves significant attack surface unchecked.
What gap analysis finds in DevSecOps programs
Assessment base: Vulnox assessment data, 2024-2025, across DevSecOps program assessments in fintech, SaaS, and healthcare verticals.
SAST tools misconfigured to exclude primary attack surfaces
The most frequent finding in DevSecOps gap assessments is SAST misconfiguration. The pattern: tool is present in CI/CD, produces output, finding trends are tracked, and the tool is systematically not checking the highest-risk code paths. Exclusions are the most common form — file types, directories, or dependency trees excluded at setup for performance or noise reasons and never revisited. In the fintech case above, JavaScript exclusion left an entire attack surface class unchecked. In a SaaS assessment, the SAST configuration excluded third-party library directories, which contained a known vulnerable version of a serialization library that was the actual entry point in a later incident.
Compliance evidence for code review does not distinguish between a tool checking everything and a tool checking the documented scope. Both produce passing status in an audit. The gap analysis question is: what is the tool configured to check, and does that match the actual attack surface? Those are rarely the same answer.
Ephemeral resources outside asset inventory and security control scope
Compliance frameworks require asset inventory, and most organizations maintain one. The inventory covers persistent infrastructure. Lambda functions, ECS tasks, and spot instances that exist for minutes to hours are structurally difficult to inventory in real time — they may terminate before the next inventory collection run. In Vulnox assessments, these ephemeral resources routinely run on the same vulnerable library versions as persistent systems, inherit broad IAM roles from their creation template, and have no security agent or monitoring deployed. They are outside the asset inventory because the inventory architecture was not designed for their lifecycle.
An attacker who knows a vulnerability exists in a specific library version used by Lambda functions does not need persistence. They need the function to invoke, the window to execute, and a path to exfiltrate. The function terminates. The access disappears from logs. The asset was never in the inventory. This is not a theoretical attack pattern.
RBAC roles compliant at deployment, non-compliant through accumulated exceptions
Access control compliance reviews assess whether RBAC is implemented and whether access follows documented role definitions. They assess role definitions at a point in time. What they do not track is role definition drift — the pattern where individual access exceptions are granted to specific users within a role, and those exceptions accumulate until the role definition no longer reflects actual permissions. In a healthcare assessment, the nurse role in an EMR system had been expanded 14 times over three years to accommodate individual exception requests. The role had access to SSNs, billing records, and system administration functions. Each individual exception was approved through the access management process. The cumulative state was never reviewed as a role definition problem.
Compliance evidence for RBAC shows that roles exist and assignments are reviewed. It does not show what the roles actually permit after three years of approved exceptions. The control exists. The principle of least privilege it was meant to enforce does not.
OWASP SAMM vs NIST 800-160: which framework fits which program
OWASP SAMM
Designed specifically for software development organizations. Maps security practices to business functions across governance, design, implementation, verification, and operations. Provides prescriptive maturity levels that development teams can act on without a dedicated security engineering resource. Weakness: assumes a baseline of security knowledge, does not provide systems engineering depth for complex or regulated environments, and does not address non-software security domains. Strong fit for developer-led DevSecOps programs building security culture from a low baseline.
If your goal is to get developers asking security questions at design time and integrating automated testing into their workflow, SAMM gives you a roadmap they can own. If your goal is formal compliance documentation for a regulated environment or systems with complex interdependencies, SAMM does not produce the artifact structure those requirements need.
NIST 800-160
Systems security engineering framework covering the full system lifecycle. Risk-based, structured, and comprehensive. Produces formal documentation artifacts suitable for compliance evidence in regulated environments. Weakness: breadth is overwhelming without a defined starting point, it is not developer-facing, and it assumes dedicated security engineering resources. Not designed to produce cultural change in development teams — designed to produce documented security engineering processes.
If your organization has an existing security program, formal engineering processes, and compliance requirements that demand structured evidence, NIST 800-160 gives you the framework depth those requirements need. If your primary problem is that developers do not think about security during design, NIST 800-160 will produce documentation that describes the problem without changing the behavior.
DevSecOps gaps that frameworks consistently do not address
Secrets in build artifacts and container images
Secret scanning is a standard CI/CD control and most organizations have it deployed. The consistent gap is build-time secrets that are baked into container images — environment variables set during build, credentials passed as build arguments, configuration files included in the image layer. Secret scanning in the pipeline checks source code before build. It does not always check the resulting artifact. A secret that is not in the repository but is in the built image is outside the scope of most secret scanning configurations. Container image scanning addresses this, but the two scanning tools are frequently configured and operated by different teams with no cross-reference of findings.
Third-party component licenses and supply chain integrity
Software composition analysis addresses known vulnerabilities in dependencies. It does not address supply chain integrity — whether the dependency you are pulling is the package you think it is. Typosquatting, dependency confusion attacks, and compromised package maintainer accounts have produced malicious packages in the major registries in the past two years. SCA tools check the package's CVE history. They do not verify the package's build provenance or detect unexpected behavioral changes in a new version of a trusted library. SLSA framework guidance addresses this, but adoption in mid-market organizations is near zero.
Security testing coverage of API endpoints not in documentation
DAST tools test the attack surface they can enumerate. In modern applications, the enumerable attack surface is what the documentation and the crawl reveals. Undocumented API endpoints — legacy routes left active during migrations, internal endpoints exposed without authentication in production, debug endpoints not disabled — are outside DAST scope by definition. These are consistently the most interesting findings in application security assessments. They are not in any inventory. They are not in any threat model. They show up when you probe outside the documented surface.
Running a DevSecOps gap analysis: the sequence that produces findings
- Step 1
Map your pipeline stages to control categories
Output:A pipeline control map that shows, for each stage, what tool runs, against what scope, with what configuration. This is the artifact that makes misconfiguration visible.
Purpose:Establish what security controls are supposed to run at each stage — secret scanning, SAST, SCA, DAST, container scanning, infrastructure-as-code analysis — and document the scope each tool is configured to cover.
- Step 2
Enumerate what is not in scope for each tool
Output:An exclusion inventory that can be evaluated against the actual attack surface. A JavaScript exclusion in a Node.js application is a critical gap. The same exclusion in a Python application may be irrelevant. Context determines severity.
Purpose:For each tool in the pipeline, document what is explicitly excluded from its configured scope. File type exclusions, directory exclusions, dependency depth limits, staging vs production coverage. The gaps are in the exclusions.
- Step 3
Assess asset inventory coverage for ephemeral resources
Output:A list of resource types that are outside current inventory coverage with the security control implications of that gap — which controls depend on inventory completeness and what attack surface exists in the untracked resources.
Purpose:Determine whether your inventory architecture captures resources that exist for less than your inventory collection frequency. For organizations running serverless or heavy container workloads, this is a structural gap that requires a different inventory approach, not just a configuration change.
- Step 4
Audit RBAC role definitions against current effective permissions
Output:A role drift report showing which roles have expanded beyond their original scope, the specific permissions added through exceptions, and the approval history for each expansion. This is the input to a role remediation process.
Purpose:Compare documented role definitions against what each role can actually access in the current system state. Identify roles that have drifted from their original definition through approved exceptions. Quantify the gap between documented least privilege and actual effective permissions.
- Step 5
Test the detection layer against known findings
Output:Detection validation evidence: confirmed that specific control configurations catch specific attack classes. This is the evidence that a compliance audit does not produce and a gap analysis does.
Purpose:Take a known finding — a specific vulnerability class the pipeline should detect — and verify that the current tool configuration would catch it. Introduce a test case that represents the finding and confirm the tool fires. This validates that the pipeline catches what it is supposed to catch, not just that the tool is running.
Where DevSecOps gap analysis is heading
By 2027, software supply chain attacks targeting mid-market organizations via compromised package maintainer accounts will become more common than direct application vulnerability exploitation as the primary initial access vector in this company size segment.
Enterprise organizations have deployed SCA and are beginning to implement SLSA-level provenance checks. Mid-market organizations have SCA for CVE detection but no provenance verification. Attackers optimize toward the path with least resistance. As direct vulnerability exploitation in well-monitored enterprise environments becomes harder, the mid-market supply chain — less monitored, less protected, often connected to enterprise customers via integrations — becomes the more efficient target. The signal: increase in package maintainer account takeover incidents in the npm and PyPI registries in 2025-2026 disclosure data.
Confidence: mediumBreach disclosure data from mid-market organizations in 2027 showing initial access vector distribution, specifically the ratio of supply chain compromise to direct application exploitation.OWASP SAMM adoption will plateau and a new framework category will emerge specifically addressing AI-assisted development security — covering code generated by LLM tools, prompt injection in development workflows, and security review requirements for AI-generated code that human reviewers cannot efficiently evaluate.
SAMM was designed for human-written code in human-managed pipelines. The assumption that code review means human review of human-written code is breaking down. AI-generated code introduces vulnerability patterns that differ from human-introduced patterns, appears in codebases faster than review capacity can handle, and may include constructs that look correct to human reviewers but contain subtle logic flaws. No current framework addresses the security review requirements specific to AI-assisted development at the tooling and process level. This gap will produce a new framework category within three years.
Confidence: mediumPublication of a recognized framework (OWASP, NIST, or equivalent authority) specifically addressing AI-assisted development security requirements by end of 2028.
Threat modeling is the one DevSecOps practice that does not scale through automation
Every other DevSecOps control benefits from automation and gets better at scale. Threat modeling is the exception. It requires someone who understands the specific application — what it does, where data flows, where trust boundaries exist — to think through what an attacker would do. You cannot automate that. You can automate the documentation. You can use tools to surface known patterns. But the thinking that produces a useful threat model is irreducibly human and irreducibly contextual. Organizations that try to replace threat modeling with automated tools get reports that list generic attack categories and miss the specific architectural decision in this application that an attacker would actually target. The fintech with the JavaScript exclusion had an automated threat model that flagged XSS as a risk category. It did not flag that their SAST tool was not checking JavaScript files. That is the gap between a category and a finding.
Counterargument
The counterargument is that imperfect threat modeling at scale is better than perfect threat modeling on a few high-priority systems. Automated tools that prompt developers to think about trust boundaries and data flows at design time produce some security thinking where none would otherwise occur. That is a fair point, and for organizations with no threat modeling practice at all, automated tooling that makes threat modeling accessible to developers without security training is genuinely valuable. The problem is when automated output is treated as equivalent to a security engineer's analysis. It is not. The categories are not the findings.
One thing to do this week
Pull the configuration files for your SAST tool. Find the exclusion list — the file types, directories, or patterns that are excluded from analysis. Then map each exclusion against your actual codebase: what percentage of your application code is in the excluded file types? What are the most common vulnerability classes in those file types for your stack? If the exclusions cover a meaningful percentage of your attack surface, you have a gap that your compliance audit did not find and your security tool is not catching. That single check will tell you more about the actual state of your DevSecOps program than your most recent compliance report.
Further Reading
Gap Analysis
DevSecOps gap analysisNIST
NIST framework questionnaireNIST Cybersecurity Framework 2.0
NIST Cybersecurity FrameworkOWASP Web Security Testing Guide
OWASP security testing guideVulnerability Assessment with NIST CSF
vulnerability assessment guide
Frequently Asked Questions
What does a DevSecOps gap analysis find that a standard compliance audit misses?
A compliance audit verifies that specific controls exist and are documented. A DevSecOps gap analysis tests whether those controls work as configured in your actual pipeline. The most common gap we find: SAST tools configured to exclude file types that are primary attack vectors. A fintech we assessed had static analysis passing in their CI/CD pipeline — the tool was misconfigured to skip JavaScript files entirely, the main attack surface for their Node.js platform. The audit saw a passing SAST stage. We saw an unchecked attack surface.
How do ephemeral cloud resources create DevSecOps gaps that frameworks miss?
Compliance frameworks require asset inventory, but they were written before Lambda functions, serverless containers, and spot instances became primary compute patterns. These resources may exist for minutes, never appear in standard inventory reports, and typically inherit permissive IAM roles from the creation template rather than the principle of least privilege. In assessments, we find attackers can target a known vulnerability in a short-lived function, execute code, pivot to adjacent services, and exfiltrate data before the resource appears in any audit log. The framework article requires inventory. It has no mechanism for resources that do not persist long enough to be inventoried.
What is the difference between NIST 800-160 and OWASP SAMM for a DevSecOps program?
NIST 800-160 is a systems security engineering framework suited to organizations with formal security programs and complex compliance requirements. It is comprehensive and risk-based but assumes dedicated security engineering resources and does not provide developer-facing workflow guidance. OWASP SAMM is built specifically for software development organizations. It maps to business functions, gives prescriptive maturity levels, and is designed for developer-led implementation. For most organizations starting a DevSecOps program, SAMM provides a faster path to embedded security practices. For organizations with formal systems engineering processes or regulated environments requiring structured compliance evidence, NIST 800-160 provides the audit-grade documentation layer.
Why do RBAC implementations pass DevSecOps compliance reviews but still create security exposure?
Compliance reviews verify that RBAC exists and is applied. They do not assess whether role definitions follow least privilege in practice. In a healthcare assessment, an organization passed access control requirements by implementing RBAC across their EMR system. The roles were defined by job function — nurses, administrators, clinicians — but each role had been expanded incrementally over time to accommodate exceptions. By the time of our assessment, the nurse role had access to SSNs, billing records, and administrative functions that had nothing to do with clinical care. The compliance control was present. The control intent had been abandoned through accumulated exceptions.
How do you integrate security testing into a CI/CD pipeline without slowing deployment velocity?
The sequence matters more than the tooling. Fast checks — secret scanning, dependency vulnerability scanning, SAST on changed files only — run on every commit and block merges. They are fast because they are scoped. Full SAST, DAST, and container image scanning run on merge to main or on release branches, not on every commit. The mistake most teams make is running everything on every commit, generating latency and alert noise that developers route around. Scope fast checks for commit gates. Reserve comprehensive scanning for the release path. The security coverage is similar; the developer experience is materially better.
What is the minimum viable threat modeling process for a small DevSecOps team?
Three questions per feature, answered before implementation starts: what are we building and what does it touch, what can go wrong at each trust boundary, and what do we do if it does. This does not require a formal threat modeling tool or a dedicated security engineer. It requires that the question gets asked in the design phase rather than the post-deployment review. In Vulnox assessments, the majority of critical application vulnerabilities we find were foreseeable at design time. They were not the result of sophisticated attacks. They were the result of trust boundary decisions made without asking what an attacker could do with the result.
How does a DevSecOps gap analysis differ from a penetration test?
A penetration test finds exploitable vulnerabilities in your running application. A DevSecOps gap analysis examines your development and deployment pipeline — the processes, tool configurations, and control integrations that determine whether vulnerabilities reach production in the first place. Penetration testing tells you what is vulnerable now. A DevSecOps gap analysis tells you why vulnerabilities keep appearing and which pipeline stages are failing to catch them. Both are necessary. Organizations that do penetration testing without a DevSecOps gap analysis keep finding the same vulnerability classes in each test cycle because the pipeline condition that allows them to reach production has not been identified or changed.
Related Articles
GovRAMP Moderate authorization: why the Significant Change Request process catches providers off guard
GovRAMP Moderate is the first tier where you need a government sponsor, annual 3PAO reassessment, and a Significant Change Request process that can pause normal product releases for months. Most providers who stall post-authorization were not prepared for what maintaining Moderate status actually costs operationally.
GovRAMP Low+ authorization: the impact level that punishes providers who get the CUI boundary wrong
GovRAMP Low+ is where providers handling limited Controlled Unclassified Information land — or discover they should not be there. The defining failure is not a missing control. It is a CUI boundary that was drawn before anyone asked what data the government actually sends through the system.
GovRAMP High authorization: why FIPS-validated crypto and personnel security controls catch providers off guard
GovRAMP High is where cloud providers discover that having strong encryption is not the same as having FIPS 140-2 validated encryption — and that distinction alone has derailed authorizations from vendors who passed every other control family. The architectural constraints at High are qualitatively different from every lower tier.
Ready to Secure Your Digital Assets?
Get a comprehensive vulnerability assessment for your website today.