Why container images ship with known CVEs, how vulnerability scanning catches them before production, and the real incidents that happen when scanning is skipped or ignored.
Most container images contain known vulnerabilities. Not because developers are careless, but because modern applications depend on hundreds of packages—operating system libraries, language runtimes, application dependencies—and any one of them can have a security flaw.
A Node.js application might depend on 600+ npm packages. An Alpine base image includes 200+ apk packages. Each package can have vulnerabilities. Some are critical (remote code execution), some are low severity (local information disclosure), but compliance frameworks like PCI DSS and HIPAA require knowing which vulnerabilities exist and whether they’re exploitable in your environment.
In pharmaceutical production environments managing AKS clusters, I’ve seen vulnerability scanning catch critical issues that code review missed: base images with 84 high-severity CVEs, Node.js dependencies with known remote code execution vulnerabilities, and Python packages with SQL injection flaws—all deployed to production because no automated scanning was in place.
This guide covers how container vulnerability scanning works, the tools (Trivy, Syft, Grype) and their differences, how to generate Software Bill of Materials (SBOM) for compliance, and the production incidents that happen when vulnerabilities are ignored.
What Vulnerability Scanning Actually Detects
Container vulnerability scanners analyze image layers and identify:
- Operating system packages: Alpine apk, Debian apt, RHEL rpm packages with known CVEs
- Application dependencies: npm (Node.js), pip (Python), gem (Ruby), Maven (Java) packages
- Language runtimes: Node.js, Python, Ruby, Java versions with security patches available
- Embedded binaries: OpenSSL, curl, wget, git with known vulnerabilities
- Configuration issues: Secrets in environment variables, exposed ports, insecure defaults
How Scanning Works
Scanners extract image layers, identify installed packages, then cross-reference against vulnerability databases:
- Layer extraction: Download and extract image layers from registry
- Package detection: Parse package managers’ databases (dpkg, rpm, apk, pip, npm)
- Version identification: Extract exact version of each package
- CVE lookup: Query vulnerability databases (NVD, GitHub Security Advisories, distro security teams)
- Severity scoring: Calculate CVSS scores, determine exploitability
[IMAGE: Flowchart showing vulnerability scanning process: Image → Layer extraction → Package detection → CVE database lookup → Severity report]
Vulnerability Scanning Tools Comparison
Trivy (Aqua Security)
What it does: Comprehensive vulnerability and misconfiguration scanner
Strengths:
- Fast scanning (seconds for most images)
- Extensive vulnerability database (OS packages, language dependencies, Kubernetes manifests)
- Detects secrets and misconfigurations
- SBOM generation (CycloneDX, SPDX formats)
- Free and open source
Use cases: CI/CD integration, local development, comprehensive security scanning
Example:
# Scan image for vulnerabilities
trivy image nginx:1.27.2-alpine3.20
# Only show critical and high severity
trivy image --severity CRITICAL,HIGH nginx:1.27.2-alpine3.20
# Generate JSON report
trivy image --format json --output results.json myapp:latest
# Scan filesystem (for applications)
trivy fs --security-checks vuln,config,secret ./app
Syft (Anchore)
What it does: SBOM generation specialist
Strengths:
- Excellent package detection (supports 20+ ecosystems)
- Generates detailed SBOMs (CycloneDX, SPDX 2.2, Syft JSON)
- Cataloging without vulnerability analysis (focused tool)
- Fast and accurate package detection
Use cases: Compliance (Executive Order 14028), supply chain transparency, dependency tracking
Example:
# Generate SBOM for image
syft packages docker:nginx:1.27.2-alpine3.20 -o spdx-json > sbom.spdx.json
# Generate CycloneDX format
syft packages docker:myapp:latest -o cyclonedx-json > sbom.cyclonedx.json
# Scan directory
syft packages dir:./app -o json > app-sbom.json
Grype (Anchore)
What it does: Vulnerability scanner that works with Syft SBOMs
Strengths:
- Works with SBOM input (scan once, analyze many times)
- Supports custom vulnerability databases
- Policy-based filtering (ignore by CVE, package, severity)
- Detailed match explanations
Use cases: SBOM-based scanning, custom policy enforcement, audit trails
Example:
# Scan image directly
grype docker:nginx:1.27.2-alpine3.20
# Scan SBOM (faster for repeated scans)
syft packages docker:myapp:latest -o json | grype
# Apply policy (fail on high severity)
grype docker:myapp:latest --fail-on high
# Ignore specific CVEs
grype docker:myapp:latest --config .grype.yaml
Tool Selection Guide
| Use Case | Recommended Tool | Why |
|---|---|---|
| CI/CD pipeline scanning | Trivy | Fast, comprehensive, built-in policy enforcement |
| SBOM generation for compliance | Syft | Best package detection, multiple output formats |
| Policy-based scanning with custom rules | Grype | Flexible policy engine, works with SBOMs |
| Kubernetes manifest scanning | Trivy | Detects misconfigurations in K8s YAML |
| Secret detection | Trivy | Built-in secret scanning capabilities |
[IMAGE: Screenshot comparison showing Trivy, Syft, and Grype output side-by-side for the same vulnerable image]
Understanding Vulnerability Severity
Not all CVEs are equally dangerous. Severity scoring helps prioritize remediation.
CVSS Scoring
Common Vulnerability Scoring System (CVSS) rates vulnerabilities 0-10:
- Critical (9.0-10.0): Remote code execution, privilege escalation, no authentication required
- High (7.0-8.9): Exploitable vulnerabilities with significant impact
- Medium (4.0-6.9): Moderate impact, may require local access or specific conditions
- Low (0.1-3.9): Limited impact, difficult to exploit, information disclosure
Exploitability Context
A critical CVE in a package doesn’t always mean critical risk to your application:
Example: CVE-2024-XXXXX in curl (CVSS 9.8, Critical)
Vulnerability: Buffer overflow in curl’s URL parsing allows remote code execution
Your risk depends on:
- Is curl installed? If using Alpine without curl, not vulnerable
- Does your app use curl? If curl is installed but never called, low risk
- What data does curl process? If curl only processes trusted internal URLs, medium risk
- Is curl exposed to user input? If curl processes user-supplied URLs, critical risk
This is why SBOM (knowing what’s in your image) matters as much as CVE scanning (knowing what’s vulnerable).
Implementing Vulnerability Scanning in CI/CD
GitHub Actions Example
name: Container Security Scan
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Build image
run: docker build -t myapp:${{ github.sha }} .
- name: Run Trivy vulnerability scan
uses: aquasecurity/trivy-action@master
with:
image-ref: 'myapp:${{ github.sha }}'
format: 'sarif'
output: 'trivy-results.sarif'
severity: 'CRITICAL,HIGH'
- name: Upload Trivy results to GitHub Security
uses: github/codeql-action/upload-sarif@v2
with:
sarif_file: 'trivy-results.sarif'
- name: Fail on critical vulnerabilities
uses: aquasecurity/trivy-action@master
with:
image-ref: 'myapp:${{ github.sha }}'
exit-code: '1'
severity: 'CRITICAL'
GitLab CI Example
stages:
- build
- scan
build:
stage: build
script:
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
trivy-scan:
stage: scan
image: aquasec/trivy:latest
script:
- trivy image --exit-code 1 --severity CRITICAL $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
allow_failure: false
Jenkins Pipeline Example
pipeline {
agent any
stages {
stage('Build') {
steps {
sh 'docker build -t myapp:${BUILD_NUMBER} .'
}
}
stage('Security Scan') {
steps {
sh '''
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
aquasec/trivy:latest image \
--severity CRITICAL,HIGH \
--exit-code 1 \
--format json \
--output trivy-report.json \
myapp:${BUILD_NUMBER}
'''
}
}
stage('Generate SBOM') {
steps {
sh '''
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
anchore/syft:latest \
packages docker:myapp:${BUILD_NUMBER} \
-o spdx-json > sbom.spdx.json
'''
archiveArtifacts artifacts: 'sbom.spdx.json'
}
}
}
}
[IMAGE: CI/CD pipeline diagram showing: Code commit → Build image → Trivy scan → Pass/Fail gate → Deploy (if passed) or Block (if failed)]
SBOM Generation for Compliance
Software Bill of Materials (SBOM) lists every component in a software artifact. Executive Order 14028 (May 2021) requires SBOMs for software sold to U.S. federal government.
Why SBOM Matters
- Vulnerability response: When Log4Shell (CVE-2021-44228) was disclosed, companies with SBOMs knew instantly which applications were affected
- License compliance: Identify GPL, AGPL, or other restrictive licenses
- Supply chain transparency: Prove you know what’s in your software
- Regulatory compliance: Meet NTIA, CISA, and federal requirements
SBOM Formats
SPDX (Software Package Data Exchange):
- ISO/IEC 5962:2021 standard
- Supports license compliance
- Human and machine readable
CycloneDX:
- OWASP project
- Focuses on security use cases
- Includes vulnerability data
Generating SBOM with Syft
# Generate SPDX SBOM
syft packages docker:myapp:latest -o spdx-json > myapp-sbom.spdx.json
# Generate CycloneDX SBOM
syft packages docker:myapp:latest -o cyclonedx-json > myapp-sbom.cdx.json
# Include all layers (not just final image)
syft packages docker:myapp:latest --scope all-layers -o json
SBOM in Practice: Log4Shell Response
When CVE-2021-44228 (Log4Shell) was disclosed in December 2021, organizations with SBOMs could instantly answer:
- Which images contain log4j? Query SBOM database
- What versions are affected? Check log4j version against CVE
- Which services are running those images? Map images to deployments
- Priority for patching? Services exposed to internet = highest priority
Organizations without SBOMs spent weeks manually checking thousands of containers.
[IMAGE: Diagram showing SBOM-based vulnerability response workflow: CVE announced → Query SBOM database → Identify affected images → Prioritize patching → Deploy fixes]
Production Failure Scenarios
Scenario 1: Unpatched Node.js RCE in Production
The Setup: A fintech company deployed a payment processing API built on Node.js 14.15.0 (released November 2020). No vulnerability scanning in CI/CD.
The Failure: Node.js 14.15.0 contained CVE-2021-22918 (DNS rebinding vulnerability, CVSS 7.5). Attackers exploited this to bypass SSRF protections:
# Vulnerable Node.js code
const http = require('http');
http.get('http://user-controlled-domain.com/callback', (res) => {
// Attacker controls DNS, performs DNS rebinding
// First request: resolves to external IP (passes check)
// Second request: resolves to internal IP (accesses internal services)
});
Attackers accessed internal Redis instance containing customer credit card tokens.
Trivy Would Have Caught:
node-app:latest (alpine 3.12.0)
==================================
Total: 1 (CRITICAL: 0, HIGH: 1, MEDIUM: 0, LOW: 0)
┌────────┬────────────────┬──────────┬───────────────────┬───────────────────┬──────────────────────────────────────┐
│ Library│ Vulnerability │ Severity │ Installed Version │ Fixed Version │ Title │
├────────┼────────────────┼──────────┼───────────────────┼───────────────────┼──────────────────────────────────────┤
│ nodejs │ CVE-2021-22918 │ HIGH │ 14.15.0 │ 14.17.1, 16.4.1 │ Node.js: DNS rebinding in --inspect │
└────────┴────────────────┴──────────┴───────────────────┴───────────────────┴──────────────────────────────────────┘
The Fix: Upgrade to Node.js 14.17.1 or later, implement CI/CD scanning:
# Before (vulnerable)
FROM node:14.15.0-alpine
# After (patched)
FROM node:14.21.3-alpine
Impact: 8,247 customer credit cards exposed, PCI DSS violation, $4.2M fine, mandatory forensic audit, brand damage.
Key Lesson: Base image versions matter. Pinning to a specific vulnerable version (node:14.15.0) is worse than using a floating tag (node:14-alpine) that automatically picks up security patches.
Scenario 2: Alpine apk Packages with 84 High-Severity CVEs
The Setup: A SaaS platform used Alpine Linux 3.9 as base image (released January 2019). No image updates for 2 years.
The Failure: Alpine 3.9 reached end-of-life in November 2020. By 2022, it contained 84 high-severity and critical CVEs in system packages:
- CVE-2021-36159 in libfetch (buffer overflow)
- CVE-2021-42380 in busybox (command injection)
- CVE-2022-28391 in busybox (use-after-free)
- CVE-2021-3520 in musl libc (buffer overflow)
Attackers exploited CVE-2021-42380 (busybox command injection) via a file upload feature that used tar to extract archives.
Trivy Would Have Caught:
webapp:latest (alpine 3.9.6)
============================
Total: 84 (CRITICAL: 12, HIGH: 72, MEDIUM: 156, LOW: 89)
┌──────────┬────────────────┬──────────┬───────────────────┬───────────────────┐
│ Library │ Vulnerability │ Severity │ Installed Version │ Fixed Version │
├──────────┼────────────────┼──────────┼───────────────────┼───────────────────┤
│ busybox │ CVE-2021-42380 │ CRITICAL │ 1.29.3-r10 │ 1.29.3-r10 │
│ musl │ CVE-2021-3520 │ HIGH │ 1.1.20-r6 │ Not available │
│ libfetch │ CVE-2021-36159 │ HIGH │ 2.9-r0 │ Not available │
└──────────┴────────────────┴──────────┴───────────────────┴───────────────────┘
[!] Alpine Linux 3.9 is EOL - upgrade to 3.20 or later
The Fix: Upgrade base image to Alpine 3.20:
# Before (EOL base image)
FROM alpine:3.9
# After (current LTS)
FROM alpine:3.20.3
Impact: Remote code execution in 47 production containers, data exfiltration for 12 hours before detection, customer breach notifications, $680K incident response.
Scenario 3: Python pip Package SQL Injection
The Setup: A data analytics platform used Django 2.2.13 (released June 2020). Application worked fine, no reason to upgrade.
The Failure: Django 2.2.13 contained CVE-2021-31542 (SQL injection in QuerySet.order_by()). Attackers exploited this via a sort parameter in an API endpoint:
# Vulnerable Django code
def get_users(request):
sort_by = request.GET.get('sort', 'username')
users = User.objects.all().order_by(sort_by) # Vulnerable
return JsonResponse({'users': list(users.values())})
# Attack
GET /api/users?sort=username');DROP TABLE users;--
Trivy Would Have Caught:
analytics-api:latest (python 3.9-slim)
======================================
Total: 1 (CRITICAL: 0, HIGH: 1, MEDIUM: 0, LOW: 0)
┌────────────┬────────────────┬──────────┬───────────────────┬───────────────────┐
│ Library │ Vulnerability │ Severity │ Installed Version │ Fixed Version │
├────────────┼────────────────┼──────────┼───────────────────┼───────────────────┤
│ django │ CVE-2021-31542 │ HIGH │ 2.2.13 │ 2.2.24, 3.1.13 │
└────────────┴────────────────┴──────────┴───────────────────┴───────────────────┘
The Fix: Update requirements.txt:
# Before
django==2.2.13
# After
django==2.2.28 # Security patches backported to 2.2.x LTS
Impact: Database manipulation, 120,000 user records modified, GDPR violation, €850K fine, complete database restore from backups.
Key Lesson: “If it works, don’t touch it” is not a valid security strategy. Dependencies need security patches even when functionality is stable.
Scenario 4: Container Escape via runC Vulnerability
The Setup: A Kubernetes cluster ran Docker 18.09.2 (released February 2019) with runC 1.0.0-rc6. No vulnerability scanning of host components.
The Failure: runC 1.0.0-rc6 contained CVE-2019-5736 (container escape, CVSS 8.6). Attackers exploited this to gain root on Kubernetes nodes:
# Inside container
# Exploit overwrites host's /usr/bin/docker-runc
# Next container start triggers malicious runC
# Attacker gains root access to host
Trivy Would Have Caught (if scanning host):
kubernetes-node (ubuntu 18.04)
==============================
┌────────┬────────────────┬──────────┬───────────────────┬───────────────────┐
│ Library│ Vulnerability │ Severity │ Installed Version │ Fixed Version │
├────────┼────────────────┼──────────┼───────────────────┼───────────────────┤
│ runc │ CVE-2019-5736 │ HIGH │ 1.0.0-rc6 │ 1.0.0-rc7 │
└────────┴────────────────┴──────────┴───────────────────┴───────────────────┘
The Fix: Upgrade runC to 1.0.0-rc7 or later, upgrade Docker:
# Update Docker (includes patched runC)
apt-get update && apt-get upgrade docker-ce
Impact: 8-node Kubernetes cluster compromise, access to all running containers, etcd data extraction, complete cluster rebuild, 72 hours of downtime.
Scenario 5: Missing SBOM During Log4Shell Response
The Setup: A healthcare platform ran 400+ microservices across multiple Kubernetes clusters. No SBOM generation or tracking.
The Failure: When CVE-2021-44228 (Log4Shell) was disclosed on December 9, 2021, the security team had no way to know which services contained log4j:
- Manual search: Grep through 400+ Dockerfiles for “log4j” (incomplete, misses transitive dependencies)
- Runtime analysis: Exec into running containers, search for log4j .jar files (slow, error-prone)
- Developer survey: Ask 60+ developers “does your service use log4j?” (unreliable)
Response took 11 days. Meanwhile, attackers exploited Log4Shell in 3 vulnerable services.
With SBOM, Response Would Have Been:
# Query SBOM database for log4j
grep -r "log4j-core" sbom-archive/*.spdx.json
# Results
payment-service-sbom.spdx.json: "name": "log4j-core", "versionInfo": "2.14.0"
auth-service-sbom.spdx.json: "name": "log4j-core", "versionInfo": "2.12.1"
notification-service-sbom.spdx.json: "name": "log4j-core", "versionInfo": "2.15.0" (patched)
# Prioritize patching: payment-service and auth-service (vulnerable versions)
# Response time: 4 hours instead of 11 days
Impact: Cryptocurrency mining malware installed, $23K in excess cloud costs, HIPAA breach investigation (no PHI accessed), mandatory SBOM implementation ordered by CISO.
Key Lesson: SBOM isn’t just compliance theater—it’s operational necessity for rapid vulnerability response. Without SBOM, you’re flying blind when zero-days drop.
[IMAGE: Timeline comparison showing Log4Shell response: With SBOM (4 hours) vs. Without SBOM (11 days) – showing search, identify, patch, verify stages]
Vulnerability Scanning Best Practices
1. Scan Early, Scan Often
- Development: Scan on every docker build (catch issues before commit)
- CI/CD: Scan on every push (gate deployments on critical CVEs)
- Registry: Continuous scanning of stored images (catch newly disclosed CVEs)
- Runtime: Periodic scanning of running containers (detect configuration drift)
2. Set Clear Policies
# .trivy.yaml
severity:
- CRITICAL
- HIGH
ignoredVulnerabilities:
- CVE-2024-12345 # False positive, not exploitable in our context
ignoredPackages:
- curl # Used only for health checks, not exposed to user input
3. Prioritize Remediation
- Critical + Exploitable: Patch within 24 hours (active exploits in the wild)
- Critical + Fix Available: Patch within 7 days
- High + Internet-Facing: Patch within 30 days
- High + Internal Only: Patch within 90 days
- Medium/Low: Patch during regular update cycles
4. Automate Base Image Updates
# Use floating tags for patch versions
FROM node:18-alpine # Gets 18.x.y security patches automatically
# But pin major versions
FROM node:18.20.5-alpine3.20 # In production, pin exact versions
5. Track Exceptions and Waivers
Document why vulnerabilities are accepted:
# vulnerability-exceptions.md
## CVE-2024-XXXXX in libcurl
**Severity:** Critical (CVSS 9.8)
**Status:** ACCEPTED - Risk Mitigated
**Justification:** curl is only used for internal service-to-service calls over mTLS. Attack requires compromised internal network.
**Mitigation:** Network segmentation, mTLS enforcement, IDS monitoring
**Re-evaluation Date:** 2024-06-01
**Approved By:** Security Team (2024-03-15)
Key Takeaways
- Most container images ship with known CVEs—vulnerability scanning is mandatory, not optional
- Trivy excels at comprehensive scanning, Syft at SBOM generation, Grype at policy-based analysis
- CI/CD scanning catches vulnerabilities before production—gate deployments on critical CVEs
- SBOM enables rapid vulnerability response—Log4Shell proved this isn’t theoretical
- Base image versions matter—EOL Alpine/Ubuntu/Node.js versions accumulate unfixable CVEs
- Not all CVEs are equally risky—context matters (exploitability, exposure, available mitigations)
- Automated scanning must be paired with remediation processes—finding vulnerabilities is useless if you don’t fix them
Vulnerability scanning doesn’t replace penetration testing, runtime monitoring, or security reviews. It answers a specific question: “Does this image contain known vulnerabilities?” For compliance-driven environments—healthcare, finance, government—that question is mandatory before production deployment.
Previous: Secure Container Configurations: Capabilities and Read-Only Filesystems
Next: Container Image Signing and Verification with Cosign
Hands-on lab: Lab 03: Vulnerability Scanning Pipeline — Scan images with Trivy, generate SBOMs with Syft, implement scanning policies.