Audit code and generate reports

Set up the lab

1. Clone the Galaxium Travels repository.

   git clone -b bob-learning-path-branch https://github.com/IBM/galaxium-travels

2. Click File then Open Folder.

3. Navigate to the galaxium-travels directory you cloned and open it.

4. Open the Bob chat interface by clicking the Bob icon beside the navigation bar, or use the shortcut Option + Command + B (macOS) or Ctrl + Alt + B (Windows).

5. In the chat input field, run /init to initialize the development environment and create the AGENTS.md files for Bob. Click Approve todo tools for task if prompted.

Create an audit skill

Create a skill, which is a reusable instruction set that Bob uses to work through a specific task.

The following skill audits the Galaxium Travels codebase against OWASP ASVS Level 1 requirements.

The skill checks the following controls:

1. Under the chat interface, click Bob - Settings, then click Bob Settings.

2. Click Skills in the left sidebar.

3. Click the + button to create a new skill.

4. Enter asvs-audit in the Skill Name field. This is the name used to invoke the skill with /asvs-audit in the chat.

5. Enter a short description in the Description field:

   Audits a codebase against OWASP ASVS Level 1 access control, input validation, API security, and configuration requirements.

6. Make sure the Allow Bob to use this skill toggle is on.

   With the toggle on, Bob can activate the skill on its own when a prompt or plan calls for it. The audit plan you create later in this tutorial does exactly that.

7. Change the Scope & Location to galaxium-travels.

   This saves the skill in the project's .bob/skills/ directory, so it is only available in this project and your team can version-control it with the codebase. The global location (~/.bob/skills/) would make the skill available in every project on your machine.

8. Enter the following skill in the Skill Instructions textbox.

   ---
   name: asvs-audit
   description: Audit a codebase against OWASP ASVS Level 1 access control, input validation, API security, and configuration requirements and produce structured findings ready for SARIF and OSCAL export.
   user-invocable: true
   ---
   
   Perform a structured security audit of this codebase. Work through the following phases in order. Do not skip phases or combine them.
   
   ## Phase 1: Discover
   
   Read and understand the application before auditing. Focus on:
   - Entry points: main files, route definitions, controllers
   - Authentication and session handling code
   - Input validation and sanitization code
   - Database query code
   - Any files flagged as high-risk in earlier analysis
   
   Summarize what you find before proceeding to Phase 2.
   
   ## Phase 2: Audit
   
   Check each control below. For each one record: PASS, FAIL, or N/A.
   For every FAIL, record the file path and line number.
   
   ### V4.1 General Access Control
   - V4.1.3 — Users can only access their own resources; other users' data is not accessible
   - V4.1.5 — Access control denies by default — unauthenticated requests are rejected
   
   ### V4.2 Operation Level Access Control
   - V4.2.1 — Sensitive resources cannot be accessed by manipulating a predictable object ID (IDOR protection)
   
   ### V5.1 Input Validation
   - V5.1.1 — All string inputs have defined maximum length constraints
   
   ### V13.1 Generic Web Service Security
   - V13.1.3 — API endpoints do not accept credentials or PII in URL query parameters
   
   ### V14.4 HTTP Security Headers
   - V14.4.1 — HTTP responses include appropriate security headers such as Content-Security-Policy, X-Frame-Options, and X-Content-Type-Options
   
   ### V14.5 HTTP Request Header Validation
   - V14.5.3 — CORS origin is validated against an explicit allowlist — wildcard origins are not permitted
   
   ## Phase 3: Generate Findings
   
   For each FAIL, produce a finding in this format:
   
   **Finding [N]:**
   - Rule: ASVS [control number]
   - Severity: Critical / High / Medium / Low
   - File: [path]
   - Line: [number or range, if identifiable]
   - Issue: [one sentence describing what was found]
   - Fix: [one sentence describing the recommended change]
   
   ## Phase 4: Summary
   
   Produce a short summary:
   - Total controls checked
   - Pass / Fail / N/A counts
   - Two-sentence overall security posture assessment
   
   Save the findings to the location specified by the plan or prompt that invoked this skill. Do not generate SARIF, OSCAL, or other report files — report generation is a separate task. Report that the audit is complete and wait for the next instruction.

   This is the complete instruction set Bob follows during the audit.

9. Click Create.

Find high-risk areas to audit

To save tokens, ask Bob to identify the files and folders most relevant to security. You will run the audit skill on these areas.

1. If the chat panel is not already open, open it with Option + Command + B (macOS) or Ctrl + Alt + B (Windows).

2. Select Ask from the mode selector.

   Each mode has different capabilities and reasoning styles. Ask mode works best for questions and analysis, but you cannot write or edit files in Ask mode.

3. In the chat input field, enter the following prompt to explore the codebase to find the highest security risk areas:

   Explore this codebase as a Senior Security Analyst. Give me a short summary covering:
   
   1. The primary tech stack and framework
   2. Identify the files and folders most relevant to security
   
   Make sure to also review:
   1. How authentication and session management are handled
   2. How user input is accepted and validated
   3. Where database queries are made
   4. Any API endpoints that accept external input
   
   I want to understand the highest security risk areas before running an audit.

   Bob reads the files and responds with a summary of the application structure.

Create a plan for auditing the high-risk areas

Create a plan for Bob to follow when auditing the high-risk areas.

1. Switch to Plan mode.

2. Prompt Bob to create a plan for auditing the high-risk areas. Click Approve todo tools for task if prompted.

   Create a plan for auditing the high-risk areas found in the previous
   exploration.
   
   When auditing, use the asvs-audit skill to guide the process.
   
   When the plan runs, create the security/ directory if it does not exist and
   save the findings to security/audit-findings.md
   
   Save the plan to plan/audit-plan.md

3. Bob might ask follow-up questions to clarify the scope of the audit or the specific areas to focus on. You can answer them or tell Bob to use your recommendation.

4. Open the plan file to understand the audit approach and what Bob will do when you run it.

Audit the codebase

1. Click the + button to start a new task.

2. Make sure you are in Agent mode in the chat interface.

   Agent mode gives Bob full capabilities, including file writing and execution. This is necessary for the audit and report generation phases.

3. Click the Permissions selector in the chat interface and check the Read, Edit, Execute, and Skill checkboxes. Leave all other toggles unchecked for this task.

   Permission	State	Why
   Read	✅ On	Bob reads the codebase, the audit plan, and the skill
   Edit	✅ On	Bob writes the findings to security/audit-findings.md
   Execute	✅ On	Bob may run shell commands to resolve paths or confirm file structure
   Skill	✅ On	The audit plan invokes the asvs-audit skill
   MCP	❌ Off	Not needed for local code analysis

4. Ask Bob to implement the audit plan.

   Implement the @plan/audit-plan.md

5. Review the findings in security/audit-findings.md.

   The plan tells Bob to create the security/ directory if it does not already exist and save the findings to security/audit-findings.md.

   Saving the findings lets you start a new chat with a model using a fresh context window. You can point Bob at the findings file to generate reports without re-reading the entire codebase and skill instructions, which preserves the context window for report generation.

   Context window note: All models have a set context window. When auditing a large codebase, you may exceed a model's context window. For large codebases, try auditing one ASVS category at a time. Run V4 first, then V5, V13, and V14, and ask Bob to consolidate findings at the end. This is also a good reason to keep SKILL.md concise and to use focused @ context mentions rather than directing Bob at the entire repository at once.

Generate security reports

SARIF is the standard interchange format for static analysis results. IDEs including Bob, GitHub Advanced Security, and most CI/CD pipelines can consume SARIF files directly.

1. Click the + button to start a new task.

2. Make sure you are using Agent mode.

3. Click Permissions in the chat panel and check the Read, Edit, and Execute checkboxes. Leave all other toggles unchecked for this task.

   Permission	State	Why
   Read	✅ On	Bob reads the findings in security/audit-findings.md
   Edit	✅ On	Bob writes the SARIF report to the security/ directory
   Execute	✅ On	Bob may run shell commands to resolve paths or confirm file structure
   Skill	❌ Off	Not needed to produce reports. The skill already created the findings needed by the agent
   MCP	❌ Off	Not needed for local code analysis

4. Ask Bob to generate a SARIF report using an @ context mention.

   @security/audit-findings.md
   
   Generate a SARIF 2.1.0 report from the audit findings.
   
   Save it to `security/audit-results.sarif`.
   
   Include:
   - Tool name: "ASVS Security Audit"
   - A rule entry for each ASVS control that was checked, with the control ID and description
   - A result entry for each finding, with severity level, file path, line number, and the fix recommendation in the message field

   Bob generates the file and saves it to security/audit-results.sarif. Confirm the file contains a runs array with results entries, one per finding from the audit.

   Bob also reports the mapping decisions it made in the chat:

   Example output:

   Severity mapping used: Critical/High → SARIF error; Medium/Low → SARIF warning.
   The message.text for each result contains the full issue description and the fix
   recommendation in one field, so tooling that renders SARIF (GitHub Code
   Scanning, VS Code SARIF Viewer, etc.) will surface the remediation guidance
   inline.

5. Open security/audit-results.sarif in Bob to review the findings. You can use this file in other tools such as GitHub Advanced Security or a CI/CD pipeline to show the audit findings.

Generate an OSCAL remediation plan (POA&M)

The OSCAL POA&M is a machine-readable JSON document that maps each finding to a structured remediation task with risk information, implementation guidance, and milestone assignments. Bob can read this file as a work queue. It works through each item, applies fixes, and marks milestones complete as it goes.

1. Click the + button to start a new task.

2. Click the mode menu in the chat panel and select Agent.

3. Click Permissions in the chat panel and check the Read, Edit, and Execute checkboxes. Leave all other toggles unchecked for this task.

   Permission	State	Why
   Read	✅ On	Bob reads the findings in security/audit-findings.md
   Edit	✅ On	Bob writes the OSCAL POA&M to the security/ directory
   Execute	✅ On	Bob may run shell commands to resolve paths or confirm file structure
   Skill	❌ Off	Not needed to produce reports. The skill already created the findings needed by the agent
   MCP	❌ Off	Not needed for local code analysis

4. Ask Bob to generate an OSCAL POA&M report using an @ context mention.

   @security/audit-findings.md
   
   Generate an OSCAL Plan of Action and Milestones (POA&M) from the audit findings.
   
   Save it to `security/poam.json`.
   
   For each finding include:
   - A unique UUID
   - The ASVS control ID as the finding reference
   - Severity and a one-sentence risk description
   - A concrete remediation task with enough detail for an AI agent to implement it without additional context — include file path, line reference, and the specific change required
   - A milestone label based on severity: Critical and High findings get "sprint-1", Medium and Low get "sprint-2"
   
   Use OSCAL version 1.1.2 structure.

   Bob generates the file and saves it to security/poam.json.

   Example output:

   Each poam-item contains:
   
   props — severity, asvs-control, and milestone label
   risks[] — one risk with a uuid, title, one-sentence risk description, and status: "open"
   
   remediations[] — one remediation with a lifecycle: "planned" flag, a title, and a description 
   that is specific enough for an AI agent to implement without additional context 
   (includes exact file paths, line numbers, and the concrete code change required)
   
   tasks[] — one or more discrete action tasks under each remediation (e.g. the IDOR finding breaks 
   out the short-term ownership check and the long-term UUID migration as separate tasks)

You now have two artifacts in the security/ directory:

• audit-results.sarif — structured findings for developer tooling and CI/CD integration
• poam.json — a remediation work queue for AI-assisted or human-driven remediation

Review findings

With the POA&M generated, review the findings and prioritize which ones to fix first.

1. Click the + button to start a new task.

2. Click the mode menu in the chat panel and select Ask.

3. Ask Bob to highlight the most critical items from the POA&M using an @ context mention.

   @security/poam.json
   
   Review this remediation plan and identify the 3 most critical items to fix first.
   
   For each one include:
   - What the vulnerability is and why it is the highest priority
   - Which file and line to look at
   - What the fix involves and roughly how long it would take a developer to implement

   Bob reads the file and returns a prioritized summary:

   Example output:

   #1 — No Authentication on Any Endpoint (Finding 1 · ASVS V4.1.5 · critical)
   Why it's the highest priority:
   This is the foundational flaw that makes every other vulnerability worse. With
   zero authentication in place, any anonymous caller on the network can execute
   all booking operations — book flights, read all user records, cancel any
   booking, register accounts. Until this is fixed, findings 2 and 3 also have no
   meaningful barrier.
   
   ...snip...
   
   #3 — PII in URL Query Parameters on GET /user (Finding 5 · ASVS V13.1.3 · high)
   Why it's third:
   Email addresses (PII) appearing in GET /user query strings are passively and
   silently leaked into server access logs, reverse-proxy logs, CDN caches, and
   browser history — a persistent, silent data breach that can violate GDPR. Unlike
   findings 2 and 3, this is an architectural leak that accumulates over time;
   every request compounds the exposure.
   
   Sprint-1 Fix Order Summary
   
   Priority	Finding	Severity	Effort	Dependency
   1	Auth on all endpoints (V4.1.5)	Critical	2–4 h	None — fix first
   2	Wildcard CORS (V14.5.3)	High	30 min	Independent — fix in parallel
   3	PII in query string (V13.1.3)	High	1–2 h	Independent — fix in parallel
   
   Findings 2 and 3 are blockers in the ownership-check fixes (Findings 2 and 3 in
   the POAM) that come afterward, since those require a known caller identity —
   which only exists once authentication (Finding 1) is in place.

   You can also integrate a Model Context Protocol (MCP) server into your audit workflows and ask Bob to create tickets for each finding, linking them to the relevant code and including the remediation guidance from the POA&M.