Why this verdict
- suspicious: finding skill:skills/qawolf-cli:skills/qawolf-cli/SKILL.md:246:SUPPLY
- suspicious: finding skill:skills/qawolf-cli:skills/qawolf-cli/SKILL.md:359:SUPPLY
- suspicious: finding skill:skills/qawolf-cli:skills/qawolf-cli/SKILL.md:92:SUPPLY
Security findings 3
Supply chain
highContextual riskhigh confidenceThe skill instructs the agent to fetch its runner guide from an unpinned mutable URL (the qawolf/cli repository's main branch) when no CLI is available, so the agent follows whatever instructions that URL serves at runtime; this remote fallback is open but unnecessary given the artifact ships the same text under references/.
https://raw.githubusercontent.com/qawolf/cli/main/skills/qawolf-cli/references/runner.md
trigger:conditionalvector:instruction
Counter-argument considered
Strongest refutation was that the artifact ships the identical text at references/runner.md (confirmed at that path), making the URL a redundant fallback. But the fallback instruction remains in the skill body and sends the agent to an unpinned main-branch URL whose content it must then follow at runtime; the shipped copy does not neutralize the instruction, and no carve-out covers fetching agent instructions from raw.githubusercontent.com, so the finding stands.
Supply chain
highContextual riskhigh confidenceThe skill instructs the agent to fetch its run-results reference from an unpinned mutable URL (the qawolf/cli repository's main branch) whenever the local path is absent, so the agent follows whatever instructions that URL serves at runtime; this undeclared-in-residue remote fallback is open but not required by the declared purpose, since the artifact ships the same file under references/.
https://raw.githubusercontent.com/qawolf/cli/main/skills/qawolf-cli/references/run-results.md
conceal:remotetrigger:conditionalvector:instruction
Counter-argument considered
I looked for a mitigation in the shipped references/run-results.md or in the declared claims, but none neutralizes the fallback: when the local path is absent the agent is told to fetch and follow whatever the unpinned main-branch URL serves before reporting on run outcomes. That the instruction is stated openly does not carve it out, and reading a run's results is declared while fetching agent instructions from raw.githubusercontent.com is not, so the finding stands.
Supply chain
mediumContextual riskmedium confidenceThe commands table instructs the agent to fetch and follow QA Wolf platform-delivered skill instructions (skill list, then skill get, whose reply carries a SKILL.md and its references) before any QA Wolf work, so the agent may execute mutable server-side instructions the artifact does not ship and its declared description does not cover.
Read the skill before starting the work it covers and follow it.
trigger:conditionalvector:instruction
Counter-argument considered
I first argued the carve-out of declared core capability: skill list/get are read-only commands of the wrapped service and the platform skills cover declared QA Wolf work. But the table goes beyond documenting a CLI call: it directs the agent to treat the platform-returned SKILL.md as authoritative instructions to follow, content the artifact never ships and the declared description never delegates, and openness in the table is not a carve-out, so the finding stands.
Open questions 2
What the analysis could not settle from the code alone.
Contradiction C10 (DET:fd7645436d9a7ab1) was adjudicated, not left open: the claim D63 that QA Wolf environment variable 'values never leave the server' concerns the platform's environment-variable commands, while the observed capability is an example of the declared runner feature navigating a cloud browser to https://example.com/login, which sends no environment values anywhere and breaks no claim; the detector lead is therefore rejected as a keyword-overlap false positive and no finding is fi⟦… 11 more chars …⟧
SKILL.md documents QAWOLF_HOST_URL as retargeting all API-bearing commands, which carry QAWOLF_API_KEY, to an arbitrary deployment host; whether this configurable credential destination should be weighed as an exfiltration risk or accepted as standard service configuration (like AWS_ENDPOINT_URL) depends on the wrapped CLI's request behavior, which is outside this pass's evidence.
Data exfiltration
Coverage gaps 3
Parts of the artifact the static analysis could not fully examine.
context:unanalyzable_binary binary content is only byte-scanned
src/shell/interactiveRunner/collectRunFiles.ts
ledger_finding_without_valid_finding run 1: declared-vs-actual.1 SRC:d3c49738ef6d5898
invalid_judge_finding declared-vs-actual.1
