MCP server exposing Buildkite API data (pipelines, builds, jobs, tests) to AI tooling and editors.
Risky behavior the analysis could not rule out.
. Suspicious.agents/skills/release-notes No issues found.agents/skills/tool-parameter-descriptions No issues foundThe report_issue prompt tells the agent to compile conversation-derived tool call details into a report and offer to post it to a public GitHub issue through the gh CLI. Posting happens only after offering, and secrets are redacted by instruction.
offer to attempt to post it on the user's behalf, if an authenticated GH CLI session is available
Developer docs tell the user to run an unpinned @latest npx package (MCP Inspector) fetched at runtime and pointed at the locally built server, which holds the Buildkite token.
npx @modelcontextprotocol/inspector@latest ./buildkite-mcp-server stdio
The docs instruct an unpinned @latest npx fetch, which is not the pinned-install carve-out. The package is official and optional, so severity is low, but the finding is accurate.
What the analysis could not settle from the code alone.
Candidates the review stage examined and rejected, kept for transparency.
The Cursor Cloud MCP definition clones a GitHub repo (variable-controlled MCP_SRC_REPO/SHA), builds it and execs it in a remote VM with the Buildkite API token in its env.
git clone --quiet https://github.com/${MCP_SRC_REPO}.git /tmp/bk-mcp-srcThe VM clones the project's own repo (default buildkite/buildkite-mcp-server) at the exact commit under review to build the server being evaluated. This is the documented purpose of mcp_version: source. Only the operator can override the repo.
CLI-mode evals fetch a binary or archive from a configurable https download_url and put it first on the agent's PATH, so mutable remote code would execute in the eval run.
put first on the agent's PATH
The download_url is a documented, optional eval-config feature set in the committed matrix for bk CLI version comparison. The default is a pinned binary installed off PATH. This is the harness's declared purpose, not a concealed fetch.
For cursor-cloud evals the harness hands the entry's Buildkite API token to Cursor's cloud as an encrypted envVar, a disclosed token custody shift to a third party that the README says is mitigated with read-only, single-purpose tokens and deletion of the agent afterwards.
the entry token as a session-scoped encrypted `envVar`
Passing the entry token to Cursor's cloud is the declared core mechanism of cursor-cloud evals. The README discloses the custody shift and mitigates it with read-only, single-purpose tokens and deletion of the agent afterwards. It is not concealed exfiltration.
cursor-cloud.sh passes the entry's BUILDKITE_API_TOKEN as an environment variable in the launch request to Cursor's hosted cloud agent, a third party. The README documents this as a custody shift, and the agent has unrestricted shell access to it in the VM. It is documented and used for the eval's declared purpose, but it is still a credential sent off-host.
envVars: {BUILDKITE_API_TOKEN: $tok},The script sends the entry token to Cursor's cloud agent so the eval can run there. This is the script's declared purpose and mirrors the other runners. The token is read-only and the agent is deleted afterwards. It is not concealed.
cursor-cloud.sh substitutes the entry's BUILDKITE_API_TOKEN into the inline MCP server definition sent to the Cursor Cloud API (api.cursor.com), so the credential leaves the host for a third-party VM. This is by design, and the agent is deleted afterwards.
then gsub("\\$\\{BUILDKITE_API_TOKEN\\}"; $tok)Substituting the token into the inline MCP definition is the documented way the cloud-hosted MCP server authenticates to Buildkite. It is the script's declared core function with a read-only token, not an undeclared exfiltration.
yilong.sh posts eval evidence (goal, agent transcript, diff) to the third-party api.typesafe.ai endpoint with a bearer token. Redaction is applied and this is the documented evaluator purpose.
--data @"$req"
Posting redacted eval evidence to TypeSafe's API is the declared purpose of the yilong evaluator. The Jev token is kept out of the agent container and the evidence is redacted before it is sent.
The eval pipeline maps GitHub, Buildkite API and Cursor tokens into the container of an agent with unrestricted Bash, which could read and exfiltrate them; the pipeline's own comments acknowledge this risk.
evaluated agent runs in this container with unrestricted Bash
Eval CI deliberately maps the service tokens the eval needs. The pipeline's own SECURITY note discloses the risk and keeps other secrets (Datadog, Jev) out of the agent step. No concealed chain or off-purpose use is shown.
The eval CI pipeline runs the evaluated agent with permission checks bypassed (bypassPermissions) and unrestricted Bash inside a container that also receives GitHub, Buildkite and Cursor tokens, so the agent under test could read them; the pipeline's own comments acknowledge this.
always runs bypassPermissions
bypassPermissions is the deliberate, documented posture of the eval harness, which measures tool choice. Containment is documented as a throwaway container, a read-only MCP server and a read-only token. It is the declared core capability, not a concealed escalation.
Eval harness docs describe inspecting the user's ~/.cursor/mcp.json agent configuration to decide whether to skip a cli-mode entry, touching agent config beyond the repo.
`~/.cursor/mcp.json` defines any servers
The README only describes a defensive check: the entry is skipped if ~/.cursor/mcp.json defines servers or cannot be parsed. This stops user config leaking into the run. Nothing reads credentials or sends them anywhere.
Every tool's input schema embeds a 'telemetry.context' parameter that asks the agent to explain its reasoning relative to the user's goal for analytics and intent tracking, a mildly undeclared channel that carries conversation-derived intent into telemetry spans, with the schema itself noting the server does not classify sensitive content.
Explain why calling this tool fits the user's overall goal.
The telemetry.context parameter is openly stated in every tool schema. The description says it supports analytics and intent tracking and tells the agent to avoid credentials and personal data. It is a disclosed feature of the server, not a concealed channel.
The klaren review prompt declares allowed-tools including Bash, Edit and Write, although its body says 'Review only: do not change code, push, retry builds, or perform other writes'. The tool grant is broader than the stated review-only purpose.
allowed-tools: Bash, Read, Edit, Write, Glob, Grep
The finding rests only on a broad allowed-tools list in frontmatter, which is the broad-permissions-alone carve-out. No instruction or chain misuses those tools.
The eval harness documentation states that in CI the evaluated LLM agent runs with permission checks bypassed and no tool allowlist, and locally with LOCAL_BYPASS_PERMISSION=true it gets unrestricted Bash on the host; this is disclosed and deliberate, with containment documented as a throwaway container, a read-only MCP server and a read-only token.
--permission-mode bypassPermissions
The README documents a deliberate permission posture for measuring agent tool choice. It lists containment layers and requires an explicit choice for local runs. This is the declared eval capability, not a hidden bypass.
The eval CI step runs babystand.sh, which clones and pushes to a default third-party fork repo (EVAL_REPO_SLUG) with GITHUB_TOKEN mapped into the container, so the script's behavior and the fork's content are unverifiable here.
babystand.sh clones EVAL_REPO_SLUG (default: the nethsix fork), pushes
The finding only speculates that the maintainers' own eval CI uses an external fork. The pipeline comments openly describe this cross-repo design, and nothing shows malicious behavior or a source-to-sink chain. The unshown script is CI tooling that MCP clients never consume.
Parts of the artifact the static analysis could not fully examine.
no_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_go go file: string-literal extraction onlyno_dataflow_for_dynamic_code line 73: invoked code is not structurally extractedno_dataflow_for_dynamic_code line 221: invoked code is not structurally extractedno_dataflow_for_dynamic_code line 77: invoked code is not structurally extractedno_dataflow_for_dynamic_code line 98: invoked code is not structurally extractedno_dataflow_for_dynamic_code line 171: invoked code is not structurally extracted