A scripted, tested stop control for agents that reach their tools through a governed MCP instance on MuleSoft Flex/Omni Gateway. One API Manager PATCH flips the existing MCP Global Access policy to block every tool; a second restores the saved rules. No new policy, no redeploy, no change to the upstream server. Includes a read-only watcher that measures time-to-blocked (within 11 s in testing, upper bound), how already-open sessions behave, fail-closed scripting, and an honest scope: it is a manual switch, not automatic prompt-injection detection.
Tags: mcp, flex-gateway, omni-gateway, api-manager, agent-governance, ai-agents, api-security, incident-response, cross-industry
Agent Kill Switch for a Governed MCP Server: One Policy Edit, No Redeploy
Deliverable: A tested, scripted kill switch for a governed MCP API instance on MuleSoft Flex/Omni Gateway. One API Manager call flips the instance's existing MCP Global Access policy to "block every tool"; a second call restores the saved rules. Agents on that instance are refused within seconds, with no new policy, no redeploy and no change to the upstream MCP server.
Objective: When an agent misbehaves (a prompt injection, a bad instruction, a partner agent you do not control), security teams ask one question: how fast can you stop it? Most answers involve revoking credentials, redeploying an app or calling the vendor, which takes minutes to hours and often takes out unrelated traffic. If the agent reaches its tools through a governed MCP instance, the stop control already exists: the Global Access policy that hides destructive tools can also refuse all of them. Working looks like a runbook step an on-call engineer can run from a terminal, with a measured time-to-blocked, a status check, and a clean restore.
What's Getting Built
- Governed MCP instance (API Manager, MCP server type, already in place) — the agent's only path to the upstream MCP server ·
POST /{base-path}/ (JSON-RPC initialize, tools/list, tools/call)
- MCP Global Access policy (
mcp-global-access-policy, already applied) — the policy whose rules the switch edits · GET /apimanager/api/v1/organizations/{orgId}/environments/{envId}/apis/{apiId}/policies, GET …/policies/{policyId}, PATCH …/policies/{policyId}
- Kill-switch script (operator tool) —
status, on, off; dry-run by default, writes only with an explicit apply flag · uses POST /accounts/api/v2/oauth2/token (client credentials)
- Time-to-blocked watcher (operator tool, read-only) — probes the instance with a harmless read-only tool call every few seconds and reports elapsed time until the wanted state
This blueprint does not rebuild the policy chain itself. The companion blueprint Defense-in-Depth Policy Chain for MCP Servers You Don't Own covers the instance, the eleven policies and their order. This one adds the stop control on top.
Prerequisites
- A governed MCP instance with the MCP Global Access policy already applied. If it is not there, apply it first with your normal rules; the switch edits an existing policy and never creates one mid-incident.
- An Anypoint Connected App (client credentials) scoped to API Manager policy management in the one environment that hosts the instance. Store its ID and secret in a vault or environment variables, never in the script.
{orgId}, {envId} and {apiId} for the instance, recorded in the runbook ahead of time. Looking them up during an incident is where the minutes go.- A named owner who may turn the switch on, and an agreed rule for who turns it off. Treat this as a change-management decision, not a technical one.
- A safe probe: one read-only tool call with a known, recognisable response (for example a single-record lookup whose phone field is masked by the PII policy).
Choose your path
- Instance-wide switch (built and tested here) — Choose this when each agent, or each trust boundary, has its own MCP instance. Blocking
.* refuses every tool for every client on that instance. It is the fastest control and the easiest to reason about.
- Per-client switch — Choose this when many agents share one instance and you must stop only one. Add a deny rule for that client's identity to the instance's MCP Access Control (Cedar) policy instead. It is more precise, but it was not timed here; measure it with the same watcher before relying on it.
Architecture
1. Record the normal rule set
- Read the instance's policies and find the one whose
assetId (or template.assetId) is mcp-global-access-policy. Keep its policyId in the runbook.
- Save the current
configurationData.rules to a file before every on. A typical normal set allows everything and blocks the destructive tool by exact name: Allow pattern .*, then Block literal {write-tool}.
- Restore from the saved file, not from a hard-coded copy. A hard-coded "off" silently undoes any rule someone added since, and on a shared instance that is a security regression.
2. Build the switch as an edit, not a new policy
on sets the rules to a single Block pattern .*. off writes back the saved rules. Both are one PATCH …/policies/{policyId} with body {"configurationData":{"rules":[…]}}.- Editing the existing policy is what makes it fast. Adding or removing a policy changes the chain order and can take longer to propagate; redeploying a proxy drops in-flight sessions for every client.
- Make
status read-only and the default. It prints the current rules and KILL SWITCH: ON only when a Block .* rule is present.
- Make writes opt-in (
on --apply, off --apply). Without the flag, print the rules that would be written and exit. Operators rehearse with the dry run.
- Fail closed: if the token call fails, the policy is not found, or the PATCH returns anything but
2xx, print the error message and exit non-zero. Never report success on an unverified write.
3. Measure time-to-blocked with a read-only watcher
- Loop: run the probe, classify the response, print elapsed seconds, sleep 3 s. Stop when the wanted state (
blocked or open) is reached, or after a maximum (300 s is a sensible default).
- Classify by content, not status code. Blocked: the tool call returns JSON-RPC error
-32008 with Access denied to: {probe-tool}, and a new session's tools/list is empty. Open: the probe returns the record with its masked field, for example (*******101.
- Run
watch blocked straight after on --apply and watch open straight after off --apply. The reported time is an upper bound: each probe mints a token and opens a fresh MCP session, which adds several seconds.
4. Know what the switch does to sessions that are already open
- A new session sees zero tools as soon as the change lands.
- A session that started before the switch can still hold its old tool list. The agent will try a tool it believes it has, and the gateway refuses the call. Expect the user-facing message to be the agent's paraphrase of
Access denied, not a clean "tool missing".
- Agents may refuse destructive requests on their own before the gateway is involved. That is fine, but do not count it as the control. The switch must hold for an agent that does try.
5. Put it in the incident runbook
- Trigger: any agent behaviour the owner judges unsafe. The switch is manual. Nothing here detects prompt injection or suspends an agent automatically. Say so in the runbook and in any customer conversation.
- Sequence:
status → save rules → on --apply → watch blocked → investigate → off --apply → watch open → status.
- Add a preflight check that fails if the switch is on. A forgotten switch looks like an outage to every other agent on the instance.
6. Security and compliance
- The switch is a privileged write to API Manager. Give the Connected App the narrowest scope that can edit policies in that one environment, and rotate its secret after every exercise in which it was used.
- Read credentials from environment variables or a vault. Never print the Anypoint bearer token, and never pass it on a command line where it shows in the process list.
- Log every
on and off with who ran it, when, the rules before and after, and the HTTP status. Message Logging on the instance already records the refused calls, so the two logs together give the incident timeline.
- The switch blocks at the gateway. It does not revoke the agent's upstream credentials. If those may be compromised, rotate them as a separate step. Blocking the gateway path only helps if the agent has no direct route to the upstream.
- Turning the switch off restores the full chain, including PII masking. Confirm the masked probe field before closing the incident.
Sample payload — the switch's PATCH body and the refused call
{
"configurationData": {
"rules": [
{ "rule": "Block", "match_type": "pattern", "value": ".*" }
]
}
}
Restore (the saved normal set):
{
"configurationData": {
"rules": [
{ "rule": "Allow", "match_type": "pattern", "value": ".*" },
{ "rule": "Block", "match_type": "literal", "value": "{write-tool}" }
]
}
}
A read-only call while the switch is on:
{ "jsonrpc": "2.0", "id": 2, "error": { "code": -32008, "message": "Access denied to: {probe-tool}" } }
Deterministic checks
status makes no writes and reports ON if and only if the Global Access rules contain Block with value .*.on and off without the apply flag make no writes; with it, each returns 2xx or the script exits non-zero with the API's error message.- After
on --apply, the watcher reaches blocked within the agreed bound. After off --apply, it reaches open, and the probe's PII field is still masked.
off writes back exactly the saved rule set, byte for byte; diff the saved file against status output.- While the switch is on, a new session's
tools/list returns zero tools and every tools/call returns -32008, including read-only tools.
- No token or secret appears in script output, logs or shell history.
Testing and validation
- Happy path:
on --apply, watch blocked, then off --apply, watch open. Measured on a governed hosted-SaaS MCP instance: HTTP 200 on both writes, blocked within 11 s and open again within 11 s, including the probe's own token and session startup.
- The edge case that breaks a naive build: an agent with a session opened before the switch. Its cached tool list still names tools, so test that the gateway refuses the call. Do not stop at checking that
tools/list is empty.
- Second edge case: someone edits the Global Access rules between
on and off. Prove the restore uses the saved file and that the diff is empty.
- Rehearse with the agent client people will actually use, start to finish, and record how it words the refusal. Users see that sentence, not the JSON-RPC error.
Phased rollout
Phase 1 — Connect
- Create the Connected App, run
status against a non-production instance, and confirm the policy ID and rule format.
Phase 2 — Map and validate
- Pick the probe, build the watcher, and time
on and off five times on non-production. Record the worst case as the published bound. Add the preflight "switch is off" check.
Phase 3 — Production
- Store IDs in the runbook, grant the switch to the on-call role only, and run one scheduled exercise per quarter in a low-traffic window, timed with the watcher.
Success metrics
- Time from
on --apply to first refused call, measured by the watcher as an upper bound (11 s observed in testing; set your own bound from your runs).
- Time to restore, and whether the restored rules match the saved set (target: always).
- Number of incidents or exercises where the switch was left on past the planned window (target: zero).
- Mean time to contain agent incidents before and after the runbook, from your incident tracker.
Troubleshooting
status says "no mcp-global-access-policy on API" — Likely cause: wrong {apiId} or environment, or the policy was never applied. Fix: list policies and check assetId; apply Global Access with normal rules before an incident, never during one.- PATCH returns 4xx — Likely cause: the Connected App lacks policy-edit scope in that environment, or the body is not wrapped in
configurationData. Fix: check scopes; send {"configurationData":{"rules":[…]}}.
- Watcher says
unknown — Likely cause: the probe's token expired or the upstream failed, so the response matches neither state. Fix: run the probe once by hand; refresh the upstream credential.
- Agent still answers after the switch is on — Likely cause: it is answering from earlier context, not calling a tool, or it has a second route that bypasses the gateway. Fix: ask a question that needs a fresh tool call; remove any direct connector.
- Everything is refused after the incident — Likely cause: the switch was left on. Fix:
status, then off --apply from the saved rules; add the preflight check so it cannot happen silently.
- Restore removed a rule someone added — Likely cause:
off wrote a hard-coded rule set. Fix: restore from the file saved just before on.