Skip to content

These pages describe PrivacyFence 8.0.0. Download it · What changed

Upstream MCP servers behind PrivacyFence

An organization server can put another MCP server (an "upstream", for example a knowledge hub) behind PrivacyFence. The operator declares which of its tools exist and how each is gated; PrivacyFence enforces that tool policy with cards, scoped tokens and the audit log, like a built-in connector. Every call PrivacyFence forwards carries a short signed claim, bound to that one call, that names the person, the audiences they hold and what is calling; the upstream filters its own queries by it.

Why it is built this way: ADR 0285, ADR 0297, ADR 0298 and ADR 0299.

For operators

Organization servers only. In single-user mode a non-empty PRIVACYFENCE_UPSTREAM_MCP_FILE or PRIVACYFENCE_UPSTREAM_MCP refuses the start. On a Docker install see Upstream MCP file.

The operator file

Set PRIVACYFENCE_UPSTREAM_MCP_FILE to the path of a YAML or JSON file (see the configuration reference). A name ending in .yaml or .yml is read as YAML, any other name as JSON. Or put the same text in PRIVACYFENCE_UPSTREAM_MCP (always read as YAML, so one-line JSON works too). Set one of the two, never both: both refuse the start. The declaration is read once, at startup; a change to it needs a restart.

{
  "version": 1,
  "upstreams": [
    {
      "id": "kh",
      "label": "Knowledge hub",
      "url": "https://kh.example.internal/mcp",
      "audience": "https://kh.example.internal",
      "tools": {
        "search":         {"gate": "auto",  "verb": "search"},
        "get_doc":        {"gate": "auto",  "verb": "read"},
        "propose_change": {"gate": "popup", "verb": "add",    "effect": "Adds a change proposal to the knowledge hub."},
        "write_derived":  {"gate": "popup", "verb": "update", "effect": "Writes a derived document to the knowledge hub."}
      }
    }
  ]
}

The same declaration in YAML (quote an id or tool name that YAML reads as a boolean, such as "no" or "on"):

version: 1
upstreams:
  - id: kh
    label: Knowledge hub
    url: https://kh.example.internal/mcp
    audience: https://kh.example.internal
    tools:
      search: {gate: auto, verb: search}
      propose_change: {gate: popup, verb: add, effect: Adds a change proposal to the knowledge hub.}

The AI client sees each declared tool as <id>_<tool>, here kh_search and kh_propose_change. Any mistake in the declaration refuses the start; the message names the path of the bad value (PRIVACYFENCE_UPSTREAM_MCP_FILE: upstreams[0].tools.search.gate: ...) and never quotes the value.

Rule Detail
Keys version must be 1. Only version, upstreams, claim and membership at the top; only id, label, url, audience, tools, identity_claim, claim_lifetime_seconds, allowed_principal_types, email_for_visibility, audiences_tool and audiences_refresh_seconds in an upstream (an old grants key is refused with grants were replaced by groups); only gate, verb, effect in a tool. Anything else is refused.
Size At most 8 upstreams and 50 tools per upstream; the file or inline text is at most 256 KiB.
Repeated key A key that appears twice in one mapping or object is refused (the file repeats a key. or the text repeats a key.), so a repeated tool name cannot silently keep the last gate.
Format A YAML file is checked by the same rules as JSON; a wrong top level still reads must be a JSON object.
id 2-16 characters, lowercase letters and digits, starting with a letter; unique. It may not collide with a built-in connector name (such as gmail, drive, privacyfence): upstream id "<id>" collides with the built-in connector "<name>".
label 1-60 characters, no control characters. It names the server on cards and in the tool descriptions.
url https:// with any host, or http:// only for localhost, a loopback address or a single-word host name (a container on the same Docker network, such as http://knowledge-hub:8080/mcp). No user name, no fragment.
audience Required, 1-200 characters, no whitespace. It becomes the aud of every claim; the upstream accepts only claims with its own audience. Not to be confused with the audiences a person holds.
identity_claim required (the default) signs every call; off sends no claim and resolves no caller.
claim_lifetime_seconds 5-60, default 60: how long a claim is valid.
allowed_principal_types A non-empty list from claude_client (a person's AI client) and n8n_workflow (any scoped token); default both. A caller of another type is refused before anything is sent.
email_for_visibility true or false, default false. With true the claim carries the person's verified, lower-cased email, for use in the upstream's own visibility rules (see the contract). Refused with identity_claim: off and for any value that is not a boolean; a YAML file reads yes and on as true.
audiences_tool The name of a tool of the upstream that tells PrivacyFence who holds each audience (see Audiences from the upstream). 1-48 letters, digits, underscores or hyphens; not also a key of the same upstream's tools (is also declared under tools; the audiences tool is called by PrivacyFence only); refused with identity_claim: off (needs identity_claim required).
audiences_refresh_seconds A whole number from 30 to 3600, default 300: how often PrivacyFence asks again. Only together with audiences_tool (is only allowed with audiences_tool); anything else reads must be a whole number from 30 to 3600.
tool name 1-48 letters, digits, underscores or hyphens; <id>_<tool> is at most 64 characters and must not collide with an existing tool.
gate auto (runs at once), review (runs, then the person sees the result before it is returned) or popup (the person approves before the call is sent).
verb One of the policy verbs. A read verb (read, download, search) takes auto or review and no effect. Every other verb takes popup and must carry an effect: 1-200 characters, ending in a full stop, shown on the card.

So an ungated write cannot be declared, as long as the verb says it is a write.

Declaring a writing tool with a read verb skips the approval card. PrivacyFence cannot see what an upstream tool does, only what you declared. A tool that changes data must carry a verb such as add, update or delete.

PrivacyFence asks the upstream for its tool list once, at startup, and retries with a growing delay until it answers. It never fetches the list again, even when the upstream has an audiences tool (that tool is called on an interval, the list is not): restart PrivacyFence to pick up a change in the upstream's tools or descriptions. A declared tool the upstream does not list, or lists with a schema PrivacyFence cannot pass on (not an object schema, a reason property of its own, over 32 KB, or any $ref, $defs or definitions), is skipped and logged. Each exposed tool gets one extra required reason parameter, like every PrivacyFence tool. The schemas the upstream sends are passed to AI clients as they are, so whether a given client accepts a nested type or an enum is the operator's responsibility to check.

Approval cards show the server, the tool and the argument names, never the argument values. A popup tool's card also offers the arguments' text for review. Upstream tools have no "Always allow" button and no auto-accept rules yet.

Groups and audiences

An audience is the upstream's own label for a slice of its data (all, mgmt, finance). A person holds the audiences of the Workspace groups they belong to. The map from groups to audiences is set in admin_config.json under upstream_mcp (on an organization server, org/admin_config.json; see the configuration reference):

"upstream_mcp": {
  "kh": {"groups": {
    "[email protected]": ["all"],
    "[email protected]":     ["mgmt"],
    "[email protected]": ["mgmt", "finance"]
  }}
}
  • The file may be written before the first start, for example by a deployment tool. It must hold "version": 1. A file that holds only some sections is filled with the defaults, and PRIVACYFENCE_ALLOWED_DOMAINS does not seed the allowed domains: set them on Admin → Sign-in.
  • A group name is lower case and email-shaped; an audience is 1-63 characters (a-z, 0-9, _, -, starting with a letter or digit). At most 200 groups per upstream. A person's audiences are the union over every group they belong to.
  • all is the one unrestricted audience: an addition to it applies at once. Every other audience is restricted and needs an administrator's approval for each person (below).
  • Grants were replaced by groups. A grants key in the declaration is refused at start with this message; nothing is seeded from the declaration any more, and admin_config.json holds groups.
  • A group map for an id the declaration does not declare is logged once and ignored. An invalid section is reported as admin_config.json: upstream_mcp.<id>.groups: <problem>; at startup it refuses the start, and in a running server the last good map stays in force and the error is logged.
  • "Verified" matters. Audiences come only from an email address PrivacyFence treats as verified: the identity provider said the address is verified, or the identity provider's trust_unverified_email is on and the address is in an allowed domain (see Who may sign in). The single-user person is verified. An unverified or empty address gets no audiences, so the upstream returns no data for that person.
  • Edits apply without a restart: PrivacyFence re-reads the file when its modification time changes. Admin → Upstream MCP servers shows the map in force but cannot change it; it is a file edit. Do it while no one is saving on the Admin page.
  • An upstream that declares audiences_tool does not use this map at all. Its audiences come from that tool, see below; a map written for it is ignored and logged once at start.

Audiences from the upstream

An upstream that already knows who may see what, such as a knowledge hub with its own teams, can be the source of its own audiences (ADR 0325). The operator adds audiences_tool: <tool name> (and, if wanted, audiences_refresh_seconds) to that upstream in the operator file and restarts. PrivacyFence then calls that tool itself, at startup and every audiences_refresh_seconds (default 300, so every five minutes), with a service claim (contract), with arguments {}. The upstream answers with a JSON object, as structured content or as the text of the result:

{"version": 1, "audiences": {"mgmt": ["[email protected]", "[email protected]"], "finance": ["[email protected]"]}}

Exactly the keys version (the number 1) and audiences; at most 200 audiences (names as above) and 20000 addresses in all; the answer is at most 1 MB. An address PrivacyFence cannot use (not a plain ASCII address) is dropped and counted, never a reason to reject the answer. A person's audiences on that upstream are the audiences whose lists hold their verified email.

  • No approvals. The upstream decides who holds each audience, all included, and there is no cap on additions and no parking. The group map in admin_config.json and Admin → Group membership are not used for that upstream. Other upstreams are untouched, and an upstream's answer never reaches another upstream's audiences.
  • Refresh and stale rule. A failed refresh keeps the last answer and is retried with a growing delay (never longer than the refresh interval). An answer older than three refresh intervals gives nobody any audience until the next success, and before the first success nobody holds one. So a removal at the upstream reaches PrivacyFence within one interval normally, and within three intervals plus the 60 second call timeout while the upstream is down (3 hours 1 minute at 3600 seconds).
  • A scoped token still holds only the audiences its creator holds now, narrowed by the ones ticked at creation. A token made before the switch keeps its ticked names, which are now read against the new source.
  • The tool list is still fetched once; the upstream should list its audiences tool, but PrivacyFence never exposes it to AI clients.

Operator keys

Besides the per-upstream keys in the table above, the operator file has two top-level objects:

Key Meaning
claim.key_overlap_hours 1-720, default 24: how long a rotated key stays in the key set after it stops signing.
membership.source snapshot (the default: membership comes from the sync tool) or static.
membership.file With static only: the path of a JSON file {"version": 1, "groups": {"[email protected]": ["[email protected]"]}}, re-read when it changes. No approvals; for a first try and for tests.
membership.max_additions 1-1000, default 20: how many additions one snapshot and the last 24 hours together may hold before the snapshot is rejected.

identity_claim, claim_lifetime_seconds, allowed_principal_types and email_for_visibility are described in the table of rules above. A change to any of these needs a restart.

Group membership

PrivacyFence does not read your directory. A job of yours (an n8n workflow that lists the Workspace groups) tells it who is in each group by calling the tool privacyfence_update_group_membership with {"groups": {"[email protected]": ["[email protected]", "[email protected]"]}}. Each call is a snapshot of the groups it names. The tool answers {"status": "applied" | "rejected", "added", "removed", "parked", "ignored_groups", "reason"}.

  • Only a scoped token can call it, and only an administrator can create that token. In Settings → Scoped tokens → New token an administrator ticks privacyfence_update_group_membership under "Tools that need no rule". An AI client cannot call the tool or propose a token for it. Keep the secret in the job's credential store, never in a chat.
  • Removals apply at once. A person who is absent from a group in the snapshot loses that group's audiences at the next call.
  • Additions to all apply at once; additions to any other audience wait. The person is parked until an administrator opens Admin → Group membership and approves or declines them for that group and audience (one person, or everyone pending for that group and audience). An approval is only for that group and audience, and is forgotten when the person leaves the group.
  • An upstream with audiences_tool is skipped: its audiences come from its own tool, so the tool stores no member of a group mapped only for it and Admin → Group membership shows no pending row for it.
  • Groups no upstream's map names are ignored and never stored; mapping a group later makes it appear in the next snapshot.
  • Limits. At most 500 groups and 20000 member entries per snapshot. A snapshot is rejected, and nothing is applied, when its additions together with those of the last 24 hours exceed membership.max_additions (the very first snapshot is exempt), or when more than 500 approvals would be pending. A rejection is an audit entry, a log line and a row on the Admin page.
  • Every call, approval and decline is in the audit log with counts and group names, never addresses.

The tool is also listed in the tools reference.

What a person and a scoped token see

  • A person (through an AI client) gets the audiences their groups give them, with principal_type claude_client.
  • A scoped token gets only the audiences its creator ticked, and never more than the creator holds. In Settings → Scoped tokens → New token, tick an upstream tool and a fieldset Audiences for <label> appears with one box per audience you hold, none ticked. A token made without ticking any sees no audience. A token only narrows: if the person later loses an audience, the token loses it too. Its claim has principal_type n8n_workflow and no email.
  • An AI client cannot propose a scoped token that includes an upstream tool: the proposal is refused with Upstream tools cannot be proposed; create the token in Settings.

Letting a token call a gated upstream tool

An upstream tool declared popup or review needs a person's card, and a scoped token has nobody to answer one. An administrator can approve such a tool for one token instead (ADR 0300):

  • In Settings → Scoped tokens → New token, an administrator ticks the tool under "Upstream tools that change data or release results unseen". Nobody else sees that group, and a local server never does.
  • The tick approves the tool for every call of that token, with no card. The tool's effect text is shown next to the tick, with "writes data" for a popup tool and "result not reviewed by a person" for a review one.
  • If the operator changes the tool's declaration (gate, verb, effect, url, audience or identity claim mode), the approval stops working and the call is refused before it reaches the upstream.
  • A review result goes to the token without a person seeing it, but personal data in it still blocks the call.
  • An AI client cannot propose such a token.
  • An administrator who later loses the role keeps the token until it expires or is revoked.
  • In the audit log look for upstream_forwarded with metadata.gate (auto, review or popup), and for auto_accepted whose rule is the token's approval row.

Rotating the claim key

Claims are signed with an Ed25519 key kept in upstream_claim_keys.json in the org directory and published at <PrivacyFence address>/upstream-mcp/jwks. To rotate it, run privacyfence-app --rotate-upstream-claim-key as the account the daemon runs as. The new key is published at once and starts signing five minutes later, so an upstream that caches the key set for five minutes has it first; the old key stays published for claim.key_overlap_hours. --revoke-upstream-claim-key KID removes a key from the key set at once (for a leaked key) and refuses the only signing key. See the configuration reference.

Audit

Every call PrivacyFence forwards to an upstream writes one upstream_forwarded audit entry with the claim's jti, sub, principal type, audiences and req and how the call ended (including a refusal or a signing failure), so your upstream's log can be matched to PrivacyFence's. The claim and the arguments are never logged.

An upstream with an audiences tool adds two decisions, never with an address: upstream_audiences_applied (the first success, a success whose table differs from the last applied one, and the first success after a failure; its summary has the counts of audiences, people and dropped addresses) and upstream_audiences_failed (a failure when the previous attempt did not fail; its summary names the kind: unreachable, too_large, tool_error, claim_failed or invalid). Neither is an approval. The refresh writes no upstream_forwarded entry; the claim's jti of each success is logged at info level.

On the Admin page

Admin → Upstream MCP servers shows each declared upstream: whether its tool list has arrived, its address (only its scheme and host, then /… when it has a path or query), its claim audience and identity-claim settings, who may call it, each declared tool with its gate, verb and effect and whether it is exposed, and the groups mapped to its audiences. For an upstream with an audiences tool the third card is titled <label>: audiences and shows the source (tool and interval), its state (waiting, ready or stale, with the counts of audiences and people), the last refresh time, the last error and how many mapped groups are ignored, never an address. It is read-only: change the declaration and restart, or edit admin_config.json. It never shows the declaration file's path, the inline text or the upstream's own tool descriptions (ADR 0319).

privacyfence_status

On a server with upstreams, privacyfence_status carries an upstreams list:

{"id": "kh", "label": "Knowledge hub", "tools_exposed": 4, "state": "ready", "missing_tools": [],
 "audiences_source": "upstream", "audiences_state": "ready"}

state is waiting_for_tool_list until the first successful tools/list (tools_exposed is 0 and AI clients see no tools from that upstream until then), then ready. missing_tools names declared tools the upstream did not list or whose schema was skipped. audiences_source is groups (the group map) or upstream (the upstream's audiences tool); only with upstream there is an audiences_state: waiting (no answer yet, nobody holds an audience), ready or stale (no answer for over three refresh intervals, nobody holds an audience).

Errors and size

If the upstream cannot be reached, the AI client gets The <label> server could not be reached. A result over 1 MB is refused (ask for less). If the upstream answers with an error, its text reaches the AI client, with control characters removed and cut to 500 characters, like its results do: do not put secrets in an upstream's error messages. Each call opens one short MCP session to the upstream.

For upstream implementers

PrivacyFence sends the header X-PrivacyFence-Claim: <jwt> on every HTTP request to your MCP endpoint, a new claim each time. It never sets Authorization; that stays free for your own use. Treat the claim as the only proof of who is asking; PrivacyFence holds the approvals and the audit trail, you hold the data.

The full contract, with the claim fields, the req binding of a claim to one tool call, the verification checklist and the versioning, is identity-claim.md. The short form:

  • Verify with contrib/verifier/pf_claim_verify.py, one file you can vendor (Python, with PyJWT and cryptography). Give it your audience, PrivacyFence's issuer address and the JWKS address <PrivacyFence address>/upstream-mcp/jwks (Ed25519 keys, cache five minutes, honor the kid). Pass it the raw request body so it can check req.
  • Only sub, audiences and principal_type may change what a query returns. Everything else is for your own audit.
  • A request that is not a tool call carries no audiences. A service claim gets no data, except on a tools/call of the audiences tool the operator declared (see the contract).
  • Filter by audiences inside the query, not afterwards, so ranking and pagination stay correct and a guessed document id is still refused. PrivacyFence never filters your results. A row-level security sketch is in identity-claim.md.
  • Answer 401 for any failure, with no detail.

Reference upstream

scripts/upstream_mcp_reference_server.py is a small upstream for trying all of this and for tests. It is not for production. It verifies claims with pf_claim_verify.py and serves search, get_doc, list_proposals and propose_change over four in-memory documents in the general (two), hr and finance audiences, returning only the caller's audiences; a service claim gets the tool list and no documents. It also lists list_audiences, which answers PrivacyFence's service claim (sub privacyfence) with the table it was built with and refuses everyone else (build_app(audience_members=...) takes that table as a dict, kept by reference and read on every call, so a test can change it between refreshes); use it as audiences_tool to try audiences from the upstream.

Run it against a PrivacyFence server (the Ed25519 keys come from its JWKS address):

python3 scripts/upstream_mcp_reference_server.py --host 127.0.0.1 --port 8099 \
    --audience ref-upstream --issuer https://privacyfence.example.org \
    --jwks-url https://privacyfence.example.org/upstream-mcp/jwks

Use --jwks-file <file> (a saved copy of the JWKS) instead of --jwks-url when the upstream cannot reach PrivacyFence. The --issuer is your PrivacyFence address with no suffix.

The operator file the manual check uses (ref, tools exposed as ref_search, ref_get_doc, ref_list_proposals, ref_propose_change), with a static membership file so no sync job is needed:

{
  "version": 1,
  "membership": {"source": "static", "file": "/etc/privacyfence/membership.json"},
  "upstreams": [
    {
      "id": "ref",
      "label": "Reference upstream",
      "url": "http://refup:8099/mcp",
      "audience": "ref-upstream",
      "tools": {
        "search":         {"gate": "auto",  "verb": "search"},
        "get_doc":        {"gate": "auto",  "verb": "read"},
        "list_proposals": {"gate": "review", "verb": "read"},
        "propose_change": {"gate": "popup", "verb": "add", "effect": "Adds a change proposal to the reference upstream."}
      }
    }
  ]
}

The static membership file says who is in which group:

Map the groups to audiences in admin_config.json under upstream_mcp.ref.groups ("[email protected]": ["general"], "[email protected]": ["finance"]); the file is re-read when it changes, so the next ref_search returns the general and finance documents with no restart. Remove the person from the finance group in the membership file and the next call returns general only. Do this only while no one is editing the Admin page: the server also writes admin_config.json.

Testing

An upstream has no live provider to sign in to and nothing external to drift, so it has no live QA fixture (testing policy). The reference server stands in for it: unit tests drive it in process, and an integration test runs it on loopback against a real organization server.