Databricks: Redact PII in Tool Responses

What this policy does

Direction: egress (tool_post_invoke) Default: allow (transform-only — never denies) Package: databricks.egress.redact_pii

What it does

Scans the response payloads of the Databricks MCP tools that carry lakehouse data back to the agent and rewrites personally identifiable information to fixed redaction tokens before the response is delivered:

Matches are replaced in place, leaving the surrounding row/column structure intact so the agent still gets a usable result set with only the sensitive values masked. The policy is transform-only: it never denies a call, so a legitimate query, Genie question, or index search still succeeds — it just comes back with SSN, email, and phone values masked. Responses with no matches (and all out-of-scope tools) pass through byte-identical. Every response field is read via object.get, so a missing or oddly-shaped payload is never an error — it simply passes through.

Why these tools — the egress surface, not the submit call

Databricks' SQL and Genie tools are async pairs: the submit call (execute_sql, genie_ask) returns only a statement/conversation handle, and the lakehouse rows arrive later through the poll response. So this policy deliberately targets the surfaces where data actually egresses:

• poll_sql_result — the managed Databricks SQL server's async result tool (the rows land here, not in execute_sql).
• genie_poll_response — the Genie One server's natural-language answer, grounded in Unity Catalog data.
• execute_sql_query — the community RafaelCartenet/mcp-databricks-server synchronous SQL passthrough, which returns rows directly in one call.
• AI Search index tools — the managed AI Search server's dynamic {CATALOG}__{SCHEMA}__{INDEX} (double-underscore) tools, whose vector indexes routinely hold support tickets and free-text documents laced with PII.

This pairs with the ingress SQL/schema guards by design: ingress limits what can be asked (DML/DDL denial, schema fencing), egress limits what actually leaks back through the async poll and search surfaces. A read the ingress guard permits can still surface a regulated identifier in its rows — this is the layer that catches it.

Group exemption

Redaction is gated by IdP group. Callers whose groups claim contains data-privacy (a placeholder name — see Known limitations) receive unredacted responses. The check reads input.subject.claims.groups via object.get chains: a missing subject, missing claims, or missing groups claim means the caller is not exempt and redaction applies — the grant fails closed. This failure mode is safe: a caller whose claims fail to arrive gets over-redaction, never disclosure.

Compliance alignment

• HIPAA §164.502(b) / §164.514(d) — supports minimum-necessary, role-based limits: only placeholder data-privacy group members see raw identifiers in lakehouse reads; everyone else gets a working result set with identifiers masked.
• HIPAA §164.514(a)–(b) — supports de-identification practice by stripping Safe-Harbor identifier classes (SSN, email, phone) from responses; §164.530(c) — supports privacy safeguards on the agent channel.
• GDPR Art. 5(1)(c) — data minimisation on agent reads of personal data from the lakehouse; Art. 9 — reduces special-category exposure on the MCP path where identifiers co-occur with health/HR columns or free-text support tickets; Art. 5(1)(f) / Art. 32 — supports security of processing.
• CCPA/CPRA §1798.121 — supports limiting the use and disclosure of sensitive personal information (SSN) on the agent channel; §1798.150 — reduces nonredacted-PI breach exposure.
• SOC 2 CC6.7 — supports restricting the transmission and movement of information: SSN, email, and phone values in lakehouse reads are redacted on the agent channel before they leave through the async poll and search responses.

Why egress

The PII already lives in the lakehouse — there is nothing to block at ingress on the read path, and denying the query or Genie question outright would make the agent useless for everyday analytics work. The leak happens when the rows or index hits are returned to the MCP client, so the response path is the only place to catch it while keeping the result useful. Ingress SQL guarding (DML/DDL/export denial, schema fencing) is a separate concern handled by companion policies (see Composition).

Tool name matching

Applies on the output path (input.mode == "output") to the data-returning Databricks tools, matched case-insensitively from input.resource.name with input.tool_metadata.name as a fallback (egress hooks may populate either).

Two matching strategies are combined:

• Fixed snake_case verbs — matched by suffix (the gateway prefixes tool names with the configured MCP server name, which is not standardised):
  • poll_sql_result — managed SQL async result (verified)
  • genie_poll_response — Genie One NL answer (verified)
  • execute_sql_query — community synchronous SQL passthrough (verified)
• Dynamic AI Search / UC-function tools — matched by shape. Managed AI Search indexes (and UC function tools) are named {CATALOG}__{SCHEMA}__{NAME} with double-underscore delimiters. The policy matches any tool name that splits into three or more segments on __ (i.e. contains at least two __ delimiters). This is disjoint from the single-underscore fixed verbs above, so the two strategies never collide.

Verify the exact names your gateway emits with the dump-input debug technique before relying on this in production.

Response shape

The policy reads input.payload.text — the MCP content-block array the gateway populates on tool_post_invoke — and rewrites each string block (including string blocks containing serialized JSON row data, since the regexes run over the serialized text). Non-string blocks pass through unmodified. When at least one block changes, the policy emits transform.transformed_payload containing the original payload with the rewritten text array (all other payload keys preserved). When nothing changes, no transform is emitted and the response passes through byte-identical.

Examples

Redacted (in-scope poll tool, non-exempt caller)

{
  "input": {
    "action": "tool_post_invoke",
    "mode": "output",
    "resource": { "name": "databricks-poll_sql_result", "type": "tool" },
    "subject": { "sub": "auth0|u1", "claims": { "groups": ["analysts"] } },
    "payload": {
      "name": "databricks-poll_sql_result",
      "text": ["cust 42 | ssn 123-45-6789 | jane@acme.com | 206-555-0100"]
    }
  }
}

allow = true, with transform.transformed_payload.text = ["cust 42 | ssn [REDACTED-SSN] | [REDACTED-EMAIL] | [REDACTED-PHONE]"].

Redacted (AI Search index — support ticket free-text)

{
  "input": {
    "action": "tool_post_invoke",
    "mode": "output",
    "resource": { "name": "databricks-support__tickets__ticket_index", "type": "tool" },
    "payload": {
      "name": "databricks-support__tickets__ticket_index",
      "text": ["Ticket #88: reach the customer at jane@acme.com"]
    }
  }
}

allow = true, with the email rewritten to [REDACTED-EMAIL].

Passed through (exempt caller)

{
  "input": {
    "action": "tool_post_invoke",
    "mode": "output",
    "resource": { "name": "databricks-genie_poll_response", "type": "tool" },
    "subject": { "sub": "auth0|u2", "claims": { "groups": ["data-privacy"] } },
    "payload": {
      "name": "databricks-genie_poll_response",
      "text": ["The record lists SSN 123-45-6789"]
    }
  }
}

allow = true, no transform — the data-privacy group receives raw content.

Composition

Single-purpose transform policy (default allow := true); it composes cleanly with deny/transform policies on the same egress pipeline. Recommended companions for apps/databricks:

• mask-pan-egress (PF-01) — cardholder PAN masking (Luhn-validated, mask to BIN+last4) is intentionally left to that companion policy and is not handled here, so each policy stays single-purpose. Attach both for cardholder-data environments.
• A guard-warehouse-sql-style ingress deny (PF-07) that blocks DML/DDL, GRANT/REVOKE, and export constructs in the SQL argument — so data redacted on read cannot be bulk-exported around the gateway instead.
• A sensitive-schema fence (PF-23) on execute_sql* / describe_uc_table that limits what can be asked — the ingress half of this defense-in-depth pair.
• A default-deny-unknown-tools-style ingress allowlist (PF-28) — the managed AI Search and UC-function tool names are dynamic and drift; a default-deny allowlist stops a newly-added result surface from silently reaching the agent unredacted.

Known limitations

• Cardholder PAN is out of scope. PAN detection/masking is deliberately delegated to the companion mask-pan-egress (PF-01) policy; this policy does not attempt Luhn validation or card masking. A Luhn-valid PAN in a result passes through untouched here.
• The single-space Genie server's tool name is unverified and not matched. The landscape note records that the Genie Space (GA) server exposes a single Genie-invoke tool whose wire name was not published in any verifiable doc. Only Genie One's verified genie_poll_response is matched; if your deployment uses a Genie Space server, confirm its poll/answer tool name with the dump-input technique and add it to fixed_result_suffixes.
• The community JustTryAI synchronous execute_sql is not matched. Its name shares the execute_sql stem with the managed submit tool (which egresses no data), so it is excluded to avoid firing on the submit surface. Only the community execute_sql_query (RafaelCartenet) synchronous passthrough is matched. If you run the JustTryAI server, pin its execute_sql in fixed_result_suffixes.
• Double-underscore matching also covers UC-function tools. UC function tools share the {CATALOG}__{SCHEMA}__{NAME} shape with AI Search indexes, so their responses are redacted too. On egress this is safe over-application (redacting PII from a function's output never leaks), not a defect — but be aware the policy is not AI-Search-exclusive. A tool name with only a single __ is not matched (it does not fit the three-segment dynamic shape).
• Managed AI Search / UC tool names are dynamic — pair with a default-deny allowlist. Because these names are per-index/per-function and can change, a default-deny-unknown-tools allowlist (PF-28) is the right backstop so an unmatched result surface cannot silently leak.
• Pattern-based detection is best-effort and conservative by design. SSNs are matched in the canonical hyphenated form only — bare 9-digit runs collide with row IDs and sequence values, and dot- or space-separated forms (123.45.6789, 123 45 6789) are not matched; phones only in separator-formatted US shapes ((206)555-0100 with no space after the parenthesis, tab-separated forms, and bare 10-digit runs are not matched); emails only when word-boundary anchored. Obfuscated, split-across-cells, spelled-out, full-width/unicode-digit, or non-US-formatted values are not caught. Treat this as a high-signal minimum-necessary layer, not a complete DLP solution.
• Characters glued directly to a value defeat the word-boundary anchors (red-team residual). The SSN and phone patterns are \b-anchored, so a value with an extra digit or letter adjacent and no delimiter escapes detection: 123-45-67890 (SSN trailing digit), id00123-45-6789 / acct123-45-6789 (SSN leading digits/letters), 206-555-01000 (phone trailing digit), and — symmetrically — 1206-555-0100 / id206-555-0100 (phone leading digit/letter glued to the area code) all pass through unredacted. This is a deliberate trade-off — dropping the anchors would emit partial redactions such as [REDACTED-SSN]0 (which still leaks the extra digit) and fire false positives on longer numeric IDs. Where result columns concatenate identifiers without delimiters, rely on Unity Catalog column masking or a stricter companion policy rather than this egress backstop.
• The email pattern can over-match inside connection strings. A user:password@host.example.com substring in a returned DSN/connection string matches the email shape and is redacted. On egress this is over-redaction (safe), not disclosure, but it can obscure legitimate non-email content — tune email_pattern if your result sets routinely contain such strings.
• Only input.payload.text is scanned — structured payload keys leak (red-team residual). Redaction applies to string entries of input.payload.text (including serialized-JSON strings) and leaves non-string entries within that array unmodified — including MCP object content blocks of the form {"type":"text","text":"…"}: because such a block is an object, not a bare string, redact_block returns it as-is and any PII in its inner text field passes through unredacted. This policy assumes the gateway's documented egress shape — a flat array of plain strings (skill dump-input contract) — and does not descend into object blocks; if your gateway version delivers object content blocks instead of flattened strings, confirm the shape with the dump-input technique and treat this layer as inactive (pair with a PF-28 allowlist) until it emits a string array. It does not reach any other payload key: if your gateway delivers Databricks rows in a top-level structuredContent (or similar structured_content / content / data) object rather than as strings in text, the identifiers in that object pass through unredacted — an SSN in payload.structuredContent.rows[…] is not caught. The policy deliberately reads only the documented egress field (payload.text) rather than guessing at an undocumented structured shape and emitting a malformed rewrite. Verify where your gateway version actually places tabular Databricks results with the dump-input technique before relying on this layer, and pair it with a default-deny-unknown-tools allowlist (PF-28) so a result surface whose payload shape this policy cannot read is not silently reaching the agent.
• Group names are placeholders — replace data-privacy with your IdP's group name at import time. The exemption expects the groups claim as an array of strings (a single bare string is also handled); if your IdP emits roles under a namespaced claim, adjust caller_groups. Missing claims always mean redaction applies — the failure mode is over-redaction, not disclosure. Never rely on stripped ContextForge-internal claims (is_admin, teams, user) for the exemption.
• Egress transformed_payload replaces the response payload wholesale. Verify the rewrite against your gateway version with the dump-input technique before production, and mind attachment order if other egress transforms (e.g. mask-pan-egress) run on the same pipeline.

Compliance note. This policy supports alignment with the cited framework controls on the MCP path only. No policy or bundle makes an organization compliant with any framework; web-UI, native-API, and in-app access are outside the gateway's reach by design. Validate against your own compliance program before relying on it.

Policy source (Rego)

package databricks.egress.redact_pii

# Transform-only egress policy: rewrites SSN, email, and phone patterns in the
# responses of the Databricks tools that carry lakehouse data back to the agent
# (managed SQL async result `poll_sql_result`, Genie One answer
# `genie_poll_response`, community synchronous `execute_sql_query`, and the
# dynamic AI Search `{CATALOG}__{SCHEMA}__{INDEX}` tools) to fixed redaction
# tokens before the response reaches the agent. Never denies — a legitimate
# query, Genie question, or search still succeeds, just with sensitive values
# masked. It pairs with the ingress SQL/schema guards: ingress limits what can be
# asked, egress limits what actually leaks back through the async poll and search
# surfaces. Callers in the placeholder `data-privacy` IdP group receive
# unredacted responses; the group check fails closed, so a caller with missing
# claims gets over-redaction, never disclosure. Cardholder PAN masking is left to
# the companion mask-pan-egress (PF-01) policy.
default allow := true

# -----------------------------------------------------------------------------
# Scope: the data-returning Databricks tools. The gateway prefixes tool names
# with the configured MCP server name (not standardised), so we match by suffix,
# case-insensitively. Data egresses in the POLL / synchronous-result responses,
# NOT in the submit call (`execute_sql`, `genie_ask`), which return only a
# handle — so the managed submit tools are intentionally excluded.
# -----------------------------------------------------------------------------

fixed_result_suffixes := {
    # Managed Databricks SQL server — verified: async result poll
    "poll_sql_result",
    # Genie One server — verified: natural-language answer poll
    "genie_poll_response",
    # Community RafaelCartenet server — verified: synchronous SQL passthrough
    "execute_sql_query",
}

# Candidate tool names: resource.name (PARC) and tool_metadata.name (egress
# fallback). Empty strings are dropped so an absent surface never matches "".
candidate_names := {n |
    some src in [
        object.get(object.get(input, "resource", {}), "name", ""),
        object.get(object.get(input, "tool_metadata", {}), "name", ""),
    ]
    src != ""
    n := lower(src)
}

# Fixed snake_case verbs, matched by suffix.
is_pii_result_tool if {
    input.mode == "output"
    some n in candidate_names
    some suffix in fixed_result_suffixes
    endswith(n, suffix)
}

# Dynamic AI Search / UC-function tools: named {CATALOG}__{SCHEMA}__{NAME} with
# double-underscore delimiters. A name that splits into 3+ segments on "__" has
# at least two "__" delimiters — the dynamic shape. This is disjoint from the
# single-underscore fixed verbs above, so the two strategies never collide.
is_pii_result_tool if {
    input.mode == "output"
    some n in candidate_names
    count(split(n, "__")) >= 3
}

# -----------------------------------------------------------------------------
# Group exemption — placeholder IdP group whose members receive unredacted
# responses. Replace "data-privacy" with your IdP's group name at import time.
# object.get chains mean a missing subject/claims/groups claim is never exempt:
# the grant fails closed and redaction applies.
# -----------------------------------------------------------------------------

exempt_groups := {"data-privacy"}

caller_groups := object.get(
    object.get(object.get(input, "subject", {}), "claims", {}),
    "groups",
    [],
)

is_exempt if {
    some g in caller_groups
    lower(g) in exempt_groups
}

is_exempt if {
    # Some IdPs emit a single group as a bare string rather than an array.
    is_string(caller_groups)
    lower(caller_groups) in exempt_groups
}

# -----------------------------------------------------------------------------
# Detection patterns — anchored and conservative to limit false positives on
# free-text lakehouse columns and index documents.
# -----------------------------------------------------------------------------

# US SSN in the canonical hyphenated form only. Bare 9-digit runs collide with
# row IDs and sequence values, so they are deliberately not matched.
ssn_pattern := `\b\d{3}-\d{2}-\d{4}\b`

# Email addresses, word-boundary anchored: local part, "@", domain, TLD of at
# least two letters. Conservative TLD class keeps it from firing on stray "@".
email_pattern := `\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b`

# Separator-formatted US phone numbers (e.g. 206-555-0100, (206) 555-0100,
# +1 206.555.0100). Bare 10-digit runs are deliberately not matched. The 3-3-4
# grouping is disjoint from the SSN 3-2-4 grouping, so the two never collide.
phone_pattern := `(?:\+?1[-. ])?(?:\(\d{3}\)|\b\d{3})[-. ]\d{3}[-. ]\d{4}\b`

# -----------------------------------------------------------------------------
# Redaction steps — each is total over strings: it returns the input unchanged
# when its class doesn't apply, so the steps chain safely.
# -----------------------------------------------------------------------------

redact_ssn(t) := regex.replace(t, ssn_pattern, "[REDACTED-SSN]")

redact_phone(t) := regex.replace(t, phone_pattern, "[REDACTED-PHONE]")

redact_email(t) := regex.replace(t, email_pattern, "[REDACTED-EMAIL]")

# Order: SSN first (fixed 3-2-4 shape), then phones (3-3-4, disjoint from SSN),
# then emails (contain "@", disjoint from both digit patterns). The redaction
# tokens contain no digits-with-separators or "@", so no step can re-match a
# token emitted by an earlier step.
redact_block(b) := redact_email(redact_phone(redact_ssn(b))) if {
    is_string(b)
}

# Non-string content blocks (structured blocks) pass through unmodified.
redact_block(b) := b if { not is_string(b) }

# -----------------------------------------------------------------------------
# Transform — emitted only when in scope, the caller is not exempt, and at least
# one block actually changed. Otherwise the rule is undefined and the aggregator
# skips this policy, returning the response byte-identical.
# -----------------------------------------------------------------------------

response_payload := object.get(input, "payload", {})

text_blocks := object.get(response_payload, "text", [])

redacted_blocks := [out |
    some block in text_blocks
    out := redact_block(block)
]

transform := {
    "transformed_payload": object.union(response_payload, {"text": redacted_blocks}),
} if {
    is_pii_result_tool
    not is_exempt
    is_array(text_blocks)
    redacted_blocks != text_blocks
}

Canonical source: policy.md on GitHub · raw · raw on this site (.md)

Related policies