fahmek in an existing LiteLLM proxy

fahmek joins a LiteLLM proxy you already run, with its models, keys and applications. It does not replace the proxy, change its models or need a local model. There are two ways in:

The kit adds two LiteLLM guardrails (standard CustomGuardrail plugins, plain Python files):

fahmek_trace.py is a helper both import. The guardrails call two services that run beside the proxy: fahmek (+ Redis) and, optionally, auto-gad (+ LanguageTool). The proxy needs no outbound Internet access.

Kit contents:

(Dockerfile and litellm_config.yaml here build the ready-made fahmek-litellm demo image; you do not need them for an existing proxy.)

1. Run fahmek beside the proxy#

On a host the proxy can reach, with the images loaded from the offline archive (docker load -i fahmek-image.tar, and fahmek-autogad-image.tar for auto-gad):

export FAHMEK_API_KEY="$(python -c 'import secrets; print(secrets.token_hex(32))')"
mkdir -p license && cp /path/to/received.license license/fahmek.license
docker compose -f docker-compose.services.yml up -d                     # fahmek + Redis
docker compose -f docker-compose.services.yml --profile autogad up -d   # + auto-gad
curl -s localhost:8100/healthz

Keep the key: the proxy needs the same value. fahmek listens on port 8100 (FAHMEK_PORT) and auto-gad on 8200 (AUTOGAD_PORT). Without a valid license file fahmek refuses to inspect (403) and the proxy then rejects calls; GET /v1/license (with X-Fahmek-Key) shows the license state.

From the LiteLLM web UI (no file on the proxy)#

After section 1, everything happens in LiteLLM's admin UI (http://<proxy>:4000/ui, which needs the proxy's database). The guardrails are stored in that database; nothing changes on the proxy's disk or image. Checked by clicking through the UI of LiteLLM v1.105.1; the same pages exist from v1.89.

Use the Submitted Guardrails tab, not Add New Guardrail → Add Provider Guardrail: that form has no API Base or API Key field for the "Generic Guardrail API" provider, so it cannot reach fahmek.

1. Set the masking policy on the fahmek side, in the .env next to docker-compose.services.yml (FAHMEK_POLICY_ID=bank-pilot, for example; FAHMEK_TENANT_ID names this proxy), then docker compose -f docker-compose.services.yml up -d. Requests cannot choose another policy.

2. Teams → Create Team, Team Name fahmek, Create Team. The team only labels the submissions: the guardrails below apply to every key and every request of the proxy (default_on).

3. Guardrails → Submitted Guardrails → Add Guardrail, three times:

   Guardrail Name	Mode	API Base URL	Additional litellm_params
   fahmek-mask	Pre Call	http://<fahmek host>:8100	{"api_key": "<FAHMEK_API_KEY>", "default_on": true, "unreachable_fallback": "fail_closed"}
   fahmek-restore	Post Call	http://<fahmek host>:8100	{"api_key": "<FAHMEK_API_KEY>", "default_on": true, "streaming_transform_mode": "incremental_diff"}
   auto-gad (optional)	Pre Call	http://<auto-gad host>:8200	{"api_key": "<FAHMEK_API_KEY>", "default_on": true}

   Team: fahmek. Click Submit for Review each time. The UI accepts an API Base URL only if its host is an IP address, localhost, or a name with a dot (fahmek.bank.local, not fahmek). LiteLLM adds /beta/litellm_basic_guardrail_api by itself and sends api_key as x-api-key, which fahmek accepts on that path only.

4. Still in Submitted Guardrails, click Approve on each of the three, then Approve again to confirm. They are active at once.

5. Check with the request of section 5: the answer shows the identifiers again; your provider's logs show «MA_…» labels.

New models and the Playground#

Models added later need nothing: the guardrails are default_on, so every model of the proxy goes through fahmek. Checked in LiteLLM v1.105.1:

• Add a model: Models + Endpoints → Add Model. Provider (e.g. Anthropic, or OpenAI-Compatible Endpoints with its API Base), LiteLLM Model Name(s) (from the list, or Custom Model Name (Enter below)), Public Model Name (what applications call), API Key, then Test Connect and Add Model.
• Test in the Playground: Playground → Chat, Virtual Key Source Current UI Session, Endpoint Type /v1/chat/completions, the model; leave Guardrails empty (default_on applies). Send a synthetic identifier such as Mon RIB est 999001123456789012345691 et ma CIN AB123456. The answer shows it again. The Playground streams, so incremental_diff (v1.96+) is required; otherwise it shows «…» labels.
• What the model saw: the Playground shows only the restored answer. Logs → the request shows the guardrails evaluated (fahmek-restore counts once per streamed round); the provider's own request logs show the «MA_…» labels.
• LiteLLM's logs keep no message text by default. If store prompts in spend logs is switched on (Admin Settings → Logging Settings), LiteLLM stores conversation text in its own database, outside fahmek's protection: keep it off.

To try this on the demo machine first, docker-compose.litellm-ui.yml starts a second, unmodified LiteLLM with its database on port 4001 (see the comments in that file); there, the API Base URLs are http://fahmek.internal:8000 and http://autogad.internal:8200.

The same steps can be scripted with the calls the UI makes: POST /team/new, POST /guardrails/register (team_id, guardrail_name, litellm_params with guardrail: generic_guardrail_api, mode, api_base and the fields above) and POST /guardrails/submissions/<guardrail_id>/approve, all with a LiteLLM admin key. tests/test_litellm_ui_integration.py does exactly this.

What differs from the kit:

• If fahmek is unreachable, LiteLLM's unreachable_fallback decides, not the policy: keep fail_closed with bank-pilot or bank-ma-strict. Every fahmek refusal (license, too long, rate limit, blocked identifier, Redis down under a strict policy, images, identifiers in function-call arguments) comes back as a block that LiteLLM always enforces.
• Function calling. LiteLLM does not take masked function-call arguments back from this kind of guardrail: a request whose function-call history holds an identifier is refused, and arguments the model writes keep their «…» labels. Use the kit for such applications.
• Streaming is restored only with streaming_transform_mode incremental_diff (LiteLLM v1.96 or later). Without it, or on older releases, streamed answers keep the «…» labels (nothing leaks).
• Mapping lifetime. A call's masked values stay in Redis until the policy's TTL (120 s for bank-pilot), not until first use, because LiteLLM restores a streamed answer several times.
• No readable trace: check masking in the provider's logs (make trace in the demo covers only its own gateway on port 4000).

2. Make the guardrails importable by the proxy#

Pick the option that matches how the proxy runs. Every option only adds three Python files and PYTHONPATH; the proxy's own command and config stay yours.

One rule for every option: put the three files in the directory of the proxy's config.yaml and put that directory on PYTHONPATH. LiteLLM releases before v1.96 load a custom guardrail only from a file next to the config (measured: v1.89.7 and v1.93.2 fail to start otherwise with ImportError: Could not import FahmekGuardrail); later releases also find it on PYTHONPATH. The files import each other by name, hence PYTHONPATH in both cases.

A. Docker: extend your image (recommended). Build once per LiteLLM upgrade, from this directory:

docker build -f Dockerfile.extend \
  --build-arg BASE_IMAGE=ghcr.io/berriai/litellm:v1.105.1 \
  --build-arg GUARDRAIL_DIR=/app \
  -t registry.example.internal/litellm-fahmek:v1.105.1 .

Use your current image as BASE_IMAGE (official or your own build) and the directory where your config.yaml is mounted as GUARDRAIL_DIR (/app, the default, is where the LiteLLM documentation mounts it). Then run the new tag exactly as before. If a whole directory is mounted over GUARDRAIL_DIR at run time (a Kubernetes ConfigMap without subPath), the copied files are hidden: use option B instead.

B. Docker or Kubernetes: mount the files. No new image. Mount fahmek_guardrail.py, autogad_guardrail.py and fahmek_trace.py next to the config, and set PYTHONPATH to that directory:

# docker compose (config mounted at /app/config.yaml)
services:
  litellm:
    volumes:
      - ./config.yaml:/app/config.yaml:ro
      - ./fahmek-litellm-kit/fahmek_guardrail.py:/app/fahmek_guardrail.py:ro
      - ./fahmek-litellm-kit/autogad_guardrail.py:/app/autogad_guardrail.py:ro
      - ./fahmek-litellm-kit/fahmek_trace.py:/app/fahmek_trace.py:ro
    environment:
      PYTHONPATH: /app

In Kubernetes, add the three files to the ConfigMap that already holds config.yaml (they are then mounted in the same directory) and set PYTHONPATH to that directory in the container env.

C. No Docker (LiteLLM installed with pip install 'litellm[proxy]'). Install the wheel into the proxy's own Python environment:

pip install --no-deps fahmek_litellm-0.1.0-py3-none-any.whl

It declares no dependencies on purpose, so it never changes the versions of what your proxy already uses (httpx, fastapi and pydantic come with litellm[proxy]). LiteLLM v1.96 or later then finds the guardrails with no other step (checked with pip install 'litellm[proxy]==1.105.1'). Older releases still need the three .py files next to config.yaml. Build the wheel with pip wheel --no-deps -w dist integrations/litellm (scripts/package.py already does).

3. Add the guardrails to your config.yaml#

Paste the two entries of guardrails.yaml into your config. If you already have a guardrails: list, add them to it; keep fahmek before auto-gad and before any guardrail or callback that sends text out, so those only see masked text. Drop the auto-gad entry if you do not run it.

default_on: true protects every model, key and application of the proxy. With false, only requests that ask for the guardrail are protected, which an application can simply omit: keep true for PII.

4. Add the environment#

Copy the variables of litellm.env.example into the proxy's environment (compose environment:, Kubernetes Secret, systemd EnvironmentFile...):

Air-gapped proxies should also set LITELLM_LOCAL_MODEL_COST_MAP=True, so LiteLLM does not fetch its price list at start (nothing to do with fahmek, but the most common outbound call).

5. Restart and check#

Restart the proxy. Its start-up log must not show ModuleNotFoundError (PYTHONPATH or file missing) and should not show fahmek.policies.prefetch_failed (FAHMEK_URL or key wrong; it retries on the first request, but fix it).

Send a synthetic identifier through any of your models:

curl -s http://<proxy>:4000/v1/chat/completions \
  -H "Authorization: Bearer <a LiteLLM key>" -H 'Content-Type: application/json' \
  -d '{"model":"<one of your models>","messages":[{"role":"user","content":"Repeat exactly: CIN AB123456, email test@example.test"}]}'

The answer shows AB123456 and the email again: they were masked before the model and restored after. To see the masking itself, look at the request in your provider's logs, or set FAHMEK_TRACE=1 on a test proxy and read its log (fahmek masked: MA_CIN x1, EMAIL x1). Never enable the trace in production: it prints text that fahmek did not detect, such as names.

What a shared proxy must know#

• Failure mode follows the policy. If fahmek is down, a fail_open: true policy (default, pilot-high-recall) lets requests through uninspected; a fail_open: false policy (bank-pilot, bank-ma-strict) rejects them, for every application of the proxy. A fahmek rejection (4xx: too long, blocked identifier, malformed tool arguments) is never let through. auto-gad never blocks anything.
• Latency. fahmek adds a few milliseconds per message (15 ms CPU budget per message); the kit inspects every message of the conversation one after another, so a 10-message turn gains about 50 ms. auto-gad adds ~20–50 ms on French/English messages. Sizing for many users: docs/production-sizing.md.
• What is inspected. OpenAI Chat Completions (/v1/chat/completions) messages of every role, text-part arrays and function-call arguments. Content outside messages (embeddings input, /v1/completions prompt, /v1/responses input, a top-level Anthropic system) is not inspected; images and audio parts are rejected.
• What is detected. The seven Moroccan identifiers, EMAIL and Luhn-checked card numbers. Not names, addresses or dates of birth.
• Upgrading LiteLLM. The guardrails use LiteLLM's guardrail hooks; check the new release in COMPATIBILITY.md or run python scripts/litellm_compat.py --versions <tag> before upgrading.