Using cloud LLMs on confidential documents
- Question
- How can a lawyer use a cloud language model on a client file without the identifiers reaching the provider, and how do you prove it?
- Status
- Built and in use as a prototype. A paper on the evaluation method is in preparation, so details here are deliberately general.
- Languages
- Dutch, French, German, Italian, English
A browser-based pseudonymisation gateway for confidential legal and fiduciary documents. Detection and replacement run on the device before anything reaches a model provider, and the answer is restored locally. The design principle is exposure accounting: every optional layer can only add protection, every server payload is either what the provider receives anyway or ciphertext, and each rule or model decision that leaves an identifier unchanged is listed with a measured leak rate.
- Document in the browser
- Local detection and pseudonymisation
- Optional encrypted check and server model on the protected text
- Protected text to the model provider
- Answer restored locally
What I built
- A deterministic evidence engine: candidate patterns, checksum and format validators, multilingual context evidence, negative rules for public citations, and per-class scoring into redact, review or keep, each with an explanation.
- My own multilingual transformer NER model, fine-tuned and compressed for in-browser inference (ONNX Runtime Web, int8 and fp16, pruned vocabulary), trained over several documented rounds on licence-vetted corpora and compared head to head with public models on identical documents.
- A server-side tagger that sees only the already-protected text and returns spans that can only add protection.
- A homomorphic-encryption second check: pooled model states are encrypted in the browser (BFV, SEAL compiled to WebAssembly) and scored by a keyless server.
- A research line on a "blind" server model that receives non-character token features, with a trained inversion attack used to price what each feature group leaks.
How it is evaluated
- A black-box comparison of the masked text that leaves the device, against open detectors on a standard anonymisation benchmark and seven public labelled sets, with bootstrap intervals and paired tests. Production egress, utility and round-trip tracks are planned.
- A gold set built by multi-reader labelling with adjudication and a deterministic checksum veto, under a 118-class taxonomy.
- Standing red-team corpora, security reviews and decision records.
What I found
- Synthetic personal-data corpora overstate precision on real court text. One entity type fell from the high eighties to zero on real judgments.
- A single propagation mechanism produced almost all of the false positives, and a silent window-truncation bug hid recall on long documents until it was audited.
- Certifying a leak bound is limited by the number of documents, not the number of spans.