A permission-aware policy assistant for a bank

A working local prototype on synthetic documents, plus a proposed six-week pilot. It has not been deployed at a bank, and no productivity results are claimed.

Context

A hypothetical but realistic brief, from my article. A major UK bank says its employees spend hours searching internal documents, policies and procedures, and asks for an AI assistant. There is no requirements document, only a business problem.

Problem

Staff spend too long finding policy answers, ask the same questions repeatedly, and interpret policy inconsistently, in an environment where every answer must be auditable and restricted documents must stay restricted.

Approach

Treat it as a trusted decision-support system, not a search box. That means permission-aware hybrid retrieval, answers grounded in approved documents with checkable citations, abstention when evidence is weak, and an evaluation loop built from reviewed feedback.

What I did

I framed the problem, discovery questions and architecture (published as an article), then built the prototype. It covers ingestion, hybrid retrieval, role-based access, cited generation with quote validation, audit logging, feedback and human-reviewed evaluation, with 36 automated tests.

Key decisions

Enforce permissions before the model sees any text	A prompt instruction is not access control. Anything in the context can leak.
Hybrid retrieval (semantic + keyword, fused)	Employees paraphrase, and policies also contain exact codes and names.
Answers as structured claims, each with an exact quote	Citations can be checked mechanically; an unmatched quote becomes an abstention.
Prefer “I don’t know” to a plausible guess	In a bank, a confident wrong policy answer costs more than a referral.

Risks managed

Permission leakage
Hallucinated or misquoted policy
Outdated documents
Prompt injection through document text
Weak retrieval on messy PDFs
Cost and latency at scale

How success would be measured

Time-to-answer against a measured baseline
Pass rate on a reviewed evaluation set (~50 real questions)
Share of answers with valid citations
Repeated questions reaching HR and compliance
Adoption and feedback ratings
Permission violations, with a target of zero

Intended outcome

Faster, consistent, traceable answers across one department’s ~100 approved documents, proven in a measured pilot before any wider rollout.

Proposed, not built: Enterprise SSO, permission filtering inside the retrieval query, a cross-encoder reranker if evaluation justifies it, automated re-indexing, an evaluation dashboard, tracing and cost accounting.