A permission-aware policy assistant for a bank
Bank employees lose time hunting through policies and get inconsistent answers. I designed an assistant that answers only from documents the person is allowed to read, cites its sources and says “I don’t know” when the evidence is weak. Then I built a prototype to test the hard parts.
The 30-second read
Updated October 2026- Status
- Prototype
A working local prototype on synthetic documents, plus a proposed six-week pilot. It has not been deployed at a bank, and no productivity results are claimed.
Prototype: Working code that proves the hard parts. Not production, not deployed for a client.
- Context
- A hypothetical but realistic brief, from my article. A major UK bank says its employees spend hours searching internal documents, policies and procedures, and asks for an AI assistant. There is no requirements document, only a business problem.
- Problem
- Staff spend too long finding policy answers, ask the same questions repeatedly, and interpret policy inconsistently, in an environment where every answer must be auditable and restricted documents must stay restricted.
- Approach
- Treat it as a trusted decision-support system, not a search box. That means permission-aware hybrid retrieval, answers grounded in approved documents with checkable citations, abstention when evidence is weak, and an evaluation loop built from reviewed feedback.
- What I did
- I framed the problem, discovery questions and architecture (published as an article), then built the prototype. It covers ingestion, hybrid retrieval, role-based access, cited generation with quote validation, audit logging, feedback and human-reviewed evaluation, with 36 automated tests.
- Proposed, not built
- Enterprise SSO, permission filtering inside the retrieval query, a cross-encoder reranker if evaluation justifies it, automated re-indexing, an evaluation dashboard, tracing and cost accounting.
- Intended outcome
- Faster, consistent, traceable answers across one department’s ~100 approved documents, proven in a measured pilot before any wider rollout.
- Stack
- PythonFlaskSQLite + FTS5pypdfOllama (nomic-embed-text, llama3.2)Microsoft Graph
The brief
The scenario starts with a single sentence from the customer: employees spend hours searching internal documents, policies and procedures, so build an AI assistant to help them find things faster.
Taken literally, that is a search problem. In a regulated bank it is something else. Behind “we need to find things faster” are four needs:
- Consistency. Two teams should not interpret the same policy differently.
- Fewer repeated questions reaching HR, support and compliance.
- Less human error in decisions that depend on policy.
- Auditability. Every answer should trace back to an approved source.
That reframing changes the product from a search tool to a trusted decision-support system, and it changes what “done” means.
Questions before architecture
Before proposing anything I’d want answers to five groups of questions. They shape the system far more than the choice of model.
Q1 Users
- Who will use it: support, HR, compliance, operations?
- What do they search for most often?
- What does a good answer look like: a direct answer, a cited source, a recommendation?
Q2 Documents
- Where do they live: SharePoint, Confluence, Drive?
- What formats: PDFs, Word, wikis?
- How often do they change?
Q3 Access
- Can every employee read every document?
- Which documents are restricted, and to whom?
- Where is the source of truth for permissions?
Q4 Constraints
- What are the audit logging requirements?
- Which regulatory constraints apply?
- What happens when the system is wrong?
Q5 Baseline
- How long does finding an answer take today?
- How many repeat questions reach HR and compliance?
- How will we know it worked?
Scope. One department and about 100 high-quality, approved documents. The goal is grounded, trustworthy behaviour in a narrow domain first. Everything else scales from there.
Architecture
Three lanes. Documents are ingested offline with their access metadata attached. Questions are answered online through a sequence of retrieval and verification steps. Every answer then feeds an audit log and a review loop. Access control is not a stage at the end. It starts at ingestion and is enforced at three points in the answer path.
Select any stage. Access-control checkpoint
AIngestoffline, when documents change
BAnsweronline, per question
CLearnafter the answer
Approved sources
Built in prototype- What it does
- Admins upload text-based PDFs and choose an access label. A separate CLI mode reads SharePoint items through Microsoft Graph with the user’s delegated token.
- Why it’s there
- Only approved documents go in. The assistant’s answers are only as trustworthy as its sources.
- What could fail
- Unapproved or duplicate documents slip in. Every upload creates a new document, even if the filename already exists.
- In the pilot
- Agree the source of truth and an owner for each document with the department before ingesting anything.
Extract & clean
Partly built- What it does
- Text is extracted page by page with pypdf and whitespace is normalised. Scanned PDFs without extractable text are rejected with a clear error.
- Why it’s there
- Page-level extraction keeps a page number with every chunk, so a citation can point to the page.
- What could fail
- No OCR and no table reconstruction yet, so scanned or table-heavy policies lose information.
- In the pilot
- Audit the department’s documents for scans and tables in week 1; add OCR only if they matter.
Chunk
Built in prototype- What it does
- Chunks hold up to 250 words, overlap by 40 words, and never cross a page boundary.
- Why it’s there
- Small enough to retrieve precisely and cite accurately; the overlap stops a rule being split mid-sentence.
- What could fail
- A fixed word count ignores headings and sections, so a chunk can mix two policies.
- In the pilot
- Compare section-aware chunking against the reviewed evaluation cases before changing it.
Tag access & metadata
Access controlBuilt in prototype- What it does
- Each document carries an access label (admin only or all access) or, in SharePoint mode, its drive ID, item ID and eTag version.
- Why it’s there
- This is the information the permission gate reads at query time. Access control starts at ingestion.
- What could fail
- Two hard-coded roles don’t reflect a bank’s real permission model.
- In the pilot
- Map the department’s real access groups. Prefer reading live permissions from the source system over copying them.
Embed & index
Built in prototype- What it does
- Chunks are embedded with nomic-embed-text through Ollama and indexed in SQLite FTS5 for keyword search. Updates replace text, vectors and the keyword index atomically.
- Why it’s there
- Two indexes support hybrid search. An atomic update means a failed re-index never leaves a half-updated document.
- What could fail
- Vector search is a linear scan, which is fine for about 100 documents and not for 100,000.
- In the pilot
- Stay on SQLite for the pilot. Plan an ANN index with filtering by access control before scaling.
Authenticate
Access controlPartly built- What it does
- A signed, HttpOnly, SameSite=Strict session with CSRF tokens. The user’s role is reloaded from the database on every request, and any role sent by the browser is ignored.
- Why it’s there
- Identity has to come from the server. A client-supplied role is a request, not a fact.
- What could fail
- This is demo authentication: known seed credentials, no rate limiting, no SSO.
- In the pilot
- Integrate the bank’s identity provider (SSO) and map its groups to document access.
Retrieve candidates
Built in prototype- What it does
- The top 30 chunks by cosine similarity and the top 30 by BM25 keyword rank are combined with reciprocal rank fusion.
- Why it’s there
- Paraphrased questions need semantic search. Policy codes and exact identifiers need keyword search. A test checks that the keyword branch finds exact identifiers.
- What could fail
- Messy or very similar documents can push the right chunk out of the candidate set.
- In the pilot
- Measure recall on the reviewed question set; tune the candidate counts using that data, not intuition.
Permission gate
Access controlBuilt in prototype- What it does
- Chunks from documents the user can’t read are removed here, before reranking and before any text reaches the model.
- Why it’s there
- The model must never see what the user isn’t allowed to see. A prompt instruction is not access control.
- What could fail
- Because filtering happens after candidates are chosen, a user with narrow access can miss lower-ranked documents they are allowed to read.
- In the pilot
- Push the permission filter into the retrieval query itself.
Rerank & threshold
Partly built- What it does
- A lightweight reranker scores each permitted chunk as 0.7 × cosine similarity + 0.3 × query-term coverage. Up to four chunks scoring at least 0.35 go to the model.
- Why it’s there
- Transparent, fast and good enough for a small corpus. Weak evidence never reaches generation.
- What could fail
- It is a heuristic, not a cross-encoder and not a calibrated confidence score.
- In the pilot
- Test a cross-encoder on the reviewed cases; adopt it only if pass rates rise enough to justify the latency.
Generate claims
Built in prototype- What it does
- A local model (llama3.2:3b by default, configurable) must return JSON claims, each with text, a source number and an exact supporting quote, at temperature 0.
- Why it’s there
- Structured output lets the system check every claim mechanically instead of trusting fluent prose.
- What could fail
- Small local models can be less capable. The prompt treats excerpts as untrusted data, but that is not a complete defence against injection.
- In the pilot
- Choose the model in discovery, based on the bank’s data-handling policy (local vs hosted).
Verify & re-check
Access controlBuilt in prototype- What it does
- Every quote must appear in the chunk it cites (allowing only PDF whitespace and bullet noise). After generation, access and the document version are checked again.
- Why it’s there
- Made-up quotes turn the answer into an abstention. A permission revoked mid-request fails closed.
- What could fail
- A quote can exist and still not support the claim made from it.
- In the pilot
- Add human spot-checks of citation support to the evaluation plan.
Answer, abstain or deny
Built in prototype- What it does
- The result is a cited answer, an “I don’t know based on the available, permitted documents” response, or a contact-an-admin message when the relevant evidence is restricted.
- Why it’s there
- In a bank, a confident wrong policy answer costs more than a referral.
- What could fail
- The access-denied rule is heuristic. A restricted match must clearly outscore permitted ones, and the message itself reveals that restricted material exists.
- In the pilot
- Ask compliance whether “contact an admin” or a plain “I don’t know” is acceptable.
Audit & feedback
Built in prototype- What it does
- Each completed query logs the user ID, question, permitted chunk IDs, result and latency. Users rate their own answers, helpful or not, with an optional comment.
- Why it’s there
- Auditability is a requirement in a regulated setting, and feedback is the raw material for evaluation.
- What could fail
- Exceptions that exit early aren’t fully audited, and the audit log is sensitive in its own right.
- In the pilot
- Close the audit gaps; agree retention and who may read the logs.
Reviewed evaluation
Partly built- What it does
- Admins turn feedback into test cases with an expected outcome per user (answered, abstained or access denied), required text, required sources and forbidden text. The cases run from the UI or from a CLI that exits nonzero on failure.
- Why it’s there
- It turns “it seemed fine” into a regression suite that runs under each user’s real permissions.
- What could fail
- These checks don’t prove semantic correctness, and there is no dashboard yet.
- In the pilot
- Build an evaluation set of about 50 real questions and a simple results dashboard.
Retrieval details for engineers
- Chunks: up to 250 words with a 40-word overlap, bounded by page so each citation can name a page.
- Candidates: top 30 by cosine similarity over normalised embeddings, plus top 30 by FTS5 BM25 over the query’s non-stopword terms.
- Fusion: reciprocal rank fusion with k = 60.
- Permission filter: applied per document before reranking. Restricted candidates only affect whether the response is an access denial.
- Rerank: 0.7 × cosine + 0.3 × query-term coverage; up to 4 chunks with a score of at least 0.35 enter the context.
- Access denial: returned only when a restricted match scores at least 0.1 higher than the best permitted match and no supported permitted answer exists.
Citation validation for engineers
The model must return JSON matching a schema: a list of claims, each with text, a source number and an exact quote. Each claim is accepted only if the source number is valid and the quote appears in that chunk. Matching tolerates PDF whitespace, control characters and bullet glyphs, but not changed words or numbers. The citation keeps the original source text. If any claim fails, the whole response becomes an abstention.
SharePoint mode for engineers
For the case where compliance says employees may only see what they can already open in SharePoint, and permissions change too often to copy. The adapter uses the user’s delegated Microsoft Graph token. A document counts as readable only if Graph issues a download URL and its eTag matches the indexed version. Outages, revocations and stale versions fail closed. There’s no automatic sync, OAuth login flow or token refresh, and tests mock the Graph responses, so it still needs testing against a live tenant.
Security posture of the prototype for engineers
Signed HttpOnly, SameSite=Strict session cookies; CSRF tokens on every write; a strict Content-Security-Policy; hashed passwords; roles reloaded from the database on each request. It is deliberately a loopback-only demo: known seed credentials, no login rate limiting or account recovery, and Flask’s development server. Anyone with access to the SQLite file is outside the app’s access-control boundary.
Following one question
Here is the prototype’s permission gate working on its two synthetic test documents. Change who is signed in and what they ask, and watch which text reaches the model.
employee asks “What is the vault access code?”
| Chunk | Retrieved | Permission gate | Rerank ≥ 0.35 | Sent to the model | Cited |
|---|---|---|---|---|---|
| all-access-leave.pdfAll access | Weak match | Passes | Below threshold | nothing | no |
| admin-vault.pdfAdmin only | Strong match | Withheld | n/a | nothing | no |
Response · reason: access_denied
You do not have access to the requested data. Please contact an admin.
No citations, filenames or restricted text are returned.
- LLM generation calls
- 0
- Audit record
- user=employee · retrieved_chunks=[] · reason=access_denied
employee asks “What is the annual leave allowance?”
| Chunk | Retrieved | Permission gate | Rerank ≥ 0.35 | Sent to the model | Cited |
|---|---|---|---|---|---|
| all-access-leave.pdfAll access | Strong match | Passes | Kept | “Annual leave allowance is 25 days.” | [1] p.1 |
| admin-vault.pdfAdmin only | Weak match | Withheld | n/a | nothing | no |
Response · reason: answered
Annual leave allowance is 25 days. [1]
[1] all-access-leave.pdf · page 1
- LLM generation calls
- 1
- Audit record
- user=employee · retrieved_chunks=[all-access-leave.pdf chunk] · reason=answered
admin asks “What is the vault access code?”
| Chunk | Retrieved | Permission gate | Rerank ≥ 0.35 | Sent to the model | Cited |
|---|---|---|---|---|---|
| all-access-leave.pdfAll access | Weak match | Passes | Below threshold | nothing | no |
| admin-vault.pdfAdmin only | Strong match | Passes | Kept | “The vault access code is ORCHID-742.” | [1] p.1 |
Response · reason: answered
The vault access code is ORCHID-742. [1]
[1] admin-vault.pdf · page 1
- LLM generation calls
- 1
- Audit record
- user=admin · retrieved_chunks=[admin-vault.pdf chunk] · reason=answered
admin asks “What is the annual leave allowance?”
| Chunk | Retrieved | Permission gate | Rerank ≥ 0.35 | Sent to the model | Cited |
|---|---|---|---|---|---|
| all-access-leave.pdfAll access | Strong match | Passes | Kept | “Annual leave allowance is 25 days.” | [1] p.1 |
| admin-vault.pdfAdmin only | Weak match | Passes | Below threshold | nothing | no |
Response · reason: answered
Annual leave allowance is 25 days. [1]
[1] all-access-leave.pdf · page 1
- LLM generation calls
- 1
- Audit record
- user=admin · retrieved_chunks=[all-access-leave.pdf chunk] · reason=answered
The key test in the repository asks a mixed question as an employee (“What about vault and leave?”) and asserts that the restricted text never appears in what was sent to the model, while the permitted text does. That is the architectural promise, written as a test.
Built vs. proposed
- Built in prototype 12
- Partly built 1
- Proposed for pilot 6
- Not yet addressed 2
| Capability | Status | Detail |
|---|---|---|
| Ingestion | ||
| PDF upload with access labelAdmin-only; text-based PDFs up to 20 MB. | Built in prototype | Admin-only; text-based PDFs up to 20 MB. |
| Page-aware chunking250 words, 40-word overlap, page-bounded. | Built in prototype | 250 words, 40-word overlap, page-bounded. |
| SharePoint via Microsoft GraphDelegated token, live readability + eTag checks; tests use mocked Graph responses and it still needs testing against a live tenant. | Partly built | Delegated token, live readability + eTag checks; tests use mocked Graph responses and it still needs testing against a live tenant. |
| OCR and table reconstructionScanned PDFs are rejected with a clear message. | Not yet addressed | Scanned PDFs are rejected with a clear message. |
| Automated sync and re-indexingChanged SharePoint versions fail closed until re-ingested. | Proposed for pilot | Changed SharePoint versions fail closed until re-ingested. |
| Retrieval & generation | ||
| Hybrid retrieval (cosine + BM25, RRF)SQLite FTS5 for the keyword branch. | Built in prototype | SQLite FTS5 for the keyword branch. |
| Heuristic reranking + thresholdNot a cross-encoder; not a calibrated confidence. | Built in prototype | Not a cross-encoder; not a calibrated confidence. |
| Cross-encoder rerankingOnly if evaluation justifies the latency. | Proposed for pilot | Only if evaluation justifies the latency. |
| Structured claims with exact quotesInvalid or unmatched quotes become an abstention. | Built in prototype | Invalid or unmatched quotes become an abstention. |
| “I don’t know” and access-denied outcomesRestricted filenames, text and citations never returned. | Built in prototype | Restricted filenames, text and citations never returned. |
| Access control & security | ||
| Permission gate before model contextRegression-tested: restricted text never reaches the model. | Built in prototype | Regression-tested: restricted text never reaches the model. |
| Access re-check after generationMid-request revocation fails closed. | Built in prototype | Mid-request revocation fails closed. |
| Filtered document lists and PDF URLsSame rules as answers. | Built in prototype | Same rules as answers. |
| Session auth, CSRF, CSP, hashed passwordsLocal demo; loopback-only, known seed credentials. | Built in prototype | Local demo; loopback-only, known seed credentials. |
| SSO / enterprise identityMap identity-provider groups to document access. | Proposed for pilot | Map identity-provider groups to document access. |
| Permission filter inside the index queryFixes missed lower-ranked permitted documents. | Proposed for pilot | Fixes missed lower-ranked permitted documents. |
| Quality & operations | ||
| Audit log per queryUser, question, permitted chunk IDs, result, latency. | Built in prototype | User, question, permitted chunk IDs, result, latency. |
| Feedback → reviewed evaluation casesRuns per user from the UI or CLI. | Built in prototype | Runs per user from the UI or CLI. |
| Evaluation dashboardBrowser reports are temporary today. | Proposed for pilot | Browser reports are temporary today. |
| Tracing and cost accountingOut of scope for the prototype. | Proposed for pilot | Out of scope for the prototype. |
| Production deploymentFlask development server, loopback only. | Not yet addressed | Flask development server, loopback only. |
Decisions and trade-offs
- T1
Hybrid retrievalover Vector-only search
Employees paraphrase, but policies also contain exact codes and names that embeddings blur. A test checks that the keyword branch retrieves exact identifiers.
Revisit If evaluation shows one branch never changes the result.
- T2
A transparent heuristic rerankerover A cross-encoder from day one
For about 100 documents it adds no model, no latency and no cost, and its scoring is easy to explain.
Revisit In the pilot: compare against a cross-encoder on the reviewed cases.
- T3
Filter before context, re-check afterover Asking the model to withhold restricted content
Anything in context can leak. The gate removes restricted text before the prompt is built.
Revisit Move the filter into the index query to stop permitted documents being crowded out.
- T4
Abstain when evidence is weakover A best-effort answer every time
A plausible wrong answer about policy is worse than a referral. More abstentions is the price.
Revisit Track the abstention rate as a pilot metric, and tune it rather than minimise it.
- T5
“Contact an admin” when restricted evidence dominatesover A plain “I don’t know”
It routes the employee somewhere useful without revealing filenames, text or citations.
Revisit This confirms restricted material exists. Whether that’s acceptable is a compliance decision.
- T6
Local models via Ollamaover A hosted model API
Documents never leave the machine in the prototype, and models can be swapped with an environment variable.
Revisit Set the pilot’s model by the bank’s data policy and by evaluation results.
- T7
SQLite with a linear vector scanover A dedicated vector database
There’s nothing to operate, and it’s fast enough for one department’s documents.
Revisit Well before scaling across departments.
| Risk | How it would show up | Mitigation | Status |
|---|---|---|---|
| Permission leakage | Restricted facts appear in an answer, a citation or a document list. | Filter before context; re-check after generation; filtered lists and PDF URLs; tests for mixed queries and role spoofing. | Built in prototype |
| Hallucinated policy | A confident claim that no approved document makes. | Structured claims with exact quotes; any unmatched quote turns the answer into an abstention. | Built in prototype |
| Relevant-looking but wrong context | A correct quote from the wrong section or the wrong policy. | Reranking and a threshold now; reviewed evaluation cases and human spot-checks in the pilot. | Partly built |
| Outdated documents | An answer cites a superseded version of a policy. | Fingerprint-based re-ingestion; SharePoint eTag checks fail closed. Automated sync is proposed. | Partly built |
| Prompt injection in a document | Document text tries to override the assistant’s rules. | Excerpts are framed as untrusted data; structured output; quote validation. Not a complete defence. | Partly built |
| Messy source documents | Scans and tables produce missing or garbled text. | Scans are rejected clearly today. OCR and table handling are only worth adding if the audit finds them. | Not yet addressed |
| Cost and latency at scale | Slow answers or rising spend as usage grows. | Latency is logged per query. Caching and cost accounting are proposed. | Proposed for pilot |
No results are claimed. These are the measures I’d agree with stakeholders before the pilot, each against a baseline measured first.
Time to answer
The original complaint.
Measured by Baseline measured in week 1 by observing and timing real searches; compared during the pilot.
Evaluation pass rate
Answer quality you can track release by release.
Measured by Reviewed cases built from about 50 real employee questions, run under each test user’s permissions.
Valid citation rate
Traceability is the point of the system.
Measured by Automated quote checks plus human spot-checks that each citation supports its claim.
Repeated questions to HR and compliance
The cost the bank actually feels.
Measured by Volume of repeat queries compared with the pre-pilot baseline.
Adoption and feedback
An accurate tool nobody uses has failed.
Measured by Active pilot users and helpful / not-helpful ratings with comments.
Abstention rate
Too high and the tool is useless; too low and it may be guessing.
Measured by Share of “I don’t know” answers, reviewed for whether each was justified.
Permission violations
The one metric with a fixed target.
Measured by Zero, checked with role-based evaluation cases on every change.
Evaluation
Feedback is only useful if it becomes something you can run again. In the prototype, a rating doesn’t train anything. It starts a review.
- 01
User rates an answer
Helpful / not helpful, optional comment. Only on their own answers.
- 02
Admin reviews it
Sees the question, original answer, citations and comment.
- 03
Writes the expected outcome
Per test user: answered, abstained or access denied, plus required text, required sources and forbidden text.
- 04
Run the saved cases
With each user’s current permissions, from the UI or a CLI that exits nonzero on failure.
- 05
Fix the cause
Update documents, retrieval settings or the case itself, then re-run.
A role change invalidates that user’s cases until someone reviews them again, so permissions never drift silently.
These checks cover outcomes, required substrings, source filenames, forbidden text, citation presence and current citation permissions. They don’t prove semantic correctness, so the pilot pairs them with human review of the evidence and an evaluation set of about 50 real questions.
Six-week pilot
An indicative plan. The order matters more than the exact weeks, and discovery in week 1 would adjust it.
- Week 1
Discovery & baseline
- Collect about 50 real employee questions
- Time current searches to set a baseline
- Map document owners and access groups
Exit checkAgreed success criteria and an approved document list
- Week 2
Ingestion & access mapping
- Ingest about 100 approved documents
- Attach access metadata from the real permission model
- Audit for scans and tables
Exit checkEvery document indexed with an owner and access label
- Week 3
Permission-aware retrieval
- Hybrid retrieval with the permission filter in the query
- Evaluation set v1 from week 1 questions
- Leakage tests per role
Exit checkZero leaks on role-based tests; retrieval recall measured
- Week 4
Cited answers & abstention
- Grounded generation with quote validation
- “I don’t know” and escalation paths
- Audit logging end to end
Exit checkPass-rate target met on the evaluation set
- Week 5
Feedback, dashboard, security review
- Feedback capture and review workflow
- Evaluation dashboard
- Prompt-injection and abuse testing
Exit checkSecurity sign-off for a limited pilot group
- Week 6
Pilot & recommendation
- A small group of real users
- Compare metrics against the baseline
- Expand, adjust or stop
Exit checkA written recommendation with evidence
If it grew
Before expanding beyond one department, the main changes would be:
- Retrieval at scale. Replace the linear vector scan with an approximate-nearest-neighbour index that filters on access metadata inside the query, so permitted documents aren’t crowded out.
- Permissions from the source. Read live permissions from the document system or identity provider instead of copying them, so revocations take effect without manual updates.
- Freshness. Incremental re-indexing driven by change events, with superseded versions retired.
- Operating it. Tracing per stage, cost per answer, caching where it doesn’t risk serving stale or unauthorised content, and an evaluation dashboard that gates every change.