How We Handle LLM Security in Regulated Industries

The standard advice for LLM deployments is to use the API, call it with your data, get back a result. In regulated industries, that advice is incomplete and in some cases non-compliant. Here is the security architecture we actually deploy for financial services, healthcare, and insurance clients.

The Three Questions Compliance Always Asks

Before any architecture discussion, there are three questions that compliance teams ask on every engagement. Having clear answers to these three questions before the first technical conversation saves weeks of back-and-forth.

1. "Does our data get used to train the model?"

The answer with most enterprise agreements is no, but you need the documentation. OpenAI's enterprise API agreement explicitly prohibits using customer data for training. So does Anthropic's. So does Google Vertex AI. Get the relevant DPA (Data Processing Agreement) and zero-data-retention confirmation in writing before the conversation with compliance.

For healthcare, the question is specifically about PHI (Protected Health Information). Azure OpenAI Service is HIPAA-eligible with a BAA included in Microsoft Enterprise Agreements. AWS Bedrock (for Claude and other models) is also HIPAA-eligible under the AWS BAA. Anthropic now offers a direct API BAA for Enterprise customers — you can enable HIPAA-ready configuration from organization settings without a separate legal cycle. If you are processing PHI, your model must be accessed through a covered configuration regardless of provider.

2. "Where does our data go when a query is processed?"

This is a data residency question. The answer depends on your provider and your configuration:

• Azure OpenAI: your Azure region, within your tenant, with your existing Azure compliance controls
• AWS Bedrock: your AWS region, within your account
• Anthropic API (direct): US data centers; EU region available for enterprise contracts
• Google Vertex AI: your selected region

For UK and EU financial services, data processed under GDPR needs to stay in the EEA or a country with adequacy status unless specific transfer mechanisms are in place. Azure UK South and Azure EU regions address this cleanly.

3. "Can you prove what the AI had access to when it made a decision?"

This requires an audit trail. Not just logging the output. Logging the complete context: the prompt, the retrieved documents with their versions, the model used, the timestamp, and the user identity. If a compliance officer asks why the system flagged a transaction or generated a particular analysis on a specific date, you need to be able to reconstruct exactly what information the system had access to.

The Security Architecture We Deploy

User Request
    → Authentication (SSO / SAML / existing identity provider)
    → Authorization (role-based access control on document corpus)
    → PII Detection (scan prompt for accidental PII injection)
    → Rate Limiting (per-user and per-team quotas)
    ↓
LLM API Call
    → Private endpoint (Azure VNet / AWS VPC, no public internet)
    → Zero-data-retention header where supported
    → Request ID logged before API call
    ↓
Response Processing
    → Output scanning (detect unexpected PII in responses)
    → Structured output validation
    → Confidence threshold check
    ↓
Audit Log Entry
    → user_id, timestamp, request_id
    → prompt_hash (not the full prompt, to protect document confidentiality)
    → document_ids accessed (full list, with versions)
    → model_id and model_version
    → response_hash
    → latency, tokens_used, cost

The audit log does not store the full prompt or response text by default. It stores hashes and identifiers that allow reconstruction if needed, without creating a permanent copy of sensitive document content in a separate store. Full content logging can be enabled for specific high-risk workflows where it is explicitly required.

PII and Sensitive Data Handling

The biggest risk in enterprise LLM deployments is not the model provider. It is your own employees accidentally including sensitive data in prompts.

A PII detection layer before the LLM call scans prompts for patterns like credit card numbers, social security numbers, account numbers, and similar identifiers. Detected PII is either redacted automatically or flagged for review depending on your policy. The detection library we use is Presidio (open source, from Microsoft), which handles multiple languages and entity types.

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine

analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()

def sanitize_prompt(text: str) -> str:
    results = analyzer.analyze(text=text, language='en')
    # Flag high-confidence PII findings
    high_confidence = [r for r in results if r.score > 0.7]
    if high_confidence:
        # Either block the request or anonymize
        anonymized = anonymizer.anonymize(text=text, analyzer_results=results)
        return anonymized.text
    return text

Network Architecture for Maximum Control

For the highest security requirements, the LLM API call goes over a private network path, not the public internet:

• Azure: Azure Private Endpoints for Azure OpenAI, keeping traffic within your VNet
• AWS: VPC endpoints for Bedrock, keeping traffic within your VPC
• On-premise option: Deploy an open-weight model (Llama 3, Mistral) on your own infrastructure. No data leaves your environment at all.

The on-premise option is increasingly viable. A Llama 3 70B model on a 4x A100 server handles thousands of document processing requests per hour with performance that is adequate for most business automation tasks. The infrastructure cost is higher upfront, but for organizations where the regulatory requirement is that data literally cannot leave their environment, it is the only compliant path.

Access Controls on the Document Corpus

Role-based access in a RAG system needs to operate at retrieval time, not just at login. A user who should not be able to see HR documents should not get HR document content returned as context, even indirectly.

The implementation: every document in the vector store carries a permitted_roles metadata field. Every retrieval query includes a filter on that field based on the authenticated user's roles. This happens at the database level, not in application code, so it cannot be bypassed.

Sources

• Microsoft — Azure OpenAI HIPAA Compliance Status
• Anthropic — HIPAA-ready Enterprise Plans
• AWS — Bedrock Anthropic Models HIPAA Compliance
• Microsoft Presidio — Open-source PII Detection