A hospital system with twelve facilities and 14,000 clinical staff wanted to use large language models to assist with clinical documentation. Physicians spent an average of two hours per day on documentation — writing notes, completing prior authorization requests, and responding to patient messages. The hospital’s CIO believed that an LLM-based assistant could reduce documentation time by thirty to forty percent, giving physicians more time with patients.

The obstacle was HIPAA. The hospital handled protected health information for 2.3 million patients. Any system that processed PHI had to comply with HIPAA’s security and privacy rules, which impose strict requirements on data access, storage, transmission, and audit. The commercial LLM APIs that the hospital’s IT team wanted to use — the same APIs that power consumer chatbots — did not meet HIPAA requirements. The API providers processed requests on their own infrastructure, retained inputs for model improvement (with opt-out options that were insufficient for HIPAA’s minimum necessary standard), and could not guarantee the data isolation that HIPAA required.

The hospital’s compliance team was clear: no patient data could leave the hospital’s network. Not to an API. Not to a cloud service. Not to a vendor. The LLM had to run inside the hospital’s infrastructure, or the project could not proceed.

The compliance constraints

HIPAA’s requirements for an LLM-based clinical tool fell into four categories. Data residency: all PHI must remain within the hospital’s controlled infrastructure. No patient data could be transmitted to an external service, even in encrypted form, unless the service was covered by a business associate agreement that met the hospital’s security standards.

Access control: the LLM could only access PHI that the requesting physician was authorized to see. A physician in the cardiology department could not access psychiatry notes through the LLM, even if the LLM was technically capable of processing them.

Audit trail: every interaction between a clinical user and the LLM had to be logged, including the input text, the output text, and the user’s identity. The audit trail had to be retained for six years, consistent with HIPAA’s record retention requirements.

Model isolation: the LLM’s weights and inference infrastructure had to be isolated from any system that contained PHI, except through a controlled access layer that enforced the access control and audit requirements. The model itself could not have direct access to patient records.

The architecture: local LLM with a PHI firewall

We designed an architecture that kept all PHI within the hospital’s network while providing a useful clinical documentation assistant.

The PHI firewall was the core component. It sat between the physician’s workstation and the LLM inference server. When a physician requested documentation assistance, the PHI firewall performed three functions. First, it checked the physician’s authorization against the hospital’s access control system to verify that the physician was permitted to access the patient’s records. Second, it retrieved the relevant clinical context from the EHR system and de-identified it — replacing patient names, dates of birth, medical record numbers, and other direct identifiers with consistent placeholders. Third, it logged the request, including the physician’s identity, the patient’s identity, and the timestamp.

The de-identified context was passed to the local LLM for inference. The LLM generated a draft documentation note based on the de-identified clinical context. The draft used placeholders — “the patient,” “MRN-XXXXX” — instead of real identifiers. The LLM never saw real patient data.

The PHI firewall then re-identified the draft, replacing placeholders with the actual patient identifiers, and returned the re-identified draft to the physician’s workstation. The physician reviewed the draft, made corrections, and saved the final note to the EHR.

The local LLM deployment

The hospital’s IT team provisioned a dedicated GPU server within their data center. The server ran an open-source LLM with 70 billion parameters, selected for its performance on clinical documentation benchmarks. The model was quantized to reduce memory requirements, allowing it to run on four A100 GPUs rather than eight.

The model was not fine-tuned on hospital data. Fine-tuning would have required the model to process real patient records during training, which would have violated the data isolation requirement. Instead, the team used a model that had been pre-trained on publicly available medical literature and clinical guidelines. The documentation quality was adequate for draft generation, though physicians reported that it occasionally used terminology that was not consistent with the hospital’s documentation standards.

The team addressed this with prompt engineering rather than fine-tuning. The context builder included hospital-specific documentation guidelines, preferred terminology, and template structures in every prompt. The LLM followed these guidelines with reasonable consistency, though not as reliably as a fine-tuned model would have.

What we gave up

The local LLM was less capable than the commercial APIs. The commercial models were trained on far more data and had more refined instruction-following capabilities. Physicians who had used commercial LLMs outside of work noticed the difference. The local model produced drafts that required more editing — an average of four minutes of editing per note, compared to an estimated two minutes if a commercial model had been available.

The second trade-up was cost. The GPU server cost $180,000 to procure and approximately $4,000 per month to operate. Commercial LLM APIs would have cost approximately $8,000 per month for the same usage volume but required no capital expenditure. The hospital accepted the higher total cost because the commercial option was not compliant.

The third trade-off was model updates. Commercial LLM providers update their models regularly. The hospital’s local model was static unless the IT team manually deployed a new version. The team planned quarterly model updates, but each update required validation testing to ensure that the new model’s outputs met clinical documentation standards.

Results

In the first six months, physicians used the documentation assistant for approximately thirty percent of their clinical notes. Documentation time for notes that used the assistant decreased by thirty-five percent on average, from forty-two minutes to twenty-seven minutes per note. Physicians reported that the primary time savings came not from the draft quality but from the assistant’s ability to pre-populate structured fields — medication lists, problem lists, assessment codes — from the clinical context.

The HIPAA compliance posture was validated by the hospital’s internal audit team and by an external security assessment. No PHI was detected in any log, network trace, or inference server memory dump during the assessment. The PHI firewall’s audit trail was complete and consistent.

The decision heuristic

If your AI use case requires processing PHI, the compliance constraint is not negotiable and should not be treated as an afterthought. The architecture must enforce data isolation at the infrastructure level, not at the application level. An application-layer guardrail that prevents the model from returning PHI is insufficient if the model has seen PHI during inference. The PHI firewall pattern — de-identify before inference, re-identify after inference — provides infrastructure-level enforcement. It is more complex than calling a commercial API. It is also the only pattern that can pass a HIPAA audit.