Continuously validates Epic predictive and generative AI model features against local outcomes using the Trust and Assurance Suite pattern: harvest model inputs and predictions, join to observed outcomes, compute performance and fairness metrics, and detect drift before it reaches clinicians.
Tags: healthcare, epic, fhir, ai, analytics, cogito, interoperability, system-api
Validating AI Features
Objective: Epic-hosted and third-party predictive models (deterioration, no-show, sepsis, generative note-drafting) degrade silently as local populations, workflows, and documentation habits drift from the data they were validated on. The failure is not a crash; it is a still-green dashboard while calibration quietly rots and clinicians keep trusting a score that no longer earns trust. Done looks like: every scored prediction is captured with its feature snapshot and the observed outcome, local AUROC, calibration, alert-fatigue, and subgroup-fairness metrics are recomputed on a schedule, and any metric breaching a governance threshold raises an alert and can gate the model from surfacing in Hyperdrive. This accelerator packages the Validating AI Features Epic Data Sharing Playbook as a MuleSoft-anchored solution plan.
What's Getting Built
- FHIR (System API) — reads scored RiskAssessment predictions and bulk-exported outcome resources from Epic ·
GET /RiskAssessment, GET /Observation, GET /Encounter, GET /$export
- Join and Validate (Process API) — pairs each prediction with its observed outcome and drives metric recomputation and gating ·
POST /predictions/harvest, POST /outcomes/join, POST /metrics/recompute
- Prediction State (Object Store) — holds open predictions keyed by patient plus encounter plus model version until the outcome window closes ·
GET /{patient|encounter|modelVersion}, PUT /{patient|encounter|modelVersion}, DELETE /{key}
- Validation Metrics Engine (Batch flow) — computes local AUROC, calibration, Brier, subgroup fairness, and PSI drift per model per window ·
SCHEDULER on outcome-window close, compute AUROC/PSI/calibration
- Alert and Suppression (Experience API) — on a governance-threshold breach raises an alert and writes the model suppression flag Hyperdrive honors ·
POST /alert, PUT /models/{id}/suppression, GET /models/{id}/suppression
- Cogito Warehouse (target) — persists the metric time series for dashboards and audit ·
JDBC INSERT metric_series, GET /metrics/{model}
Prerequisites
- Epic Cognitive Computing Platform enabled with the model registered and Trust and Assurance Suite monitoring turned on for the target predictive or generative model.
- FHIR R4 read access via OAuth 2.0 client credentials (backend services) with scopes for Observation, RiskAssessment, Encounter, Patient, and DocumentReference.
- A ground-truth outcome definition agreed with clinical governance for each model (e.g., ICU transfer within 12 hours, 30-day readmission, no-show at appointment).
- Anypoint Platform with CloudHub 2.0 or RTF, Object Store v2 for prediction-outcome join state, and an Anypoint MQ or VM queue for the dead-letter path.
- A model governance policy defining metric thresholds, evaluation cadence, protected subgroups, and the escalation and suppression owners.
- Cogito or an external analytics warehouse target (JDBC, Snowflake, or S3) to persist metric time series for dashboards and audit.
1. Epic interfaces and standards
This accelerator implements the Validating AI Features Epic Data Sharing Playbook. Register the connecting app on Epic on FHIR and confirm the customer has the relevant Epic modules licensed before building.
- Epic Cognitive Computing Platform and Trust and Assurance Suite
- FHIR R4 RiskAssessment and Observation resources
- FHIR R4 Bulk Data Export (Flat FHIR) for backfill of outcomes
- Cogito and Reporting Workbench as the metrics warehouse target
- SMART on FHIR backend services OAuth 2.0 client-credentials flow
Architecture
2. Business requirements
- Treat every model output as a claim that must later be scored against a real outcome, not as a fact to be trusted on emission.
- Capture the prediction, the feature snapshot, the model version, and the scoring timestamp at inference time so drift can be attributed to inputs versus model.
- Recompute local discrimination, calibration, and subgroup fairness on a governance cadence rather than relying on the vendor's original validation study.
- Make suppression a first-class outcome: a breaching model must be gate-able out of Hyperdrive without a code deploy.
- Keep a defensible audit trail linking each metric value to the exact prediction-outcome pairs that produced it.
3. Harvesting predictions and outcomes
- Pull scored predictions from Epic as RiskAssessment resources, retaining prediction.probabilityDecimal, method, and the model identifier in RiskAssessment.basis.
- Harvest ground-truth outcomes via targeted FHIR reads or scheduled Bulk Data Export of Observation and Encounter, keyed by patient and encounter.
- Normalize heterogeneous outcome signals to a single boolean or ordinal label per model in DataWeave, with the label logic version-stamped.
- Persist the open prediction in Object Store v2 keyed by patient plus encounter plus model version until its outcome window closes.
- Correlate every record end-to-end with a correlation ID so a metric anomaly can be traced back to source resources.
4. Computing validation metrics
- Compute AUROC or AUPRC, calibration slope and intercept, and Brier score per model per evaluation window in the metrics engine.
- Compute alert-fatigue and workload metrics such as number-needed-to-evaluate and firing rate, which drift faster than discrimination.
- Compute the same metrics sliced by protected subgroup and flag statistically meaningful gaps, not just aggregate performance.
- Compare current-window feature distributions to the reference window using population stability index to separate input drift from label drift.
- Version every metric computation so a recomputation on the same data is reproducible for audit.
5. Orchestration and gating
- Run harvest and join on a short cadence, but run metric recomputation only after each prediction's outcome window has closed.
- On a governance-threshold breach, emit an alert and write a suppression flag that the Experience API and Hyperdrive integration honor.
- Route unresolvable predictions whose outcome window expires without a label to a reconciliation queue, not silent deletion.
- Keep model version as a partition key everywhere so a vendor model upgrade starts a clean metric series instead of blending regimes.
- Alert on the absence of an expected prediction volume, which is the failure mode dashboards miss most often when a model silently stops firing.
6. Security and compliance
- Encrypt PHI in transit and at rest.
- Audit-log every access to and modification of patient data, and carry a correlation ID end to end.
- Enforce minimum-necessary access for integration operators and support users.
- Scope OAuth backend-services tokens to read-only, resource-specific access; the blueprint never writes clinical data back, only reads predictions and outcomes.
- De-identify or hash direct identifiers before persisting feature snapshots to the metrics warehouse, retaining only the surrogate key needed to join.
- Restrict the suppression-flag write path to a governance-owned role, so gating a model is an auditable, authorized action, not a side effect.
- Segregate subgroup-fairness data with additional access controls, since protected-attribute metrics are sensitive even when aggregated.
Sample payload — Epic RiskAssessment for a scored deterioration model prediction
{
"resourceType": "RiskAssessment",
"id": "ra-deterioration-88213",
"status": "final",
"subject": { "reference": "Patient/e63wRTbB4uEapmXVprHIqA3" },
"encounter": { "reference": "Encounter/eS4hp5CmQ2Xkl" },
"occurrenceDateTime": "2026-08-20T14:32:00Z",
"method": { "text": "Epic Deterioration Index v3.2" },
"basis": [ { "display": "model:epic-deterioration-index", "reference": "#featureSnapshot" } ],
"prediction": [
{
"outcome": { "text": "ICU transfer or rapid response within 12 hours" },
"probabilityDecimal": 0.71,
"whenPeriod": { "start": "2026-08-20T14:32:00Z", "end": "2026-08-21T02:32:00Z" }
}
]
}
Testing and validation
- MUnit-test the DataWeave label-derivation logic against fixtures for true positive, false positive, censored, and outcome-window-expired cases.
- Feed a golden dataset with a known AUROC and confirm the metrics engine reproduces it within tolerance.
- Inject a synthetic feature-distribution shift and assert the population stability index crosses the drift threshold.
- Simulate a prediction whose outcome never arrives and verify it lands in the reconciliation queue at window close, not the happy path.
- Verify a governance-threshold breach both emits an alert and sets the suppression flag the Experience API reads.
- Verify subgroup-sliced metrics compute correctly and that a small-cell subgroup is suppressed rather than reported on noise.
Phased rollout
Phase 1 — Connect
- Establish OAuth backend-services access and confirm RiskAssessment reads and a Bulk Export outcome pull against a non-production Epic environment.
- Register one target model and its ground-truth outcome definition with clinical governance.
Phase 2 — Map and test
- Build the prediction-outcome join in Object Store and validate label derivation with MUnit against clinician-adjudicated cases.
- Backfill a historical window and reproduce the vendor's reported metrics to calibrate local thresholds.
Phase 3 — Production
- Enable scheduled harvest, join, and metric recomputation with alerting on breach and on missing prediction volume.
- Turn on the suppression flag path and rehearse a governed gate-out and restore of a model in Hyperdrive.
Success metrics
- Local AUROC or AUPRC per model per evaluation window, trended against the reference baseline.
- Calibration slope and intercept, with an alert when the slope drifts outside the governed band.
- Population stability index on model features, separating input drift from performance drift.
- Prediction capture completeness: share of surfaced predictions successfully paired with an outcome or explicitly censored.
- Mean time from metric breach to suppression or governance acknowledgement.
Troubleshooting
- Local AUROC looks far worse than the vendor's published number. — Likely cause: The local ground-truth outcome definition does not match the label the model was trained against, so correct predictions are scored as misses. Fix: Re-adjudicate a sample of prediction-outcome pairs with clinical governance and align the DataWeave label logic to the model's intended outcome and time window.
- Metrics are stable but clinicians report the score feels wrong. — Likely cause: Aggregate discrimination is masking a calibration or subgroup problem that only surfaces in a slice. Fix: Inspect calibration curves and subgroup-sliced metrics; a drifted calibration intercept degrades trust even when ranking-based AUROC holds.
- Prediction volume drops to near zero but no alert fires. — Likely cause: Monitoring watches for bad metric values, not for the absence of expected predictions when a model stops firing upstream. Fix: Add an expected-volume floor per model and window and alert on the absence of predictions independently of metric-value thresholds.
- Metric series shows a sudden step change with no clinical explanation. — Likely cause: A vendor model version upgrade blended two regimes into one time series because model version was not a partition key. Fix: Partition all state and metrics by model version and start a fresh baseline series on any version change.
Deliverable: A deployable Anypoint blueprint: a System API to Epic Cognitive Computing/FHIR endpoints, a Process API that harvests predictive-model inputs and outcomes, a validation-metrics engine computing local performance and drift, and a monitoring dashboard feed to Cogito, with sample datasets, MUnit suites, and runbooks.