Build the shared services every integration program ends up needing — one notification API instead of SMTP scattered through forty applications, cached reference data, a durable error record with replay, and monitoring that opens a ticket rather than an email nobody reads.
System and Utility Operations: Monitoring, Notification, and Reference APIs
Objective: Stop every project from rebuilding the same four things. By the third or fourth integration, a program has SMTP credentials in several applications, reference data cached inconsistently, error handling that differs per developer, and alerting that goes to a mailbox nobody owns. This plan builds those as shared, versioned services so the next project consumes them rather than reinventing them — and so an operations team has one place to look when something breaks.
What's Getting Built
- Notification API (Process API) — one notification path for every app — email, chat, and ITSM ticket, templated and rate-limited ·
POST /notify/email, POST /notify/chat, POST /notify/ticket
- Reference Data API (System API) — cached lookups served from one place with a stated freshness ·
GET /reference/{set}, GET /reference/{set}/{key}
- Error & Audit API (System API) — durable record of every failure, correlation ID carried end to end ·
POST /errors, GET /errors/{correlationId}
- ITSM connector (target) — deduplicated incidents raised into the ITSM instance ·
POST /incident
Prerequisites
- Anypoint Platform org with API Manager and a non-production deployment target.
- Secrets Manager or secure properties configured — shared services hold the credentials everything else would otherwise duplicate.
- Named platform owner, because a shared service with no owner rots faster than duplicated code.
- Access to the notification channels in scope: SMTP relay, chat workspace, and the ITSM instance.
- An agreed correlation identifier convention across applications, without which tracing is guesswork.
Choose your path
- Library first — Choose when teams are few and co-located — publish shared connectors, error types, and policies as reusable artifacts, with no runtime dependency between applications.
- Shared services — Choose when many teams ship independently — deploy the utilities as APIs so a change lands once, accepting that they become production dependencies that need their own availability target.
Platform notes
Anypoint Platform
- Anypoint Monitoring covers infrastructure and application metrics; this plan adds the business-level signal it cannot infer, such as a file that never arrived.
- MuleSoft's public Catalyst repositories are worth reading before building: the metrics toolkit collects and visualizes platform metrics, and the error handler and circuit breaker policy projects cover patterns most teams write from scratch.
ITSM and alerting
- Alerts must be deduplicated before they reach ITSM, or one bad night creates two hundred incidents and the team learns to ignore the queue.
- Attach the correlation identifier to every incident so an operator can retrieve the full trace without asking a developer.
Notification channels
- Keep templates and recipients in configuration rather than in code, so a distribution list change is not a release.
- Rate-limit per template and per recipient — a retry storm that emails a customer four hundred times is worse than the original failure.
Architecture
1. Business requirements
- One notification path for every application, with templates and recipients owned outside code.
- Reference data served from one place with a stated freshness, rather than cached per application.
- Every failure recorded durably with enough context to replay it, not just logged and lost.
- Operational signal reaching the team that owns it, through the channel they actually watch.
2. Notification service
- Expose one API accepting a template identifier, recipients, and a payload; keep channel selection behind it.
- Deduplicate on template and key within a window, so a retry loop cannot fan out to hundreds of messages.
- Support escalation paths — chat first, ticket if unacknowledged — rather than sending everything everywhere.
- Record every notification sent, because 'nobody told us' is otherwise unanswerable.
3. Reference data service
- Serve lookups — country codes, unit conversions, mappings, feature configuration — from one cached API.
- Publish a freshness timestamp with every response so consumers know what they are trusting.
- Invalidate on change events rather than relying on time-to-live alone, and expose a manual invalidation path for operations.
- Version the payload contract, since reference data outlives the applications that consume it.
4. Error and audit service
- Define a shared error taxonomy — transient, validation, downstream, and fatal — and make every application classify against it.
- Persist failures with the correlation identifier, the inbound payload reference, and enough context to reprocess.
- Offer a replay path that is idempotent by design, so reprocessing is a routine action rather than a risky one.
- Keep audit records immutable and separate from application logs, which rotate away exactly when they are needed.
5. Monitoring and operational integration
- Emit business-level signals alongside technical metrics: files expected, records processed, and queues aging.
- Alert on the absence of an expected event, which is the failure mode dashboards miss most often.
- Open and resolve ITSM incidents automatically, and close the loop when the underlying condition clears.
- Publish a per-interface health view that operations can read without understanding the implementation.
6. Security and operability
- Hold every channel and downstream credential in the platform's secrets management, never in application properties.
- Apply client ID enforcement and rate limiting to the shared services themselves — a utility API is a single point of failure worth protecting.
- Keep payloads out of error records where they carry sensitive data; store a reference and restrict retrieval to an authorized role.
- Give shared services their own availability target and change process, because every consumer inherits their downtime.
- Review notification templates for data exposure — an alert email that quotes a payload leaks it to whoever is on the distribution list.
Sample payload — Notification request
{
"templateId": "interface.failure.v2",
"correlationId": "b3f19c2e-77a4-4a51-9d2c-88110f4c2d31",
"severity": "high",
"dedupeKey": "orders-inbound:2026-07-31:connection-refused",
"dedupeWindowSeconds": 900,
"channels": ["chat", "itsm"],
"escalateAfterSeconds": 1800,
"recipients": { "group": "integration-operations" },
"context": {
"interface": "orders-inbound",
"environment": "production",
"errorType": "downstream",
"payloadRef": "s3://error-store/2026/07/31/orders-inbound/0f21.json"
}
}
Testing and validation
- Fire the same notification a hundred times inside the dedupe window and confirm one message and one incident result.
- Fail a downstream system and confirm the error record carries enough context to replay without touching the source.
- Replay a failed message twice and confirm the target is unchanged after the second attempt.
- Stop an expected inbound feed and confirm the absence alert fires without any error having occurred.
- Invalidate a reference dataset and confirm consumers pick up the change without a redeploy.
Phased rollout
Phase 1 — Connect
- Build the error and audit service first, since every other utility depends on its taxonomy.
- Agree the correlation identifier convention and retrofit it into the two busiest existing interfaces.
Phase 2 — Map and validate
- Add the notification service and migrate one application off its own SMTP configuration.
- Introduce absence alerting for the interfaces with a known schedule, and tune before widening.
Phase 3 — Production
- Migrate applications as they are next touched rather than in a big-bang refactor.
- Hand the per-interface health view to operations with named ownership per interface.
Success metrics
- Applications consuming the shared services rather than their own implementation, trending up per release.
- Mean time to diagnose, measured from alert to identified cause, which is what correlation buys.
- Duplicate incidents per outage, which should approach one.
- Failures resolved by replay rather than by manual data fixes.
Troubleshooting
- Alert storm during a downstream outage — Likely cause: deduplication applied per message rather than per condition. Fix: Dedupe on a condition key with a window, and escalate the single incident instead of repeating it.
- Errors recorded but not replayable — Likely cause: payload reference not captured, so only the message text survives. Fix: Store a payload reference with every error and keep the source retrievable for the retention window.
- Reference data stale in one application only — Likely cause: local caching layered on top of the shared service. Fix: Remove local caches or honor the published freshness and invalidation events.
- Shared service outage takes down every interface — Likely cause: utilities called synchronously in the critical path with no fallback. Fix: Make notification and audit calls asynchronous, and fail open on non-critical utilities.
Deliverable: Shared notification, reference data, and error services with a common taxonomy and correlation convention, absence-aware monitoring wired into ITSM, and a replay path that makes failure recovery routine.