You Can't Fix What You Can't See: Building Deep Observability into a Multi-Tenant Healthcare Platform

Some early panic to start the day

It is 8 AM.

A practice in the West Coast has just reported 'the site is down'.

You have a quick look at your synthetic checks dashboard and it's all green. You think it might be their Internet and have a sigh of relief.

A minute later, 2 more practices from the same region report they can't access patient schedules. They have patients on chairs and need immediate resolution. Your first hunch is that something has gone wrong in the West Coast.

Then your hypothesis falls apart as more practices from across the country start reporting they have the same issue.

Now you are back to square one staring at a platform of 100s of services, databases, clusters and third party integrations serving 1000s of healthcare professionals every day. The dashboards you have spent months building, covering every service, now resembles a traffic light. Some red, some are amber but most are unhelpfully green.

Your mind is clouded with questions:

Is the database saturated?

Is it the change you deployed to login service 3 days ago?

Is it your cloud provider having an outage?

Time is of the essence. Every minute spent chasing the wrong lead delays recovery, erodes customer trust and increases both financial and reputational risk.

Before you can fix the problem, you need an answer to a simple question. What exactly is happening?

Surprisingly, this is where most production incidents spend the majority of their time.

The case for “Observability”

Much like the real world, modern software systems suffer from an abundance of data but a scarcity of information.

Modern software emits telemetry in greater volume and richness than ever before. The problem is that most of it was designed to answer the questions we were expected to ask.

We have metrics, logs, traces, dashboards, events and alerts.

Yet during an incident, engineers still find themselves jumping between tools, correlating timestamps, forming and discarding hypotheses, and relying more on intuition than evidence.

Production incidents, however, rarely follow the paths we prepared for.

Why is only this particular workflow affected?

Why are only a handful of practices affected while everyone else is healthy?

Did the deployment from three days ago trigger this, or is it merely a coincidence?

These are not questions you can anticipate. But ones that you need to investigate.

This is where traditional monitoring reaches its limits.Observability begins where prediction ends.

Achieving “Observability”

Observability is often mistaken for another monitoring platform or simply the collection of logs, metrics and traces. It is neither.Observability is a property of a software system - How well can you infer the internal states of the system with only its external outputs. An observable system exposes enough evidence that engineers can reconstruct why it behaved the way it did, even when the failure mode is entirely new, without deploying new code or relying on tribal knowledge.

Thus, the real test that makes or breaks an observability system is whether it can handle completely abstract and random questions about the system without any prior preparation.

For example,can you answerwhy an application is slow while you have no information about how it is written or where it is running?

If you find a particular characteristic in a slow or failed request randomly, can you then find all requests with that particular characteristic?

If you find a bunch of interesting events, can you then drill down at a region or tenant or user level?

And the holy grail of all observability questions:

Can you figure out the cause even if you have never seen this particular issue before?

No matter how hard a vendor tries to convince you, buying an observability tool does not magically produce observability in your software. Much like how Kubernetes does not make your software inherently reliable.

Observability is both a technical capability and a culture. It requires an environment that values hypothesis-driven investigation over intuition alone.Rich telemetry is only one part of the equation. Engineers must also be able to correlate, explore and reason about that data quickly enough to form and validate hypotheses while an incident is ongoing.

This means a good observability system needs three things to work at the same time:

1. Ability to gather and store the required data
2. Ability to process data
3. Ability to inspect the data and get the right inferences

Modern distributed applications generate telemetry at an extraordinary scale. Every request can produce dozens of spans, hundreds of log entries and multiple metrics, all enriched with high-cardinality attributes.Retaining this level of detail is essential for answering the kinds of questions observability demands, but it also comes at a cost. A lot of cost.

Inevitable tradeoffs. Or is it?

This brings Engineering organizations to accept one of the compromises. Either burn a hole in your pocket for high cardinality observability or reduce telemetry being ingested to curb costs.

Neither were acceptable to us, which led us to stop relying on external vendors and explore implementing our own observability stack. We wanted to preserve deep observability without making it too expensive. That meant moving away from tightly coupled, vendor-managed observability platforms and building an observability stack around open standards.

This led us to OpenTelemetry, a vendor-neutral observability framework that standardizes how applications are instrumented and how telemetry is generated, collected, processed and exported. OpenTelemetry gives us the freedom to choose a backend that best suited our requirements.

For storage and analytics, we chose ClickHouse, a high-performance columnar database built for analytical workloads. Its ability to efficiently compress, store and query massive volumes of high cardinality telemetry allowed us to retain the rich context needed for effective investigations while keeping infrastructure costs predictable. Instead of sacrificing observability to control costs, we were able to design a platform that delivered both.

Observability in a multi-tenant environment

Observability becomes significantly harder in a multi-tenant platform because every tenant uses the application differently.

One practice may perform hundreds of appointment searches every hour. Another may primarily be processing billing. A third may rely heavily on external integrations. All of them share the same infrastructure, yet experience the application in completely different ways.

This makes aggregate infrastructure metrics far less useful than they first appear. CPU utilization may look perfectly healthy while appointment scheduling is degraded for only a handful of practices. Overall request latency may remain well within acceptable limits even though a specific workflow is consistently failing for one region or one customer segment. In a multi-tenant platform, business context isn't additional metadata. It is the context that turns telemetry into actual evidence.

From panic to peace

How does observability help in diagnosing issues like the one we just got hit at 8 AM?

A possible flow of diagnostics will be like this.

1. Filter requests originating from the affected practices.
2. Compare them against requests from healthy practices to identify what is different.
3. Discover that only the scheduler workflow exhibits elevated latency, while other workflows continue to perform normally.
4. Open a distributed trace for one of the affected requests.
5. Observe that the majority of the request time is spent waiting on a database operation.
6. Compare traces from before and after the incident and discover that the scheduler workflow has experienced a sustained latency increase over the past three days.
7. Drill into the database span and identify that the query being executed changed in the most recent release and is significantly slower than its predecessor.
8. Correlate the regression with the scheduler service deployment from three days earlier.
9. Roll back the deployment or release a fix.

Notice that we did not do any guesswork or rely on an engineer who knew the system inside out.Every step of the investigation was driven by evidence already present in the telemetry. Each observation narrowed the problem space until we were led elegantly to the cause.This is exactly what observability gives you. Not ready made answers, but enough context to ask progressively better questions until the answer reveals itself.

Major lessons learned in implementing observability at CareStack

1. In a multi-tenant SaaS, the ability to slice telemetry by tenant, workflow or deployment is often the difference between minutes and hours of investigation. Cost effective observability backends allow you to afford higher cardinalities that you otherwise could not.
2. Good observability requires deliberately deciding what context future engineers will need during incidents. It deserves the same attention as any other product feature.
3. Observability is an engineering capability. Tools collect data. Engineers investigate. One without the other has limited value. Your observability system should enable curious engineers to excel.
4. Observability is not just for incident resolution. It gives you enough context to predict future incidents and avoid them. It opens the door to entirely new capabilities: automated incident detection, release impact analysis, AI-assisted investigations and proactive identification of reliability risks before customers notice them.

The 8 AM incident wasn't extraordinary. Every engineer on call has suffered it at least once.

The difference lies in how quickly the investigation moves from "Something is wrong" to "We know exactly why."

You can't fix what you can't see. Deep observability ensures you can.