StacklogIn July, our Datadog invoice crossed $9,400 for a month where nothing unusual happened. No incident, no traffic spike, no new service we'd forgotten to budget for. Just the slow, compounding effect of eighteen months of adding hosts, custom metrics, and APM instrumentation without ever once going back to check whether we still needed all of it. That bill is the reason this post exists.
We're a fourteen-person engineering team running about sixty hosts across three environments, with a mix of long-running services and a handful of serverless functions. Datadog had been genuinely good to us for the first two years — the UI is excellent, the APM traces are easy to read, and when something broke at 3 a.m. the dashboards told a coherent story fast. The problem was never the product. It was the bill, and the fact that the bill's shape made no sense to anyone trying to plan a budget six months out.
So we did the thing every infra team eventually considers and mostly talks themselves out of: we priced out what it would actually cost, in time and in dollars, to run our own observability stack instead of paying for Datadog's. This post is the honest version of that math, not the sanitized one.
What we were actually paying for
Breaking down the Datadog bill took longer than it should have, because the pricing is modular by design: infrastructure hosts, APM hosts, log ingestion and retention, custom metrics, and synthetic monitoring are each billed separately, and none of them show up in one place that explains the total. Once we pulled it apart:
- Infrastructure monitoring: ~60 hosts at $15/host/month — roughly $900/mo
- APM & distributed tracing: ~40 hosts at $31/host/month — roughly $1,240/mo
- Log management: ~180GB/day ingested, with 15-day retention — roughly $3,600/mo
- Custom metrics overage: we were well past the included allotment — roughly $2,900/mo
- Synthetic monitoring and the rest: roughly $760/mo
Add it up and you get close to that $9,400 figure. The custom metrics line was the real shock — nobody had been tracking cardinality, and a handful of tags with high-cardinality values (user IDs, request paths with embedded IDs) had quietly multiplied our metrics count by a factor we hadn't noticed.
Picking the best observability tool for self-hosting
We evaluated four self-hosted stacks before picking one, and I want to be specific about what we compared, because "self-hosted observability" isn't one product, it's an assembly job:
- Prometheus + Grafana + Loki + Tempo — the "Grafana Labs" stack, all open source, each component doing one job.
- VictoriaMetrics + Grafana — a drop-in Prometheus-compatible metrics store that's dramatically cheaper on storage and CPU at our scale.
- SigNoz — an open-source, OpenTelemetry-native all-in-one with its own UI, closer to a Datadog replacement out of the box.
- OpenObserve — newer, single-binary, appealing for its simplicity but with a smaller ecosystem and fewer people we could ask for help when it broke.
We ended up on VictoriaMetrics for metrics, Loki for logs, Tempo for traces, and Grafana tying it together. It's not the single "best observability tool" in some abstract sense — SigNoz is arguably a better out-of-the-box experience if you're starting from zero — but it was the best fit for a team that already knew Prometheus's data model and wanted to swap the expensive part (storage and query cost at scale) without relearning everything else.
Pricing tiers, trials, and the fine print
Before committing, we ran two-week trials against our real traffic: Grafana Cloud's free tier (capped at 10k metrics series, which we blew past in about four hours), Grafana Cloud's paid pricing tiers (which start reasonable and scale in a way that, if we're honest, looks a lot like Datadog's curve once you factor in log volume), and a fully self-managed deployment on our own Kubernetes cluster. The self-managed trial was the only one that gave us a real read on steady-state cost, because the hosted tiers all have generous trial credits that hide the actual bill until month two.
One thing worth saying plainly: if your team is under ten engineers or you don't already run Kubernetes, paying for a hosted tier (Grafana Cloud, Honeycomb, even staying on Datadog at a smaller scale) is probably still the right call. Self-hosting observability is not free labor, and the "worth it" line depends entirely on your host count and your tolerance for owning another stateful system.
The migration, three weekends
We didn't do a flag-day cutover. For three weekends we ran both stacks in parallel, shipping the same metrics and logs to Datadog and to our new pipeline, and diffing dashboards by hand until the numbers matched. The Prometheus scrape config ended up looking like this once we'd ported our service discovery over from Datadog's agent tags:
scrape_configs:
- job_name: 'api-services'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
action: keep
regex: 'api-.*'
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
remote_write:
- url: "http://victoriametrics:8428/api/v1/write"The hardest part wasn't metrics, it was alerting. Datadog's monitors had five years of accumulated tuning baked into thresholds nobody remembered the reasoning for. We rewrote every alert rule from scratch in Grafana's alerting engine rather than trying to port the logic blindly, which took longer but meant we actually understood our alerts again by the end of it.
What it costs us now
Steady state, six months in: three dedicated nodes for the observability stack (metrics, logs, traces storage), running on our existing cloud provider, cost us about $1,850/month in compute and block storage. Add maybe 15 hours a month of an engineer's time for maintenance and upgrades — call that another $1,500/month in loaded cost if you want to be fair about it — and we're at roughly $3,350/month all-in, against the $9,400 we were paying before. That's not "it's free now," it's "it's a third of the price and we own the data."
Would we do it again
Yes, but I'd set expectations lower going in. We lost about two weeks of feature velocity across the team during the migration and the first month of alert tuning afterward. If your Datadog bill is under $3,000/month, the math almost certainly doesn't favor self-hosting yet — the engineering time costs more than the savings. Past that point, and especially once custom metrics and log volume start driving your bill rather than host count, the self-hosted stack starts paying for itself within three to four months. We're not going back, but I understand now why most teams don't make this switch until the bill actually hurts.