Production AILLM observabilitycost optimizationtoken usageLLMOps

LLM Cost Monitoring Tools Compared

Pramod Dutta · Jul 1, 2026 · 12 min read

LLM cost monitoring is the practice of tracking token usage, per-request spend, and per-user or per-feature cost across every call your app makes to a model provider, then turning that data into alerts and dashboards before a bill surprises you. If you have ever shipped a feature that felt cheap in testing and then watched your OpenAI or Anthropic invoice triple in production, you already know why this matters. The fix is not "use fewer tokens." The fix is visibility: knowing which endpoint, which customer, and which prompt template is burning your budget, in near real time, broken down by model.

This article walks through the tools people actually reach for in 2026: Langfuse, Helicone, OpenLLMetry (via OpenTelemetry), Portkey, and a plain Postgres-based DIY approach. For each one you get working code, what it's good at, and where it falls short. By the end you should be able to pick one without a week of trial spikes.

Why LLM cost monitoring is different from normal APM

Traditional application performance monitoring cares about latency and error rate. LLM cost monitoring adds a third axis that traditional tools were never built for: token-denominated spend that varies wildly by model, by prompt length, and by whether a response got cached or not.

A few things make this genuinely different:

Cost per call isn't fixed. The same endpoint can cost 10x more depending on how long the user's input is, whether retrieval augmented generation stuffed 8,000 tokens of context into the prompt, or whether the model decided to think longer on a hard question.
Provider pricing changes model to model. A request routed to a cheaper model tier costs a fraction of one routed to a flagship model. If your app does dynamic model routing, your monitoring has to track spend per model, not just per endpoint.
Streaming responses complicate token counting. You often don't know the exact completion token count until the stream closes, so cost has to be computed after the fact and reconciled.
Caching and retries distort naive request counts. A retried call that hits a semantic cache costs nothing, but a naive "count requests" metric will still show it as one more call.

This is why generic tools like Datadog or Grafana alone leave a gap. They're great at latency and uptime, weak at token-level cost attribution unless you feed them the right metrics yourself.

Langfuse: open source and self-hostable

Langfuse is the most commonly adopted LLM observability platform in production RAG and agent stacks, largely because it's open source, has a generous free tier, and self-hosts cleanly with Docker.

Instrumenting a Python app with the OpenAI SDK takes about five lines:

from langfuse.openai import openai

client = openai.OpenAI()

response = client.chat.completions.create(
    model="gpt-4.1-mini",
    messages=[{"role": "user", "content": "Summarize this support ticket."}],
)

Langfuse patches the OpenAI client so every call, including token counts, latency, and computed cost, gets shipped to the Langfuse backend automatically. No manual span creation needed for the simple case.

For more control, or when you're not using the OpenAI SDK directly (say, calling Anthropic's Messages API), you wrap calls with the @observe decorator:

from langfuse.decorators import observe, langfuse_context
import anthropic

client = anthropic.Anthropic()

@observe()
def answer_question(prompt: str):
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    langfuse_context.update_current_observation(
        usage={
            "input": response.usage.input_tokens,
            "output": response.usage.output_tokens,
        },
        model=response.model,
    )
    return response.content[0].text

Once traces are flowing, the dashboard groups cost by trace name, by user (if you attach a user_id), and by time window. You can set budget alerts per project and export raw usage data for finance to reconcile against the provider invoice.

Where Langfuse shines: prompt-level tracing for agents and RAG pipelines, session grouping for multi-turn conversations, and a scoring layer if you also want quality evals next to cost data.

Where it's weaker: it doesn't do request routing or caching itself, it's purely observability. You still need something else if you want automatic fallback to a cheaper model.

Helicone: proxy-based, zero code changes to your prompt logic

Helicone takes a different architectural approach. Instead of patching your SDK client, you route requests through a proxy by changing the base URL. This means you get cost tracking without touching your prompt or business logic at all.

from openai import OpenAI

client = OpenAI(
    api_key="your-openai-key",
    base_url="https://oai.helicone.ai/v1",
    default_headers={
        "Helicone-Auth": "Bearer your-helicone-key",
        "Helicone-User-Id": "user-8823",
        "Helicone-Property-Feature": "support-ticket-summarizer",
    },
)

response = client.chat.completions.create(
    model="gpt-4.1-mini",
    messages=[{"role": "user", "content": "Draft a reply to this refund request."}],
)

The Helicone-Property-* headers are the key feature here. You tag every request with arbitrary metadata (feature name, tenant ID, environment) and then slice the cost dashboard by any combination of tags. This is exactly what you want when a single API key serves ten different features and finance is asking "which one is expensive."

Helicone also supports caching natively at the proxy layer, so you can flip on a header and semantically-duplicate requests get served from cache instead of hitting the model again, cutting cost without any code change:

default_headers={
    "Helicone-Auth": "Bearer your-helicone-key",
    "Helicone-Cache-Enabled": "true",
}

Where Helicone shines: near-zero integration cost, per-tenant and per-feature cost breakdown via headers, built-in caching and rate limiting at the proxy layer.

Where it's weaker: because it's a proxy, you're adding a network hop between your app and the provider. For latency-sensitive paths, self-hosting the proxy (Helicone supports this) mitigates most of that concern.

OpenTelemetry with OpenLLMetry: vendor-neutral instrumentation

If your org already has an observability stack built around OpenTelemetry (traces flowing to Datadog, Honeycomb, Grafana Tempo, or similar), the cleanest path is OpenLLMetry, a set of OTel instrumentations specifically for LLM calls. This avoids locking your cost data into a dedicated LLM tool and instead lets it live next to your existing infra metrics.

from traceloop.sdk import Traceloop

Traceloop.init(app_name="support-bot", disable_batch=False)

That single init call auto-instruments the OpenAI, Anthropic, and several other SDKs. Spans include gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens, and gen_ai.request.model, following the OpenTelemetry semantic conventions for generative AI. From there, cost is a derived metric: multiply token counts by your provider's per-model rate in a downstream processing step or a Grafana transform.

A minimal cost calculation you'd run in your own pipeline, since OpenLLMetry gives you tokens, not dollars, by default:

PRICING = {
    "gpt-4.1-mini": {"input": 0.00015, "output": 0.0006},   # per 1K tokens, illustrative
    "claude-sonnet-4-5": {"input": 0.003, "output": 0.015}, # per 1K tokens, illustrative
}

def estimate_cost(model: str, input_tokens: int, output_tokens: int) -> float:
    rates = PRICING[model]
    return (input_tokens / 1000) * rates["input"] + (output_tokens / 1000) * rates["output"]

Keep that pricing table in a config file you update whenever a provider changes rates, never hardcode it deep in application logic, because you will forget it's there when prices move.

Where OpenTelemetry shines: no vendor lock-in, cost metrics sit alongside your existing latency and error dashboards, works across every language OTel supports, not just Python and JavaScript.

Where it's weaker: you own the cost calculation and the dashboard building. There's no out-of-the-box "cost by feature" view, you build it in Grafana or whatever you're already using.

Portkey: gateway with routing and budget enforcement built in

Portkey positions itself as an AI gateway rather than pure observability. That means it does everything Helicone does (proxy-based tracking, tagging, caching) plus active cost control: automatic fallback to cheaper models, load balancing across providers, and hard budget limits that block requests once a threshold is hit.

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

client = OpenAI(
    api_key="your-openai-key",
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        api_key="your-portkey-key",
        virtual_key="openai-prod",
        metadata={"feature": "support-summarizer", "tenant": "acme-corp"},
    ),
)

The distinguishing feature is config-driven routing. You define a JSON config that says "try gpt-4.1-mini first, and if it errors or exceeds a latency threshold, fall back to a different provider," and Portkey enforces it at the gateway without any change to your call site:

{
  "strategy": { "mode": "fallback" },
  "targets": [
    { "virtual_key": "openai-prod", "override_params": { "model": "gpt-4.1-mini" } },
    { "virtual_key": "anthropic-prod", "override_params": { "model": "claude-haiku-4-5" } }
  ]
}

Budget limits work per virtual key: set a monthly cap in dollars, and once a key hits it, Portkey returns an error instead of letting spend run unbounded. This is the closest thing to a circuit breaker for LLM cost that exists off the shelf.

Where Portkey shines: teams that need active cost control, not just visibility, meaning automatic downgrade to cheaper models and hard spend caps.

Where it's weaker: more configuration surface than a pure observability tool, and you're trusting a gateway in your critical request path, so test failover behavior thoroughly before relying on it in production.

DIY with Postgres: when you need full control and no new vendor

Sometimes the right answer is not a new tool, it's a table. If your app already logs structured events somewhere, adding cost tracking is a schema and a nightly rollup job.

CREATE TABLE llm_usage (
    id BIGSERIAL PRIMARY KEY,
    created_at TIMESTAMPTZ DEFAULT now(),
    user_id TEXT,
    feature TEXT,
    model TEXT,
    input_tokens INT,
    output_tokens INT,
    cost_usd NUMERIC(10, 6),
    request_id TEXT
);

CREATE INDEX idx_llm_usage_feature_date ON llm_usage (feature, created_at);

Write to it right after every provider call:

def log_usage(user_id, feature, model, response, request_id):
    input_tokens = response.usage.input_tokens
    output_tokens = response.usage.output_tokens
    cost = estimate_cost(model, input_tokens, output_tokens)

    db.execute(
        """
        INSERT INTO llm_usage (user_id, feature, model, input_tokens, output_tokens, cost_usd, request_id)
        VALUES (%s, %s, %s, %s, %s, %s, %s)
        """,
        (user_id, feature, model, input_tokens, output_tokens, cost, request_id),
    )

Then a daily rollup query gives you exactly what a dedicated dashboard would show, scoped precisely to how your product is organized:

SELECT
    feature,
    model,
    date_trunc('day', created_at) AS day,
    SUM(input_tokens + output_tokens) AS total_tokens,
    SUM(cost_usd) AS total_cost,
    COUNT(*) AS request_count
FROM llm_usage
WHERE created_at > now() - interval '30 days'
GROUP BY feature, model, day
ORDER BY day DESC, total_cost DESC;

Where DIY shines: zero vendor lock-in, cost data lives next to your existing business data so you can join it against revenue per customer, full control over retention and schema.

Where it's weaker: you build and maintain the alerting, the dashboard, and the pricing table updates yourself. For a two-person team shipping fast, this is often more overhead than it's worth early on.

Choosing between them

Match the tool to what's actually slowing you down today, not to what looks most complete on a feature comparison page.

You're debugging a RAG or agent pipeline and need to see the full trace, not just cost. Pick Langfuse. The trace view showing every retrieval step and every model call in one waterfall is worth more than the cost number alone.
You have one API key shared across many features and need to know which one is expensive, with the least integration effort. Pick Helicone. Change the base URL, add property headers, done in an afternoon.
You already run OpenTelemetry and don't want a new dashboard to check. Pick OpenLLMetry and pipe spans into your existing Grafana or Honeycomb setup.
You need to actively cap spend or automatically fall back to cheaper models when a budget is close to blown. Pick Portkey. It's the only one on this list that enforces limits rather than just reporting on them.
You're pre-product-market-fit and don't want another vendor bill while you're trying to reduce your model bill. Do the Postgres table. It takes an afternoon and gives you 80% of what a dashboard tool provides.

A pattern worth calling out: none of these are mutually exclusive. A common production setup is Helicone or Portkey as the proxy layer for routing and caching, with Langfuse running alongside for deep trace-level debugging on the subset of requests that actually go wrong. Start with the cheapest thing that answers "which feature is costing me money this week," and only add the second tool once you have a specific problem the first one can't solve.

FAQ

Does LLM cost monitoring slow down my API calls? Proxy-based tools like Helicone and Portkey add one extra network hop, typically single-digit milliseconds if the proxy is geographically close to your app or self-hosted in the same region. SDK-patching tools like Langfuse and OTel-based instrumentation add negligible overhead since they ship telemetry asynchronously in the background rather than blocking the response.

Can I track cost without changing any code at all? Only partially. Provider dashboards (OpenAI's usage page, Anthropic's console) give you account-wide totals with zero integration, but they can't break cost down by feature, user, or tenant. For that granularity you need at minimum a header or a wrapped client call, there's no way around instrumenting something.

How do I handle streaming responses when calculating cost? Wait for the stream to close and read the final usage object most providers attach to the last chunk or a trailing event. Don't try to estimate cost from partial token counts mid-stream, the completion token count isn't final until generation stops, and estimates can be significantly off for long responses.

Should I track cost per user or per feature? Track both if you can. Per-feature cost tells you which part of your product to optimize first. Per-user cost tells you which customers are unprofitable at your current pricing, which matters a lot if you're on a flat subscription and a small number of power users are running up disproportionate model spend.

What's the difference between token count and actual dollar cost? Token count is provider-agnostic and stable. Dollar cost depends on the per-model rate card, which changes over time and differs by input versus output tokens (output is almost always priced higher). Store raw token counts in your logs and compute cost as a derived value at query time or in a nightly job, that way a price change doesn't require backfilling historical data, you just update the rate table and rerun the calculation.

Do these tools work with self-hosted or open source models? Langfuse and OpenLLMetry both work fine with self-hosted models since you control the instrumentation and can log whatever usage numbers your inference server reports. Helicone and Portkey support custom endpoints too, though you'll need to point the base URL at your own inference server and may need to supply your own pricing table since there's no public rate card for a model you're hosting yourself.