Ragas Open Source Roadmap: What's Coming Next

Why Ragas's Roadmap Is Suddenly Worth Watching

If you evaluated a RAG pipeline any time in the last two years, there's a good chance you imported ragas.metrics and called evaluate() without thinking twice about it. That function has been the default entry point into LLM evaluation for thousands of teams, and it worked well enough that most people never asked what was happening underneath. That's changing. Ragas just shipped its biggest architectural rewrite since v0.2, the team behind it has rebranded from Exploding Gradients to Vibrant Labs, and several core modules — including testset generation, one of the library's signature features — are sitting in open community debate about whether they even belong in the main package anymore.

None of this is speculative. It's visible in the migration guide, the release history, and the GitHub issues where maintainers are asking users, in public, what to keep and what to cut. That's unusual for an open-source project of this size, and it's worth walking through carefully if you use Ragas or are about to start. This article covers what actually changed in the v0.3-to-v0.4 transition, what's explicitly marked "coming soon" in the official templates, what's still unresolved, and what all of it means if you're maintaining an evaluation suite today.

The Rebrand: Exploding Gradients Becomes Vibrant Labs

Start with the organizational shift, because it explains some of the roadmap uncertainty. The company behind Ragas was known as Exploding Gradients. It now operates as Vibrant Labs, describing itself as an applied research lab working on data collection techniques for AI agents, with Exploding Gradients Inc. now sitting behind it as the backing entity. The GitHub org itself moved from explodinggradients to vibrantlabsai, though old explodinggradients/ragas URLs still resolve — GitHub's org-rename redirects keep old links, issues, and PR references working, which is why you'll still see both names in search results and older blog posts.

This kind of rename is common when a small open-source team turns into a company with a broader research agenda than the single library that made them known. It doesn't automatically mean Ragas is being deprioritized — the opposite argument is just as plausible, since a funded lab has more engineering hours to put into the project than a side effort. But it does mean the roadmap for Ragas now competes for attention with whatever else Vibrant Labs is building. If you're planning to depend on Ragas for a production evaluation pipeline, that context matters when you're deciding how much custom tooling to build versus how much to lean on the library's built-ins.

The practical takeaway: watch the GitHub org, not just the marketing site, for signal. Marketing pages update slowly and optimistically. Release notes and issue threads tell you what's actually shipping.

The v0.4 Rewrite: From Metric Calls to Experiments

The single biggest thing that happened to Ragas recently is the v0.4 release, and the official migration guide is unusually candid about what changed and why. The framing the maintainers use is a shift from a "metric-centric" library to an "experiment-based architecture." In practice, that means Ragas stopped thinking of itself as a bag of scoring functions you call one at a time, and started thinking of itself as a framework for running structured evaluation experiments with integrated analysis.

Three changes make up the bulk of this:

• Metrics moved to a collections API. Instead of importing from ragas.metrics, you now import from ragas.metrics.collections. The scoring interface changed too — single_turn_ascore(sample) became ascore(**kwargs), and instead of returning a bare float, metrics now return a MetricResult object carrying both the score and, where relevant, the model's reasoning for that score.
• LLM configuration was unified. The old split between instructor_llm_factory() and llm_factory() collapsed into a single llm_factory() call that auto-detects the provider from the model name. One function now covers OpenAI, Anthropic, Google, Azure, Cohere, and Bedrock, with built-in handling for the parameter quirks of GPT-5 and o-series reasoning models.
• Prompts became first-class, typed objects. Instead of dataclass-based prompt definitions, each metric now ships dedicated *Prompt, *Input, and *Output classes, accessible via metric.prompt. Customizing a metric's prompt no longer means monkey-patching a template string — you subclass a typed prompt class and get static type checking on the input and output shape.

Here's roughly what changed in code terms:

# v0.3 style
from ragas.metrics import faithfulness
from ragas import evaluate

score = await faithfulness.single_turn_ascore(sample)

# v0.4 style
from ragas.metrics.collections import Faithfulness
from ragas.llms import llm_factory

llm = llm_factory("gpt-4o", client=client)
metric = Faithfulness(llm=llm)
result = await metric.ascore(user_input=question, response=answer, retrieved_contexts=contexts)
score = result.value
reasoning = result.reason

If you've been running Ragas in CI or inside an eval harness, this is not a drop-in swap. The evaluate() function itself is deprecated in favor of an @experiment() decorator, which tells you the direction the maintainers want the whole library to move: less "call a function, get a number back," more "define an experiment, get structured, comparable results across runs." That's a meaningfully different mental model, and it's worth budgeting real migration time for it rather than assuming a search-and-replace on import paths will cover it.

What Got Removed, and Why It Matters More Than It Looks

Rewrites always come with a casualty list, and Ragas's is worth reading closely because a few of the removals touch commonly used code paths.

• AspectCritic, SimpleCriteria, and AnswerSimilarity are gone, replaced by what the maintainers describe as a "more flexible discrete metric pattern." If you had custom aspect-critic prompts for domain-specific checks (tone, policy compliance, brand voice), those need to be rebuilt against the new pattern rather than just re-imported.
• The LangChain and LlamaIndex wrapper classes for embeddings (LangchainEmbeddingsWrapper, LlamaIndexEmbeddingsWrapper) were dropped in favor of native embedding provider classes like OpenAIEmbeddings, GoogleEmbeddings, and HuggingFaceEmbeddings, with method names changing from embed_query()/embed_documents() to embed_text()/embed_texts(). If your pipeline evaluates embeddings through a LangChain wrapper today, that call site breaks on upgrade.
• The data schema itself changed: ground_truths: list[str] became reference: str. That's not a rename, it's a shape change, from a list to a single string. Anything that built SingleTurnSample objects programmatically needs an actual data transformation step, not just a field rename.

The LLM and prompt wrapper classes (LangchainLLMWrapper, LlamaIndexLLMWrapper, PydanticPrompt) are marked deprecated rather than removed, so they still work for now — but "deprecated and still functional" in a fast-moving library usually has a shelf life measured in a couple of minor versions, not years. If you're starting a new Ragas integration today, there's little reason to reach for the deprecated path just because more blog posts and Stack Overflow answers reference it.

Metrics Coverage: What Actually Made the Cut

It's worth being concrete about scope, because "metric-centric to experiment-centric" sounds abstract until you see how much surface area it touched. The v0.4 collections migration covers more than twenty metrics across four groups:

• Core RAG metrics: Faithfulness, AnswerRelevancy, AnswerCorrectness, ContextPrecision, ContextRecall, ContextEntityRecall, ContextRelevance, NoiseSensitivity, ResponseGroundedness, SemanticSimilarity, FactualCorrectness, BleuScore, RougeScore, SummaryScore.
• Agent and tool-use metrics: ToolCallAccuracy, ToolCallF1, TopicAdherence, AgentGoalAccuracy.
• Structured data metrics: DataCompyScore, SQLSemanticEquivalence.
• Rubric-based and string metrics: DomainSpecificRubrics, InstanceSpecificRubrics, CHRFScore, QuotedSpansAlignment, ExactMatch, StringPresence, LevenshteinDistance, MatchingSubstrings, NonLLMStringSimilarity.

That breadth tells you this wasn't a cosmetic API change bolted onto the old internals — the maintainers actually rewrote the metric implementations, not just the import paths. It also explains why some documentation pages lag behind: when you touch that many metrics in one release cycle, docs drift is close to inevitable, and that shows up directly in an open issue (#2626) where a user points out that the docs describe Context Precision variants as deprecated while the metrics mind map in the same doc set still lists non-LLM context precision as part of the framework. As of this writing, that issue has no maintainer reply. If you're relying on Context Precision variants specifically, don't assume the mind-map diagram is more current than the deprecation note — check the changelog and the metric's own docstring before trusting either page.

The Rough Edges Showing Up in the Issue Tracker

A roadmap isn't just the features a team announces — it's also visible in what's currently breaking, because bug reports tell you where the maintainers' attention is actually being pulled. A few open issues on the repo are worth knowing about if you're running Ragas in production right now, not because they're catastrophic, but because they show the cost of moving fast on provider integrations.

One open issue reports that Ragas 0.4.3 has a broken ChatVertexAI import because it still points at a langchain_community path that has since been removed upstream. That's a familiar failure mode for any library that wraps LangChain integrations: LangChain's own internals reorganize frequently, and a pinned or stale import path in a downstream library turns into a hard crash rather than a deprecation warning. If you're evaluating models through Vertex AI via LangChain, check this before you upgrade, not after.

A second thread asks about running Ragas evaluations against a local, self-hosted model — specifically openai/gpt-oss-20b — instead of calling the OpenAI API directly. This is a recurring theme in the issue tracker generally: teams that can't or won't send evaluation data to a hosted API (for cost, latency, or data-residency reasons) want first-class support for local inference backends. The new unified llm_factory() helps here in principle, since provider auto-detection is exactly the kind of abstraction that makes swapping in a local, OpenAI-compatible endpoint easier than it used to be with the old instructor_llm_factory() split. But "helps in principle" and "is documented and tested" are different claims, and as of this writing that gap is closer to community-workaround territory than to an officially supported path.

Neither of these is a five-alarm fire. Both are exactly the kind of friction you'd expect from a library that just rewrote its LLM and prompt layers while also chasing fast-moving upstream SDKs (LangChain, Google's google-genai, provider-specific parameter handling for newer reasoning models). The signal worth taking from them is narrower and more useful than "Ragas has bugs": it's that provider integrations are the part of the roadmap most exposed to breakage between releases, so if your evaluation pipeline pins a specific provider integration, pin your Ragas version too, and don't let it float on pip install -U in CI without a changelog check first.

Testset Generation: An Open Question, Not a Settled Answer

The most interesting item on the roadmap isn't a shipped feature — it's a question the maintainers asked the community and haven't fully answered yet. In GitHub issue #2231, titled "Feedback Request: Future of Testset Generation Module in Ragas v0.4," the team laid out three explicit options for one of Ragas's original headline features, the synthetic testset generator that builds evaluation datasets from your documents:

1. Keep it as a core, integrated part of the main Ragas package.
2. Extract it into a standalone package outside the primary repo.
3. Phase it out entirely if it isn't pulling its weight for users.

They asked for feedback via reactions (a straightforward up/down/split vote) and via comments describing actual use cases, friction points, and improvement ideas. The issue explicitly notes that community pull requests for this module were ready to merge, but the team wanted to confirm the roadmap actually matched user needs before investing further engineering time in it.

This is a genuinely useful thing to see in the open, and also a genuine risk signal if testset generation is load-bearing in your evaluation setup. A library maintainer publicly asking "should we keep this" is different from a silent deprecation notice — it means the outcome isn't decided yet, which cuts both ways. If you're using synthetic testset generation as part of a CI gate or a recurring eval run, this is the single most concrete thing to track before your next dependency bump: if it gets spun out into a separate package, your import paths change; if it gets phased out, you need a fallback plan for building test datasets, whether that's hand-curated golden sets or a different open-source generator.

What "Coming Soon" Actually Means Right Now

Separate from the debated features, Ragas's own CLI quickstart already advertises a set of templates that aren't fully built yet. Running ragas quickstart today gives you a working template for rag_eval — full RAG evaluation, ready to use. Alongside it, the CLI lists four templates explicitly marked "coming soon":

• agent_evals — evaluating AI agents specifically, as opposed to single-turn RAG responses.
• benchmark_llm — benchmarking and comparing different LLM models against each other on the same task set.
• prompt_evals — evaluating prompt variations systematically, which is effectively prompt-engineering-as-experiment.
• workflow_eval — evaluating multi-step, complex workflows rather than a single request/response pair.

There is a benchmark_llm walkthrough already published in the docs, built around comparing models on a discount-calculation task, so that particular template is further along than "coming soon" might suggest — treat it as early-access rather than vaporware. The other three are genuinely earlier stage. The pattern worth noticing here is the direction: every one of these templates pushes Ragas further from "score a single RAG response" and toward "evaluate an agentic system doing multi-step work." That tracks with where the rest of the LLM tooling ecosystem is headed — agent frameworks, tool-calling, and multi-turn workflows have outpaced single-shot RAG as the thing teams actually need to evaluate in production, and Ragas's own metrics list (ToolCallAccuracy, AgentGoalAccuracy, TopicAdherence) already reflects that shift before the templates catch up.

Reading the Roadmap Signals Correctly

Put together, here's what the available evidence actually supports, without overreaching into things nobody has confirmed:

• Confirmed and shipped: the v0.4 architectural rewrite (collections API, unified LLM factory, typed prompts, experiment decorator), the removal of several legacy metrics and wrapper classes, and the ground_truths to reference schema change.
• Confirmed and in progress: agent, benchmark, prompt, and workflow evaluation templates, in that order of maturity based on what's already documented versus what's listed as upcoming.
• Explicitly unresolved: the future of testset generation as a module — core, spun out, or removed — pending community input the maintainers said they wanted before committing.
• Not yet clarified in docs: the exact scope of the Context Precision deprecation, where an open issue shows a live inconsistency between the changelog and the metrics reference material.

What's notably absent from all of this is a pinned, dated v1.0 roadmap document. Community discussion has referenced the team saying they'd share more about v1.0 direction after the v0.4 release settled, but as of this writing there isn't a public roadmap doc laying out version-by-version commitments. That's worth naming plainly rather than filling in with guesswork — a fast-moving open-source project without a fixed roadmap is normal, not alarming, but it does mean you should treat any date-specific claim about "Ragas v1.0 will ship X by Y" that you read elsewhere with real skepticism unless it links back to an actual maintainer statement.

What This Means for Your Evaluation Stack Today

If you're currently on an older Ragas version, the practical move isn't to panic-upgrade the moment v0.4 lands in your dependency resolver. It's to read the migration guide against your actual call sites: check whether you call evaluate() directly, whether you construct SingleTurnSample objects with ground_truths, whether you use any of the removed metrics or wrapper classes, and whether you depend on testset generation as part of a repeatable pipeline. Each of those is a specific, checkable thing, not a vague "check for breaking changes" task.

A concrete pre-upgrade checklist, based on everything above:

1. Grep your codebase for from ragas.metrics import and plan the swap to from ragas.metrics.collections import metric by metric, not as a blanket find-and-replace, since return types changed from float to MetricResult.
2. Search for ground_truths in any code that builds SingleTurnSample or EvaluationDataset objects — this needs an actual data transform to reference, not a rename.
3. Check for AspectCritic, SimpleCriteria, or AnswerSimilarity usage; these are removed, not deprecated, so they'll fail on import rather than warn.
4. Check for LangchainEmbeddingsWrapper or LlamaIndexEmbeddingsWrapper usage, and for any ChatVertexAI integration specifically, given the currently open import-path bug on 0.4.3.
5. Pin your Ragas version explicitly in CI rather than floating it, and re-run your full eval suite against the pinned upgrade in a branch before merging, since a chunk of the surface area (LLM factory, prompts, metrics) changed simultaneously.

If you're starting fresh, build against the collections API and the new llm_factory() from day one — there's no reason to learn the deprecated interface when the maintainers have already told you where they're taking it. And if testset generation matters to your workflow, that GitHub issue is worth a comment with your actual use case, not just a reaction emoji, since the maintainers explicitly said they're weighing real usage patterns before deciding its fate.

The bigger lesson generalizes past Ragas specifically: evaluation tooling for LLM systems is still young enough that the libraries underneath it are being rebuilt in public, sometimes with the entire company behind them changing names mid-stream. That's not a reason to avoid the tooling — it's a reason to build your own evaluation logic with enough abstraction that a metrics-library rewrite doesn't take down your CI pipeline, and to actually read migration guides instead of skimming changelogs for version numbers.

If you want a structured, hands-on walkthrough of Ragas — the metrics that matter, how to wire it into a real RAG evaluation pipeline, and how to build test sets and CI gates that survive library upgrades like this one — that's exactly what we cover in the Ragas Tutorial course on teachyou.ai, built to keep pace with exactly the kind of architectural shift this article just walked through.