RAG Evaluation with RAGAS

Introduction

• After you build a RAG pipeline (Basic RAG, Fusion Retrieval, Multi-Modal RAG, or GraphRAG), a natural question is: how good is it?
• RAGAS (Retrieval Augmented Generation Assessment) evaluates RAG outputs with an LLM as the judge. You provide questions, predicted answers, ground-truth answers, and retrieved contexts; RAGAS returns scores from 0 to 1.

Core metrics

Models used

• LLM judge: gemma3:4b via Ollama (local)
• Embeddings: mxbai-embed-large:335m via Ollama (local)
• Everything runs locally — no OpenAI API key required.

Step by Step Coding guide

• Pull models: ollama pull gemma3:4b and ollama pull mxbai-embed-large:335m.
• Install helpers such as ragas, datasets, langchain-ollama.

Step 0: Import packages

from ragas import evaluate
from ragas.metrics import faithfulness, answer_correctness, context_precision
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_ollama import ChatOllama, OllamaEmbeddings
from datasets import Dataset

Step 1: Set up the LLM judge and embedding model

• RAGAS uses an LLM to judge answer quality and an embedding model for semantic similarity in some metrics.
• Wrap Ollama models with RAGAS-compatible wrappers.

ollama_llm = LangchainLLMWrapper(ChatOllama(model="gemma3:4b"))
embedding_model = LangchainEmbeddingsWrapper(
    OllamaEmbeddings(model="mxbai-embed-large:335m")
)

print("LLM judge and embedding model ready")

Step 2: Test answer correctness

• Answer Correctness checks whether the predicted answer matches the ground truth.
• It combines:
  • Factual similarity — same facts? (LLM judge)
  • Semantic similarity — similar meaning? (embeddings)
• Example: ground truth is "Madrid is the capital of Spain." and the prediction is only "MadriD." — brief, but still correct.

eval_dataset = Dataset.from_dict({
    "question":     ["What is the capital of Spain?"],
    "answer":       ["MadriD."],
    "ground_truth": ["Madrid is the capital of Spain."],
    "contexts":     [["The capital of Spain is Madrid."]],
})

result = evaluate(
    dataset=eval_dataset,
    metrics=[answer_correctness],
    llm=ollama_llm,
    embeddings=embedding_model,
)

print(result)
print(result.to_pandas())

• Expect a score close to 1.0: "MadriD" is factually right even if it is not a full sentence.

Step 3: Test faithfulness

• Faithfulness asks: is the answer grounded in the retrieved context (no hallucination)?
• RAGAS roughly:
  1. Breaks the answer into factual claims
  2. Checks whether each claim can be inferred from the context

faithfulness ≈ (claims supported by context) / (total claims)

• Here the answer is "6" and the context says "3+3=6" — fully grounded, so expect 1.0.

eval_dataset = Dataset.from_dict({
    "question":     ["what is 3+3?"],
    "answer":       ["6"],
    "ground_truth": ["6"],
    "contexts":     [["3+3=6"]],
})

result = evaluate(
    dataset=eval_dataset,
    metrics=[faithfulness],
    llm=ollama_llm,
    embeddings=embedding_model,
)

print(result)
print(result.to_pandas())

Step 4: Test context precision

• Context Precision measures whether retrieved chunks are relevant and ranked well.
• Relevant chunk first → high precision; irrelevant chunks above it → lower precision.
• Simulated retrieval with 3 chunks (relevant one last):
  1. "this is a test context" — irrelevant
  2. "mike is a cat" — irrelevant
  3. "if the shoes don't fit, then go somewhere else." — relevant
     Expect precision around ~0.33.

eval_dataset = Dataset.from_dict({
    "question":     ["What if these shoes don't fit?"],
    "answer":       ["if the shoes don't fit, then go somewhere else."],
    "ground_truth": ["then go somewhere else."],
    "contexts":     [[
        "this is a test context",
        "mike is a cat",
        "if the shoes don't fit, then go somewhere else.",
    ]],
})

result = evaluate(
    dataset=eval_dataset,
    metrics=[context_precision],
    llm=ollama_llm,
    embeddings=embedding_model,
)

print(result)
print(result.to_pandas())

Step 5: Evaluate multiple questions with all metrics

• In practice, score many questions and several metrics in one call.
• Important: "contexts" must be a list of lists — one list of chunks per question.

eval_dataset = Dataset.from_dict({
    "question":     ["What is the capital of Spain?", "What is 3+3"],
    "answer":       ["Madrid is the capital of Spain.", "6"],
    "ground_truth": ["MadriD.", "6"],
    "contexts":     [
        ["The capital of Spain is Madrid."],  # question 1
        ["3+3=6"],                             # question 2
    ],
})

result = evaluate(
    dataset=eval_dataset,
    metrics=[answer_correctness, faithfulness, context_precision],
    llm=ollama_llm,
    embeddings=embedding_model,
)

print("\nAggregate scores:")
print(result)

Step 6: View per-question results

• Aggregate scores are averages. Inspect the row-level breakdown with pandas.

df = result.to_pandas()
print(df.to_string(index=False))

Summary

• Tip: build a small labeled set (question, ground truth, your RAG’s answer + contexts), run RAGAS after each pipeline change, and track which metric moves — that tells you whether to fix retrieval, prompting, or hallucination.