LLM Evaluation and Guardrails

Learning objectives

• • Design offline and online evaluations for relevance, faithfulness, safety, latency, and cost
• • Explain how to run regression checks for prompts, retrieval, and model routing
• • Connect guardrails to operational risk rather than vague policy language

Interview prompts

• • How would you evaluate a RAG assistant end to end?
• • What signals tell you hallucination is becoming a production problem?

Prerequisites