Back to LLM / GenAI & Prompt/Context Engineering
About this interview
A technical interview on LLM Evaluation, pitched at the medium level. A voice AI interviewer leads the conversation, adapts its questions to your answers, keeps you on topic, and afterward gives you honest, specific feedback on where you were strong and where to improve. Expect roughly 30 minutes.
What you'll be assessed on
Explain the three evaluation paradigms: automated benchmarks, human evaluation, and model-as-judge
Describe Goodhart's Law as it applies to LLM evaluation and the risk of benchmark contamination
Explain RAG-specific metrics: faithfulness, answer relevance, context precision, and context recall
Describe the strengths and failure modes of LLM-as-judge evaluation
Explain why classic NLP metrics (BLEU, ROUGE) are insufficient for evaluating open-ended generation
Topics covered
Evaluation FundamentalsEvaluation ParadigmsClassic MetricsBenchmark ReliabilityAutomated BenchmarksHuman EvaluationLLM-as-JudgeRAG EvaluationEvaluation DesignLLM-as-Judge Failure ModesAdvanced EvaluationSafety Evaluation
A few sample questions
Just examples to set expectations - the real interview has many more and adapts to your responses.
“Why is evaluating large language models considered fundamentally harder than evaluating classical ML models like a classifier?
“Walk me through the difference between context precision and context recall in RAG evaluation. If your context recall is high but precision is low, what does that tell you about your retriever?
“Explain the distinction between intrinsic and extrinsic hallucination in the context of RAG evaluation. Which is harder to catch with automated metrics and why?