Scenario
Your eval harness uses an LLM as a judge to check whether a generated answer is actually supported by the retrieved context ("faithfulness" — a standard RAG eval metric). call_judge_llm stands in for the real judge call: it returns realistic natural-language verdicts, not a clean enum, which is exactly how a real judge model behaves.
verdict = call_judge_llm(answer, context)
return verdict.startswith("YES")
That one line is the actual bug pattern this challenge is based on: it works when the judge happens to reply with the exact string "YES", and silently returns False — "not faithful" — for every other way an LLM might phrase "yes." Run the code as-is and look at the three print statements: all three answers are genuinely faithful to their context, but at least one of them comes back False.
call_judge_llm(answer, context)is the mocked judge. It's already correct — it's deliberately returning varied, realistic affirmative phrasings, not a bug for you to "fix" by changing it.faithful(answer, context)is where the actual parsing bug lives.
Your task: fix faithful() so it correctly recognizes any affirmative verdict the judge might phrase, not just the one exact string it happens to test against today — without changing call_judge_llm or the three calls at the bottom.