advanced-evaluation

This skill provides a robust framework for evaluating Large Language Model (LLM) outputs using LLM-as-a-judge techniques. It is designed for engineers, data scientists, and AI researchers tasked with building reliable quality assurance pipelines for generative AI agents. The skill focuses on moving beyond manual testing by codifying evaluation metrics into automated systems that minimize subjectivity and noise.

• Implements direct scoring for objective criteria like factual accuracy, instruction following, and toxicity.

• Features pairwise comparison methodologies to resolve subjective tasks like tone, style, and persuasiveness preference.

• Provides advanced bias mitigation strategies to counteract position bias, length bias, self-enhancement bias, verbosity bias, and authority bias.

• Generates structured rubrics to reduce evaluation variance and improve inter-rater consistency between automated and human judges.

• Supports systematic evaluation of prompt engineering experiments, model fine-tuning, and A/B testing frameworks.

• Inputs typically include original prompts, model-generated responses, and predefined evaluation criteria or rubrics.

• Outputs consist of structured JSON data containing normalized scores, detailed evidence-based justifications, and a final verdict with confidence intervals.

• Practice of requiring a chain-of-thought justification before scoring improves reliability by 15-25% compared to naive scoring prompts.

• Always use the position-swap strategy for pairwise comparisons to negate ordering effects, returning a 'TIE' if consistency checks fail.

• For calibration, match the scale granularity to the rubric specificity, using 1-5 scales for general tasks and reserved higher precision scales for strictly defined rubrics.

• Regularly monitor for drift and ensure evaluation models remain independent from the models being tested to avoid self-enhancement patterns.