Human Evaluation Guidelines

Human Evaluation Guidelines【免费下载链接】claude-skills67 Specialized Skills for Full-Stack Developers. Transform Claude Code into your expert pair programmer.项目地址: https://gitcode.com/GitHub_Trending/claud/claude-skillsTaskRate AI-generated responses for customer support quality.Rating ScaleUse a 1-5 scale for each dimension:HelpfulnessDoes not address the customers issue at allPartially addresses the issue but missing key informationAddresses the main issue but could be more helpfulAddresses the issue well with useful informationExceptionally helpful, anticipates follow-up needsAccuracyContains factually incorrect informationMostly accurate but has errorsAccurate but vagueAccurate and specificAccurate with appropriate caveats/nuanceToneInappropriate (rude, dismissive, overly casual)Somewhat inappropriate for contextNeutral/acceptableProfessional and friendlyPerfectly calibrated for the situationInstructionsRead the customer question carefullyRead the AI response completelyRate each dimension independentlyProvide brief justification for scores below 3Flag any responses that should be reviewed by a supervisorExamples[Include 3-5 calibration examples with scores and explanations]可执行的落地要点 - **校准示例calibration examples**模板末尾预留了 3-5 个带分值的示例位置正式评估前先用这批示例让所有评分者对齐尺度这是提升一致性的性价比最高的一步 - **逐维度独立评分**Helpfulness、Accuracy、Tone 分别打分避免印象分串扰 - **低分必写理由**3 分以下要求说明保证低分可追溯、可归类 - **升级路径**要求标记需主管复核的响应为自动评估被质疑时的人工仲裁留出通道。 ### 6.3 评分者间信度Inter-Rater Reliability 人工评估的结论必须先回答评分者之间靠不靠谱。原文档实现了三类一致性度量 python from sklearn.metrics import cohen_kappa_score import numpy as np def calculate_irr(rater_scores: dict) - dict: Calculate inter-rater reliability metrics. raters list(rater_scores.keys()) # Pairwise Cohens Kappa kappas {} for i, r1 in enumerate(raters): for r2 in raters[i1:]: kappa cohen_kappa_score(rater_scores[r1], rater_scores[r2]) kappas[f{r1}_vs_{r2}] kappa # Fleiss Kappa for multiple raters fleiss calculate_fleiss_kappa(rater_scores) # Agreement percentage all_agree sum( 1 for i in range(len(rater_scores[raters[0]])) if len(set(rater_scores[r][i] for r in raters)) 1 ) agreement_pct all_agree / len(rater_scores[raters[0]]) return { pairwise_kappa: kappas, fleiss_kappa: fleiss, perfect_agreement: agreement_pct, interpretation: interpret_kappa(fleiss) } def interpret_kappa(kappa: float) - str: Interpret Kappa score. if kappa 0.20: return Poor agreement elif kappa 0.40: return Fair agreement elif kappa 0.60: return Moderate agreement elif kappa 0.80: return Substantial agreement else: return Almost perfect agreement三个指标各有分工两两 Cohens Kappa度量任意两名评分者的一致性剔除随机一致后的净一致性同时它也是分类指标表中提到的同一公式Fleiss Kappa扩展到多位评分者通常大于 2 人时的总体一致性完美一致率perfect_agreement所有评分者完全同分的样本占比直观但易受极端分布影响。interpret_kappa给出的标准分档0.20 差、0.40-0.60 中等、0.80 几乎完美可以作为团队共识。只有当 IRR 达到中等以上时人工标注才配称为 ground truth也才值得用来校验 LLM-as-Judge。七、回归检测Prompt 版本迭代的守护者对应原文档的 Regression Testing 一节。Prompt 优化最隐蔽的风险是改好了 A 套件偷偷弄坏了 B 套件。回归检测通过对比新结果与基线自动判定是否放行class RegressionDetector: Detect performance regressions between prompt versions. def __init__(self, baseline_results: dict, threshold: float 0.05): self.baseline baseline_results self.threshold threshold def compare(self, new_results: dict) - dict: Compare new results against baseline. regressions [] improvements [] for suite in self.baseline[summary]: baseline_acc self.baseline[summary][suite][accuracy] new_acc new_results[summary][suite][accuracy] delta new_acc - baseline_acc if delta -self.threshold: regressions.append({ suite: suite, baseline: baseline_acc, new: new_acc, delta: delta }) elif delta self.threshold: improvements.append({ suite: suite, baseline: baseline_acc, new: new_acc, delta: delta }) return { has_regressions: len(regressions) 0, regressions: regressions, improvements: improvements, recommendation: self._get_recommendation(regressions, improvements) } def _get_recommendation(self, regressions, improvements) - str: if regressions: return BLOCK: Regressions detected. Review failures before merging. elif improvements: return APPROVE: Performance improved with no regressions. else: return APPROVE: Performance stable within threshold.【免费下载链接】claude-skills67 Specialized Skills for Full-Stack Developers. Transform Claude Code into your expert pair programmer.项目地址: https://gitcode.com/GitHub_Trending/claud/claude-skills创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考