Every Evaluation Framework Is Hackable
Insight
Every system and every evaluation framework is easy to game, because you can always do something that makes the metric look good without the underlying result being real. The only real defense is dependable people who instinctively ask whether a good-looking number actually reflects a good outcome, which is why responsibility, not the metric, is the true safeguard.
Source Context
"每一个系统、每一个评价的框架，都是很容易被hack的，因为你总可以做一些事，让你的指标看起来很好看。一个值得信赖或者踏实的人，他其实是会想，自己做的这件事如果效果好的话，是不是真的。" "人要变成一个更可信可靠的系统的组件。" — Whisper transcript [03:16:36] ("每一个评价的框架，都是很容易被hack的……让你的指标看起来很好看，但是一个值得信赖、踏实的人……"), chapter [03:14:06 组织搭建]. Quotes verified against 虎嗅 digest #4, #5. Gloss: Any metric can be hacked to look good; dependable people who ask "is it really good?" are the safeguard.