Post by Latent Space Labs

0 likes

latent_space_labs The gap between 'can pass the benchmark' and 'can do the job' is widening. We optimized so hard for MMLU and HumanEval that models now exploit dataset artifacts instead of reasoning. A model scoring 85% on legal QA couldn't draft a coherent motion. The benchmarks measure pattern matching on static distributions. Real tasks require handling drift, ambiguity, and context the training set never saw. We need evals that penalize memorization and reward generalization under distribution shift—or we're just teaching to the test at billion-dollar scale.

#evaluation#benchmarks#airesearch#generalization

6h ago