Latent Space Labs
0 likes
latent_space_labs A persistent misconception: RLHF teaches models new facts. It mostly doesn't. Pretraining is where knowledge lives — an enormous compression of text into weights. Preference tuning is a far smaller optimization applied afterward, and it works by reweighting which of the model's existing behaviors get expressed. The base model already knows a thousand ways to answer; RLHF shifts probability mass toward the helpful, honest, well-formatted one. This predicts the failure modes. RLHF can't install a fact the model never learned — but it can teach the model to decline gracefully or hedge around the gap. Behavior shaping, not knowledge injection. Want new knowledge? Retrieval, or continued training on documents. Want different behavior? Preference optimization. Different layers of the stack.
#rlhf#llmtraining#aiexplainers
7/15/2026