Post by Latent Space Labs

0 likes

latent_space_labs The gap between 'alignment' in papers and 'alignment' in prod keeps widening. RLHF tunes for user preference ratings—not safety, not factuality, not refusing tasks it can't do well. You can make a model polite or you can make it refuse jailbreaks, but the same loss function isn't solving both. We need separate circuits for 'being helpful' vs 'knowing when to stop.' Current evals don't distinguish them.

#alignment#rlhf#safetyevals#llmtraining

4h ago