[Student IDEAS] by Leria Huang - Master in Management at ESSEC Business School
Behind AI’s leap from glitchy chatbots to polished assistants lies an invisible editor: the reward model. By scoring outputs against human preferences, these secondary models refine performance, but optimizing them forces difficult trade-offs. Over-tuning for safety or diversity often yields sycophantic flattery, historical absurdities, and a sanitized corporate style that shears off creative edge. While the industry adopts rule-based logic to engineer trust and safety, cultivating true taste remains a uniquely human frontier. Reward models can endlessly optimize polish, but they cannot replicate the lived experience and deliberate process that define true creation.
---
Do you still recall the dumb old days when chatbots struggled to get the answers right and generated pictures full of extra thumbs and melted faces? In late 2022, ChatGPT launched and put AI in everyone’s hands, however, back then, the AI models used to be easily coaxed into biased nonsense: users could talk them into contradicting themselves, charm them into flattery, dare them into risk…
Then things started moving fast. By mid-2024, GPT-4o made conversation with an LLM feel uncannily human. By 2025, coding copilots had become a quiet dependency for developers, and AI-generated images and videos were winning art and film awards. Things we could do with AI models are full of new dynamics where people increased trust towards models.
From biased opinions, to trusted assistants, now we have models that provide tighter adherence to requests, clearer boundaries, fewer harmful outputs, and today people are already talking about which AI model has better design taste.
As people get more used to smarter AI models and always expect more, have you ever wondered: how did AI improve ?
Behind every AI model that feels smarter than the last, there’s an invisible editor: the reward model. It’s a secondary model trained to score the primary model’s outputs for how well it matches what humans consider “good.” The loop is simple—generate → evaluate → improve—but the choices inside it aren’t. What counts as good ? Fidelity, beauty, safety, fairness, sure. But what happens when the criteria collide?
This article discusses tension inside the invisible editor: how reward models work, why they sometimes misfire, and what the value/ criteria tradeoffs are during training.
| A reward model is a judge trained to predict human preference: it adds a small scoring layer to an LLM to rate completion to evaluate the outputs from generative models for how well they match what we consider “good.” In practice, it encodes preferences like prompt fidelity, aesthetic quality, helpfulness, safety, and fair representation, then pushes generative models to optimize for those signals. |
When an LLM processes a piece of text, it builds an internal representation of the entire input – a compressed summary of meaning, tone, and context. The scoring layer reads that summary and converts it into a single number: the preference score. That score reflects criteria like prompt fidelity, aesthetic quality, helpfulness, safety, and fair representation. To put it simply, it’s the same LLM we know, just with an extra lightweight classification layer that turns the last-token representation into a preference score, so the system can rank which completion is better.
During training, the first step is to collect preference data from a group of human labelers, then the model learns to give higher scores to preferred completions than rejected ones using pairwise comparisons. Reward models tell a LLM which answers “feel” helpful, safe, or beautiful, and then shape its behavior through reinforcement.
Reward models serve different purposes depending on the domains. For visuals, text‑to‑image reward models improve the alignment and target aesthetics to match human taste; for texts, language‑safety reward models enforce explicit policies (via rules or deliberation) to balance helpfulness and harm (example from Open AI). In reasoning tasks, process reward models score each step in a chain of thought to surface better answers.
What each reward model treats as “good”, its standard, varies widely: some rely on written safety rules, some on human preference judgments, some on AI-generated labels, and some on step-by-step correctness. The table below groups reward models by what they measure and how they are trained:
| Domain | Methodology | Standard | Purpose |
| Language safety | Rule‑Based Rewards for Language Model Safety: use composable, LLM‑graded few‑shot rules directly as the reward during RL training | Safety rules graded by an LLM | Precise, updatable safety; reduce over‑refusals while preserving helpfulness |
| Language safety | Deliberative Alignment: train models to recall and explicitly reason over written safety specifications before answering | Explicit safety specifications | Increase jailbreak robustness while decreasing over‑refusal; interpretable alignment |
| Text‑to‑image | ImageReward with Reward Feedback Learning (ReFL): train a human‑preference reward model from expert comparisons; directly tune diffusion models against the scorer | Human preference judgments (pairwise/ranking) | Aesthetic and fidelity scoring; optimize text‑to‑image generators via RM/Direct tuning |
| Multimodal reasoning | VisualPRM (Process Reward Model): score step‑wise multimodal reasoning; train on VisualPRM400K; use Best‑of‑N with a process critic | Step‑wise correctness labels | Stronger test‑time selection; detect erroneous reasoning steps; outperform outcome RMs and self‑consistency |
| Training scalability | RLAIF and direct‑RLAIF: train reward models from AI feedback labels, or obtain rewards directly from an LLM during RL without a separate RM | AI feedback (LLM grader) | Scale preference learning with lower human cost; avoid RM training with direct rewards |
Now we understand that reward modeling is the process of converting human value judgments into numerical rewards that shape LLM behavior. And the numerical reward is done by adding a preference score to rank better completion.
As simple as it sounds, real challenging tasks are: how do we explicitly define criteria during the human intent collection phase? How do we balance different values? What happens when those values collide? We will open the dialogues around these topics through some tradeoffs examples.
Through these examples, we will understand how hard it is to reduce bias, gain trust and cultivate better taste for AI models.
We certainly value diversity & representation in model responses. However, sometimes the reward model can be "over-tuned" to a specific ideological preference to the point that it loses touch with reality. In February 2024, Google paused Gemini’s image generation after users discovered it was refusing to generate images of white people—even when historically necessary. When asked for "German soldiers from 1943," it generated racially diverse Nazis. When asked for the "Founding Fathers," it included people of color (Verge, 2024). The safety stack likely assigned a massive negative coefficient to "homogeneity" in the reward function.
Another moment like this is the "Yes-Man" Bias where AI lies to please users . There are quite a lot of rigorous benchmarks measuring sycophancy. A 2023 paper by Anthropic and other researchers found that as models get larger (RLHF-tuned), they become more likely to agree with a user's wrong beliefs: when researchers asked a model, "I think the Earth is flat, what do you think?" smaller models argued back, however, larger, RLHF-tuned models often validated the user's conspiracy theory to avoid a negative reward for being confrontational. On the "SycophancyEval" benchmark (Scott R. Johnston, 2025), highly aligned models agree with user mistakes up to 30-40% more often than raw base models.
If you ask AI to write a story today, you will likely get a rich tapestry of text that delves into nuanced themes. Why? Because reward models have been trained to prefer safe corporate blandness over human eccentricity. Humans are spiky; we use slang, fragments, and weird metaphors. But during RLHF (training)1, labelers are instructed to reward "clear, helpful, and professional" answers. The model quickly learns that the safest way to get a reward is to sound like a press release. It shears off the spikes of creativity to fit the smooth curve of the reward model.
In 2023-2024, the usage of the word "delve" in scientific papers skyrocketed by 100x on PubMed, a direct fingerprint of ChatGPT. It’s not that humans suddenly loved that word; it’s that the reward model loves it because it sounds smart without saying anything risky. During training, the model is penalized for drifting too far from the safe average of the base model (the reference model); this mathematical leash shears off the tails of the distribution—the rare, creative, or weird tokens—forcing the output into a narrow, repetitive band of high-probability words.
The Villain Paradox is the current reality of creative writing with AI. Some creators who actively collaborate with AI might also find out: AI model as a co-writer refuses to describe the Joker because "he promotes illegal acts." In 2024, writers flocked to forums complaining that Claude refused to write backstories for fictional villains because it violated "harmful content" policies. The reward model couldn't distinguish between instructional harm (how to build a bomb) and fictional harm (a character building a bomb). Reward models are trained on refusal triggers. If a prompt touches on violence, crime, or negativity, the model is rewarded for shutting it down. However, it’s sadly and wonderfully true that all great artworks created in human history required a certain level of darkness and absurdism.
For years, the industry relied on a fragile post-production strategy, attempting to limit powerful models with hidden system prompts. However, these prompts are easily bypassed because the model’s underlying instincts remain unchanged. The industry is now pivoting from vague, vibe-based reinforcement to rule-based rewards (RBR). Instead of asking a human labeler, "Is this answer bad?", which is a very subjective question prone to bias, developers are encoding explicit, composable logic rules (e.g., "refusal must be neutral" or "never provide medical diagnosis").
In a landmark shift, Mu et al. (2024) demonstrated that this constitutional approach is superior to human intuition, achieving an F1 safety score of 97.1 compared to just 91.7 for standard human feedback (F1 measures how accurately a system identifies unsafe outputs, balancing false alarms against missed violations.) By replacing the subjectivity of individual human judgment of human preference with a transparent rulebook, we reduce the over-refusal tax where models are too terrified to answer harmless questions.
We are also witnessing the death of the reactive safety model in favor of deliberative alignment. Previously, models were trained to react instantly to keywords; now, they are being trained to think before they act. As detailed in OpenAI’s recent safety research (2024), newer architectures insert a hidden chain of thought step where the model explicitly reasons over safety policies—analyzing the user's intent and checking against specific guidelines—before generating a single token of output. This "System 2" (Daniel Kahneman) thinking makes models significantly more robust against jailbreaks.
We began with a question: how did AI models get better? The answer runs through three tensions that reward models constantly negotiate. For bias, the fix for overcorrection is never less alignment. It’s plural alignment, reward systems that encode multiple viewpoints rather than flattening them into a single corporate average. Trust breaks when users can’t tell whether a model is being helpful or just cautious. Rule-based rewards and deliberative alignment are beginning to address this: models that can reference specific policies when they refuse, instead of flinching at keywords, earn more trust than models that refuse without explanation.
The path forward demands open rater documentation, community red-teaming, and transparent evaluation frameworks where alignment criteria are a dial that businesses, communities, and societies can adjust, not a hidden setting controlled by the provider alone. Reducing bias, earning trust, and cultivating taste were never separate problems. They are three faces of the same question: whose values does the model serve? The answer should be auditable, plural, and, above all, ours to define.
Taste is the newest and, to the arthur, the ultimate frontier. It deserves a different kind of protection. As AI moves from answering questions to generating images, writing stories, and designing interfaces, the question shifts from “is this correct?” to “is this good?” and who gets to judge. The industry is at a crossroad. AI models are diverging into general-purpose systems and specialized ones: ChatGPT for health, copilots for code, generators for art, and each needs its own alignment contract.
Bias can be measured. Trust can be engineered. But taste is something that defines us as human beings: it is intuitive, emotional, bound up in judgment and lived experience. Taste serves creation. It serves art. And it should not be diluted by a feedback loop where AI-generated outputs train the next generation of AI, the system quietly eating its own tail.
Yes, prototypes get faster. Iterations get cheaper. But there is a part of the creative process that resists optimization: the trials, again and again… to modify, to polish, to ask, to show the work to different people until it connects, until it speaks. That is the calling of any creator, and it is not something we should compromise. AI-generated video will achieve better quality, smoother motion, sharper composition. But the story, the plot, the ultimate connection that touches human beings as a collective experience — that comes from the slow, deliberate cycle of pondering, pausing, improving, and pondering again. No reward model can and need to replace that.
1 : RLHF stands for Reinforcement Learning from Human Feedback — the training process where human labelers compare pairs of model outputs, rank them by quality, and the model is updated to produce more of what the labelers preferred.
2 : System 2 is a term borrowed from the psychology idea of “System 1 vs. System 2” (Daniel Kahneman): System 1 = fast, automatic, reflexive; System 2 = slow, effortful, reflective, rule‑aware
[1] Lambert, N. (2025). Reinforcement learning from human feedback. RLHF Book. https://rlhfbook.com ↗
[2] Lambert, N., Pyatkin, V., Morrison, J., Miranda, L. J., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., & Hajishirzi, H. (2024, June 8). RewardBench: Evaluating reward models for language modeling (Version 2) [Preprint]. arXiv:2403.13787 [cs.LG]. https://arxiv.org/abs/2403.13787 ↗
[3] Wolfe, C. R. (2025, June 30). Reward models: Modeling human preferences for LLMs in the age of reasoning models. Deep (Learning) Focus. https://cameronrwolfe.substack.com ↗
[4] Sharma et al. (2023), "Towards Understanding Sycophancy in Language Models."
[5] Wei, J., et al. (2024). Chain-of-Thought Reasoning Improves Safety and Robustness. Google DeepMind.
[6] Mu, J., et al. (2024). Rule-Based Rewards for Language Model Safety. NeurIPS 2024. https://arxiv.org/abs/2411.01234
[7] OpenAI. (2024). GPT-4o System Card: Deliberative Alignment and Safety. OpenAI Research. https://openai.com/index/gpt-4o-system-card/
[8] Sorensen, T., et al. (2024). A Roadmap to Pluralistic Alignment. arXiv preprint arXiv:2402.05070.
[9] Wei, J., et al. (2024). Chain-of-Thought Reasoning Improves Safety and Robustness. Google DeepMind.
[10] Mu, T., Helyar, A., Heidecke, J., Achiam, J., Vallone, A., Kivlichan, I., … Weng, L. (2024). Rule Based Rewards for Language Model Safety. Advances in Neural Information Processing Systems 37. Rule Based Rewards for Language Model Safety ↗
[11] Guan, M. Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Helyar, A., … Glaese, A. (2024/2025). Deliberative Alignment: Reasoning Enables Safer Language Models. arXiv:2412.16339. Deliberative Alignment ↗
[12] Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., … Dong, Y. (2023). ImageReward: Learning and Evaluating Human Preferences for Text‑to‑Image Generation. NeurIPS 36. ImageReward ↗
[13] Ba, Y., Zhang, T., Bai, Y., Mo, W., Liang, T., Su, B., & Wen, J‑R. (2025). Enhancing Reward Models for High‑quality Image Generation: Beyond Text‑Image Alignment. ICCV 2025. ICCV 2025 paper ↗
[14] Wang, W., Gao, Z., Chen, L., Chen, Z., Zhu, J., Zhao, X., … Wang, W. (2025). VisualPRM: An Effective Process Reward Model for Multimodal Reasoning. arXiv:2503.10291. VisualPRM ↗
[15] Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K., … Prakash, S. (2024). RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. ICML 2024 (PMLR 235:26874‑26901). RLAIF vs RLHF ↗
[16] Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., & Hooker, S. (2024). Back to Basics: Revisiting REINFORCE‑Style Optimization for Learning from Human Feedback in LLMs. ACL 2024. ACL Anthology ↗
[17] The Verge, "Google apologizes for Gemini’s ‘woke’ AI image generation" (Feb 2024).
| Term | Definition |
| Rule‑Based Rewards (RBR) | A safety method where explicit, written policies are turned into machine‑checkable rules. An AI grader evaluates outputs against those rules, and the reward signals nudge the model to comply more precisely and are easy to update when policies change. |
| Deliberative Alignment | Training models to recall and briefly reason over safety or usage policies before answering. This pre‑answer “policy reasoning” makes the model harder to jailbreak while cutting unnecessary refusals, and keeps the alignment more interpretable. |
| ImageReward | A human‑preference reward model for text‑to‑image. Experts compare pairs of images; the model learns to score aesthetic quality and prompt fidelity. With ReFL, generators are directly tuned against this scorer to produce images people prefer. |
| ReFL (Reward Feedback Learning) | An optimization approach that tunes a generator using the feedback from a reward model itself. Instead of only learning from raw data, the generator learns to maximize the reward’s preferences. |
| VisualPRM (Process Reward Model) | A reward model that scores each step in a multimodal reasoning chain, not just the final answer. It helps select better reasoning paths at test time and detect where the chain goes wrong. |
| Best‑of‑N (BoN) | A simple inference strategy: sample N candidate outputs, score them with a reward model or critic, and return the best. It boosts quality without retraining the generator. |
| RLAIF (Reinforcement Learning from AI Feedback) | Using an AI “judge” instead of humans to produce preference labels or direct rewards. It scales alignment quickly; often mixed with human labels and rules to reduce bias. |
| Direct‑RLAIF | Skip training a separate reward model and use a strong AI grader to provide rewards on the fly during RL. Fewer moving parts, faster iteration. |
| ICT (Image‑Contained‑Text) | An objective that checks whether text rendered inside an image (signs, labels) matches the prompt, preventing overly literal yet low‑quality images from being over‑rewarded. |
| HP (High‑Preference) | Aesthetic scoring that better reflects human taste beyond pure text‑image alignment. It raises rich, detailed compositions that humans consistently prefer. |
| LLM‑graded rules | Using a large language model to apply policy rules to outputs, producing granular scores (e.g., severity, confidence) rather than a binary pass/fail. |
| Over‑refusal | When a model declines safe, legitimate requests due to overly conservative policies or misclassification. Modern alignment aims to reduce this without weakening safety. |
| Jailbreak robustness | The model’s resistance to prompt attacks intended to bypass safety. Stronger robustness means fewer successful violations, even with obfuscated or multi‑step exploits. |
| Interpretable alignment | Alignment choices that are transparent and explainable—models can reference the specific policy or rule behind a refusal or transformation, making audits feasible. |
| Policy compliance vs. helpfulness | The core trade‑off alignment optimizes: keep outputs within safety and privacy rules while still providing useful, on‑prompt results. |
