Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences
Zhuoran Jin, Hongbang Yuan, Kejian Zhu, Jiachun Li, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao
We propose Omni-Reward, a step towards universal omni-modal reward modeling with free-form preferences.
Abstract
Reward models (RMs) play a critical role in aligning AI behaviors with human preferences, yet they face two fundamental challenges: (1) Modality Imbalance, where most RMs are mainly focused on text and image modalities, offering limited support for video, audio, and other modalities; and (2) Preference Rigidity, where training on fixed binary preference pairs fails to capture the complexity and diversity of personalized preferences. To address the above challenges, we propose Omni-Reward, a step toward generalist omni-modal reward modeling with support for free-form preferences, consisting of: (1) Evaluation: We introduce Omni-RewardBench, the first omni-modal RM benchmark with free-form preferences, covering nine tasks across five modalities including text, image, video, audio, and 3D; (2) Data: We construct Omni-RewardData, a multimodal preference dataset comprising 248K general preference pairs and 69K instruction-tuning pairs for training generalist omni-modal RMs; (3) Model: We propose Omni-RewardModel, which includes both discriminative and generative RMs, and achieves strong performance on Omni-RewardBench as well as other widely used reward modeling benchmarks.
Omni-Reward addresses modality imbalance and preference rigidity with omni-modal reward modeling framework.
- Omni-RewardBench: first omni-modal RM benchmark with free-form preferences across 5 modalities
- Omni-RewardData: 248K preference pairs and 69K instruction-tuning pairs for multi-modal training
- Omni-RewardModel: discriminative and generative RMs achieving strong performance across benchmarks
- Framework supporting text, image, video, audio, and 3D modalities with diverse task coverage
- Reward modeling
- Multi-modal learning
- Free-form preference modeling
Benchmark scale may not support evaluations involving millions of examples
from the paperCurrent task definitions remain relatively coarse requiring finer-grained categorization
from the paperPreference data limited to single-turn interactions without multi-turn conversational preferences
from the paperRL technique in training Omni-RewardModel-R1 is preliminary and requires further investigation
from the paperMissing modalities such as thermal, radar, tabular, and time-series data
from the paper
Incorporate video understanding and generation tasks into evaluation system
from the paper
Author keywords
- Omni-Modal Models
- Reward Models
- Alignment
Related orals
LLM Fingerprinting via Semantically Conditioned Watermarks
Introduces semantically conditioned watermarks for robust and stealthy LLM fingerprinting robust to deployment scenarios.
Steering the Herd: A Framework for LLM-based Control of Social Learning
Framework studying strategic control of social learning by algorithmic information mediators with theoretical analysis and LLM-based simulations.
Every Language Model Has a Forgery-Resistant Signature
Ellipse signatures function as forgery-resistant model output identifiers based on high-dimensional geometric constraints.
Gaussian certified unlearning in high dimensions: A hypothesis testing approach
Analyzes machine unlearning in high dimensions showing single noisy Newton step with Gaussian noise suffices for privacy-accuracy.
Differentially Private Domain Discovery
WGM-based methods provide efficient domain discovery with near-optimal guarantees for missing mass on Zipfian data.