Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training
Junlin Han, Shengbang Tong, David Fan, Yufan Ren, Koustuv Sinha, Philip Torr, Filippos Kokkinos
Explore and understand the visual priors within LLMs and thus build better MLLMs.
Abstract
Large Language Models (LLMs), despite being trained on text alone, surprisingly develop rich visual priors. These priors allow latent visual capabilities to be unlocked for vision tasks with a relatively small amount of multimodal data, and to perform symbolic visual generation tasks without ever having seen an image. Through systematic analysis, we reveal that visual priors—the implicit, emergent knowledge about the visual world acquired during language pre-training—are composed of separable perception and reasoning priors with unique scaling trends and origins. We show that an LLM's latent visual reasoning ability is predominantly developed by pre-training on reasoning-centric data (\eg, code, math, academia) and scales progressively. This reasoning prior acquired from language pre-training is transferable and universally applicable to visual reasoning. In contrast, the perception prior emerges more diffusely from broad corpora, and perception ability is more sensitive to the vision encoder and visual instruction tuning data. In parallel, text describing the visual world proves crucial, though its performance impact saturates rapidly. Leveraging these insights, we propose a data-centric recipe for pre-training vision-aware LLMs and verify it in 1T token scale pre-training. Our findings are grounded in over 100 controlled experiments consuming 500,000 GPU-hours, spanning the full MLLM construction pipeline—from LLM pre-training to visual alignment and supervised multimodal fine-tuning—across five model scales, a wide range of data categories and mixtures, and multiple adaptation setups. Along with our main findings, we also propose and investigate several hypotheses, and introduce a Multi-Level Existence Bench (MLE-Bench) to facilitate future research. Together, this work provides a new way of deliberately cultivating visual priors from language pre-training, paving the way for the next generation of multimodal LLMs.
We recommend a visit to our project page (https://junlinhan.github.io/projects/lsbs/) for an interactive reading.
Systematic study reveals LLMs acquire visual perception priors from diverse data and reasoning priors from code/math corpora.
- Data-centric roadmap for developing multimodal systems showing visual priors emerge from separable perception and reasoning components
- Visual reasoning ability developed by reasoning-centric pre-training data with progressive scaling, universally applicable
- Perception prior emerges diffusely from broad corpora, more sensitive to vision encoder and instruction tuning
- Controlled pre-training experiments
- Vision-language model training
- Data composition analysis
- Multi-scale evaluation
- Public academic datasets
Investigation primarily centered on adapter-style MLLM architectures; may not fully generalize to discrete visual tokenization or end-to-end joint training approaches
from the paperDoes not address safety and ethical implications of learned visual priors that may encode societal biases from language corpora
from the paperConfined to static images, leaving exploration of visual priors for dynamic modalities like video as open question
from the paper
Investigate learned visual priors in alternative MLLM architectures with different training paradigms
from the paperConduct thorough audit of fairness and safety of emergent priors to identify biased visual associations
from the paperExplore visual priors for temporal reasoning, action recognition and causality in video understanding
from the paper
Author keywords
- LLM pre-training
- MLLMs
- multi-modality
Related orals
Benchmarking Empirical Privacy Protection for Adaptations of Large Language Models
Benchmarks practical privacy risks in differential privacy-adapted LLMs, revealing distribution shifts and model choice impact effectiveness.
Half-order Fine-Tuning for Diffusion Model: A Recursive Likelihood Ratio Optimizer
Proposes Recursive Likelihood Ratio optimizer for efficient fine-tuning of diffusion models with lower variance gradient estimation.
Invisible Safety Threat: Malicious Finetuning for LLM via Steganography
Demonstrates LLMs can be finetuned to generate harmful steganographically-hidden outputs while appearing benign to safety systems.
Reducing Belief Deviation in Reinforcement Learning for Active Reasoning of LLM Agents
Proposes T3 algorithm to detect belief deviation in LLM agents and truncate trajectories for improved reinforcement learning in active reasoning tasks.
RefineStat: Efficient Exploration for Probabilistic Program Synthesis
RefineStat enforces semantic constraints and applies diagnostic-aware refinement for synthesizing valid probabilistic programs from smaller language models.