Improving Diffusion Models for Class-imbalanced Training Data via Capacity Manipulation
Feng Hong, Jiangchao Yao, Yifei Shen, Dongsheng Li, Ya Zhang, Yanfeng Wang
Abstract
While diffusion models have achieved remarkable performance in image generation, they often struggle with the imbalanced datasets frequently encountered in real-world applications, resulting in significant performance degradation on minority classes. In this paper, we identify model capacity allocation as a key and previously underexplored factor contributing to this issue, providing a perspective that is orthogonal to existing research. Our empirical experiments and theoretical analysis reveal that majority classes monopolize an unnecessarily large portion of the model's capacity, thereby restricting the representation of minority classes. To address this, we propose Capacity Manipulation (CM), which explicitly reserves model capacity for minority classes. Our approach leverages a low-rank decomposition of model parameters and introduces a capacity manipulation loss to allocate appropriate capacity for capturing minority knowledge, thus enhancing minority class representation. Extensive experiments demonstrate that CM consistently and significantly improves the robustness of diffusion models on imbalanced datasets, and when combined with existing methods, further boosts overall performance.
Capacity manipulation improves diffusion models' handling of class-imbalanced data by reserving capacity for minority classes via low-rank decomposition.
- Identifies model capacity allocation as key factor in class imbalance performance degradation
- Proposes capacity manipulation loss using low-rank decomposition to explicitly reserve model capacity for minority classes
- Demonstrates robustness to extreme imbalance ratios across various image datasets and training scenarios
- Low-rank decomposition
- Capacity manipulation loss
- Diffusion models
- iNaturalist
- ImageNet
While CM demonstrates robustness to extreme imbalance ratios (e.g., IR=500 on iNaturalist), limitations exist in scenarios of absolute sample scarcity where the reserved capacity lacks sufficient data to learn meaningful representations, limiting generation quality
from the paperApplicability and adaptation of capacity manipulation for different data modalities (e.g., video, 3D data) and other types of generative models remains unexplored
from the paper
Evaluate applicability and potential adaptations of capacity manipulation for different data modalities (e.g., video, 3D data) and other types of generative models
from the paperAddress few-shot to zero-shot transition with extremely low absolute number of minority samples
from the paper
Author keywords
- Imbalance
- Diffusion Models
Related orals
Depth Anything 3: Recovering the Visual Space from Any Views
DA3 predicts spatially consistent 3D geometry from arbitrary camera views using plain transformer and depth-ray targets.
Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generator
VIST3A stitches text-to-video models with 3D reconstruction systems and aligns them via reward finetuning for high-quality text-to-3D generation.
Radiometrically Consistent Gaussian Surfels for Inverse Rendering
RadioGS introduces radiometric consistency supervision for inverse rendering to accurately model indirect illumination in Gaussian-based representations.
True Self-Supervised Novel View Synthesis is Transferable
Presents XFactor, first geometry-free self-supervised model for transferable novel view synthesis without 3D inductive biases.
Locality-aware Parallel Decoding for Efficient Autoregressive Image Generation
Introduces parallel decoding for autoregressive image generation with flexible ordering achieving 3.4x latency reduction.