Pinet: Optimizing hard-constrained neural networks with orthogonal projection layers
Panagiotis D. Grontas, Antonio Terpin, Efe C. Balta, Raffaello D'Andrea, John Lygeros
Abstract
We introduce an output layer for neural networks that ensures satisfaction of convex constraints. Our approach, $\Pi$net, leverages operator splitting for rapid and reliable projections in the forward pass, and the implicit function theorem for backpropagation. We deploy $\Pi$net as a feasible-by-design optimization proxy for parametric constrained optimization problems and obtain modest-accuracy solutions faster than traditional solvers when solving a single problem, and significantly faster for a batch of problems. We surpass state-of-the-art learning approaches by orders of magnitude in terms of training time, solution quality, and robustness to hyperparameter tuning, while maintaining similar inference times. Finally, we tackle multi-vehicle motion planning with non-convex trajectory preferences and provide $\Pi$net as a GPU-ready package implemented in JAX.
Enforces convex output constraints via operator splitting enabling fast parametric optimization solving.
- Introduces PiNet output layer ensuring satisfaction of convex constraints via operator splitting
- Leverages implicit function theorem for efficient backpropagation
- Obtains modest-accuracy solutions faster than traditional solvers for single problems
- Achieves orders of magnitude faster training than learning approaches with similar inference times
- Operator splitting
- Convex optimization
- Implicit function theorem
Requires convex constraint sets; many applications involve only convex constraints but future work should relax assumption
from the paper
Investigate sequential convexification for non-convex constraints
from the paperApply to neural PDE solvers, scheduling, and robotics applications
from the paperIntegrate hard constraints into large-scale models
from the paper
Author keywords
- hard constrained neural networks
- network architecture
- implicit layers
- operator splitting
- optimization
Related orals
On The Surprising Effectiveness of a Single Global Merging in Decentralized Learning
Shows decentralized learning with single global merging achieves convergence rates matching parallel SGD under data heterogeneity.
Non-Convex Federated Optimization under Cost-Aware Client Selection
Develops efficient federated optimization algorithm with cost-aware client selection achieving best communication and local complexity.
Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Data
Analyzes phase retrieval learning dynamics with anisotropic data, deriving explicit scaling laws and three-phase trajectories.
A Representer Theorem for Hawkes Processes via Penalized Least Squares Minimization
Representer theorem for Hawkes processes shows dual coefficients are analytically fixed to unity via penalized least squares.
Quantitative Bounds for Length Generalization in Transformers
Quantitative bounds show training length required for length generalization depends on periodicity, locality, alphabet size, and model norms.