Institute Homepage
Institute Homepage EN Sign In

Back

Empirische Inferenz Perzeptive Systeme Conference Paper 2025

Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets

Thumb ticker sm zhenliu
Perzeptive Systeme
Zhen Liu
Thumb ticker sm myphoto xs
Empirische Inferenz
Tim Xiao
Collaborator
Thumb ticker sm weiyang photo
Empirische Inferenz
Weiyang Liu
  • Postdoctoral Researcher
Thumb xxl residualdb

While one commonly trains large diffusion models by collecting datasets on target downstream tasks, it is often desired to align and finetune pretrained diffusion models with some reward functions that are either designed by experts or learned from small-scale datasets. Existing post-training methods for reward finetuning of diffusion models typically suffer from lack of diversity in generated samples, lack of prior preservation, and/or slow convergence in finetuning. Inspired by recent successes in generative flow networks (GFlowNets), a class of probabilistic models that sample with the unnormalized density of a reward function, we propose a novel GFlowNet method dubbed Nabla-GFlowNet (abbreviated as ∇-GFlowNet), the first GFlowNet method that leverages the rich signal in reward gradients, together with an objective called ∇-DB plus its variant residual ∇-DB designed for prior-preserving diffusion finetuning. We show that our proposed method achieves fast yet diversity- and prior-preserving finetuning of Stable Diffusion, a large-scale text-conditioned image diffusion model, on different realistic reward functions.

Author(s): Liu, Z. and Xiao, T. Z. and Liu, W. and Bengio, Y. and Zhang, D.
Book Title: The Thirteenth International Conference on Learning Representations (ICLR)
Year: 2025
Month: April
Bibtex Type: Conference Paper (conference)
Event Place: Singapore
State: Accepted
Links:

BibTex 

@conference{Liuetal25,
  title = {Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets},
  booktitle = {The Thirteenth International Conference on Learning Representations (ICLR)},
  abstract = {While one commonly trains large diffusion models by collecting datasets on target downstream tasks, it is often desired to align and finetune pretrained diffusion models with some reward functions that are either designed by experts or learned from small-scale datasets. Existing post-training methods for reward finetuning of diffusion models typically suffer from lack of diversity in generated samples, lack of prior preservation, and/or slow convergence in finetuning. Inspired by recent successes in generative flow networks (GFlowNets), a class of probabilistic models that sample with the unnormalized density of a reward function, we propose a novel GFlowNet method dubbed Nabla-GFlowNet (abbreviated as ∇-GFlowNet), the first GFlowNet method that leverages the rich signal in reward gradients, together with an objective called ∇-DB plus its variant residual ∇-DB designed for prior-preserving diffusion finetuning. We show that our proposed method achieves fast yet diversity- and prior-preserving finetuning of Stable Diffusion, a large-scale text-conditioned image diffusion model, on different realistic reward functions.},
  month = apr,
  year = {2025},
  slug = {liuetal25},
  author = {Liu, Z. and Xiao, T. Z. and Liu, W. and Bengio, Y. and Zhang, D.},
  month_numeric = {4}
}