💡Continuous Diffusion Models Make a Comeback in 2025
Continuous Diffusion is Back, But Why?
TL;DR
After years of dormancy, continuous diffusion models for language are making a comeback. Researchers are exploring the benefits of these models, which offer unique advantages over discrete methods.
Continuous diffusion models for language are experiencing a resurgence after years of dormancy. Modern language models are typically autoregressive, but diffusion models reverse a corruption process to generate sequences. In 2021, the first attempts to apply continuous diffusion to language involved replacing a continuous corruption process with a discrete one. By 2023, virtually all new research used discrete diffusion, but in 2025, continuous methods are back. Researchers are exploring the benefits of continuous diffusion, including its ability to represent uncertainty at the token level and a rich toolbox of sampling algorithms. The key question: can continuous diffusion models match or outperform autoregressive models in specific settings?
Key Points
Continuous diffusion models for language were dormant from 2023 to 2024, with discrete methods dominating.
In 2025, continuous diffusion models are marked in yellow on a survey paper, while discrete methods are in green.
Diffusion-LM addressed the incompatibility between categorical data and corruption with Gaussian noise in a unique way.
In May 2023, Plaid-1B was quantified as 64x less efficient than likelihood-based continuous diffusion language models.
LLM community was focused on the pareto frontier of training compute versus perplexity, but LLaMA challenged this in 2023.
Why It Matters
If you're working on language generation, continuous diffusion models offer unique advantages like uncertainty representation at the token level. However, their efficiency gap compared to discrete methods remains a significant challenge. Researchers are now exploring the potential of continuous diffusion in specific settings, which could impact future model development.
Frequently Asked Questions
Why does this matter?
If you're working on language generation, continuous diffusion models offer unique advantages like uncertainty representation at the token level. However, their efficiency gap compared to discrete methods remains a significant challenge. Researchers are now exploring the potential of continuous diffusion in specific settings, which could impact future model development.
What happened?
After years of dormancy, continuous diffusion models for language are making a comeback. Researchers are exploring the benefits of these models, which offer unique advantages over discrete methods.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,399 builders reading daily.