What are diffusion models, and how exactly do they stand out from Autoregressive Models?

Since the popularity of generative models exploded, many started to pursue AR (Autoregressive) models, the brightest example of which are GPT-style Transformer models.

Fyi, AR models generate tokens from left-to-right. So, in sampling process, model doesn’t see other tokens, only previous ones, like illustrated below.

But, AR models have one of the biggest issues that costs accuracy. Ironically, this is what made them best for generative tasks, which is…. Yes, left-to-right bias. The problem is models are not perfect: next token generations depend on estimating a probability distribution over possible upcoming tokens and sampling one from that distribution. And models sometimes mess up; the token showing the highest probability may not be the correct one.

This can be seen with a simple example. Imagine the model has to generate this sentence: “The dog barked at the stranger”. The model first generates the word “The”. Since it doesn’t have context about the entire sentence, it should rely purely on the word “The” to make the next guess. In other words, it doesn’t see the words “barked at the stranger”. If the probability for “cat” is higher than “dog”, the model may be more likely to sample “cat”. Then, since cats meow instead of barking, it will naturally choose the word “meow”, which is nowhere as close to the expected output.

On top of that, Transformers are expensive, because they need to perform forward passes for every single predicted token. If we are dealing with images, instead of text, then the cost is going to be astronomical.

This leads us to another type of models — NARs (Non-autoregressive models); the models that can process the entire text or images simultaneously, unlike AR ones, at much cheaper costs.

The NAR model we are about to discuss is Diffusion Models. They work completely differently than AR models. Instead of predicting the token step-by-step, it repeatedly updates the whole sample at each denoising step. This is like comparing CPU and a GPU: the first one completes tasks sequentially, while the latter is used for parallelization. Diffusion models are much cheaper than expensive autoregressive models that make an entire forward pass just to predict the next token, word, or image pixel. This is why they are an industry standard right now.

Suppose you want to train a Diffusion model that generates high-quality images. In the training process, the model is given an image (say, of a dog). Then, the model continuously applies noise, matching Gaussian distribution, until the image is nothing but static. Diffusion has to learn how to make the static go back into the original image; in other words, the denoising process. To do that, it first needs to learn to predict the applied noise correctly. After each iteration, it compares the predicted noise with the actual noise, and updates its parameters to reduce the prediction error. You may say this contradicts the claim that diffusions are cheaper, because they still use an iterative process, just like transformers do. But, the key difference is that diffusions need an extremely small number of timesteps to denoise, while autoregressive generation is sequential”. Hence, this is a significant achievement in cost optimization.

You can see a more clearer illustration of the training process below.

To conclude, while autoregressive models are amazing for text generation, representation, and similar tasks, diffusions are the absolute leaders in image and video generation.

PS. All em dashes and semi-columns are written by me.