--- title: "How AI image generators work: diffusion, step by step" description: "AI image generators start from pure noise and remove it step by step while your prompt steers each guess. Watch diffusion run live and try CFG." published: 2026-10-31 updated: 2026-10-31 canonical: https://protrailblazer.com/posts/how-ai-image-generators-work/ tags: [diffusion, image-generation, stable-diffusion, cfg, generative-ai, demo, machine-learning] --- # How AI image generators work: diffusion, step by step **Answer:** Most AI image generators, including Stable Diffusion, are diffusion models. They start from pure random noise and remove it in a few dozen steps. At each step a neural network predicts which part of the image is noise, the sampler subtracts some of it, and your prompt steers every prediction. The picture is uncovered, not painted. Watch the right-hand pane first. It shows what the model currently believes the finished image is, and it sharpens from a blur into a picture while the left-hand pane is still mostly static. Press play to run a generation, switch the prompt halfway through, or drag the scrub slider to add noise by hand. *Interactive demo: [open the diffusion demo](https://protrailblazer.com/demos/diffusion/)* ## What you're seeing The large pane on the left is `x_t`, the image at noise level `t`. A generation starts at `t = 1000`, which is pure random static, and ends at `t = 0`, the finished picture. The small pane in the middle is the noise the model predicts is sitting on top of the image right now. The pane on the right, outlined in red, is the model's current guess at the clean image: the noisy image with the predicted noise taken out. The three panes form an equation: noisy image minus predicted noise gives the believed image. Each step, the sampler uses that guess to take a small step toward a slightly less noisy image, then asks the model again. The histogram along the bottom shows the spread of pixel values in `x_t`. It starts as a bell curve, because pure noise is Gaussian, and ends up shaped like the target picture's own histogram (the thin line). The stats in the top right track the run: `T` is the current noise level, `ABAR` is how much of the original signal survives at that level (1 is all of it, 0 is none), `SNR` is the signal-to-noise ratio in decibels, and `STEP` counts denoising steps. A few things to try: - **Switch the prompt mid-run. **Start on the ridge, then pick the nebula around step 12. The believed image pivots toward the new target within one step, while the noisy image still carries traces of the old one. - **Drag guidance to 0. **The model falls back on its unprompted guess, a smeared average of every picture it knows. Drag it to 10 and the contrast overshoots, the highlights clip to flat white, and faint negatives of the other pictures appear. - **Compare 10, 25, and 50 steps. **Ten steps finishes grainy and blotchy. Fifty finishes clean. - **Scrub the noise level. **Dragging the slider runs the forward process by hand: the clean picture at 0, static at 1000. The believed-image pane shows what the model would guess at each level. - **Reseed. **A new seed is new starting noise, so the run takes a different path. ## How do AI image generators work? Most of today's image generators are diffusion models or close relatives of them. The idea comes in two halves. The forward half is fixed and needs no learning: take a real image and bury it in noise a little at a time until nothing is left. The reverse half is the learned one: a neural network practices undoing that damage. Generation runs only the reverse half, starting from noise that never had an image in it. Your text prompt enters as a set of embeddings (a numeric representation of its meaning, the same idea as in [the 3D embeddings demo](https://protrailblazer.com/posts/embeddings-in-3d/)). Those vectors condition every noise prediction, so the model removes the noise that does not look like "a mountain ridge at dusk" and keeps the structure that does. ## What is a diffusion model? A diffusion model is a neural network trained to predict the noise in a noisy image. Training is simple to describe. Take a real image, pick a random noise level, mix in that much random noise, and ask the network to name the noise. The loss is how far its guess lands from the noise actually added, and gradient descent nudges the weights to shrink it. The [gradient descent demo](https://protrailblazer.com/posts/gradient-descent-demo/) shows that loop on its own. Repeat it across billions of images and every noise level, and the network gets very good at one job: looking at static and saying which part is static. The forward process has a closed form, which is what makes this practical. For any noise level `t`: ``` x_t = sqrt(abar_t) * x_0 + sqrt(1 - abar_t) * noise ``` `x_0` is the clean image and `abar_t` (alpha-bar) is a schedule that falls from 1 at `t = 0` to 0 at `t = 1000`. The demo uses the cosine schedule from Nichol and Dhariwal (2021), so the scrub slider computes exactly this line. Because the formula jumps straight to any noise level, training never has to simulate a thousand small steps. ## How does Stable Diffusion work? Stable Diffusion adds one trick to everything above: it does not denoise pixels. A separate network (an autoencoder) compresses each image into a much smaller "latent" grid, the diffusion happens in that compressed space, and the decoder turns the final latent back into pixels at the end. That is the "latent diffusion" design from Rombach and colleagues (2022), and it is why the model runs on a consumer graphics card. At generation time the steps are the ones in the demo: 1. Encode the prompt into embeddings with a text encoder. 2. Draw random noise from the seed. Same seed, same noise. 3. Ask the network for its noise prediction twice, once with the prompt and once without, and combine the two with guidance (next section). 4. Turn the noise prediction into a guess at the clean image, then step to the next, lower noise level. The demo uses DDIM-style steps, which jump straight between a handful of noise levels instead of walking all 1,000. 5. Repeat for the chosen number of steps, then decode the result into pixels. Newer models change the plumbing, not the idea. Stable Diffusion 3 (2024) and Black Forest Labs' FLUX models train with rectified flow (also called flow matching), which learns a straighter path from noise to image so fewer steps are needed. They still start from noise and remove it under a prompt. Not every generator works this way: OpenAI's GPT-4o system card (March 2025) describes its image generation as autoregressive, building the picture piece by piece the way a language model writes text, though outside researchers have argued a diffusion decoder may still be involved. ## What does CFG scale do? CFG stands for classifier-free guidance (Ho and Salimans, 2022). At every step the model makes two predictions: one conditioned on your prompt and one with an empty prompt. Guidance takes the difference and scales it: ``` guided = unconditional + scale * (conditional - unconditional) ``` At scale 1 you get the plain conditional prediction. Above 1, the model is pushed further toward the prompt and away from the generic answer. Many Stable Diffusion tools use a value around 7 to 7.5. Push it too far and the pixel values overshoot their valid range, the sampler clamps them, and you get the burned, oversaturated look people call "overcooked". The guidance slider in the demo runs this exact formula. Its unconditional branch only knows the average of the three pictures, which is why scale 0 gives mush. Its conditional branch only partly commits to the prompt, so scale 7.5 lands right on the target. At 10, the overshoot clips the sky to flat white and pushes the image away from the average so hard that faint negatives of the other two pictures show through. ## Why do AI images start as noise? Because noise is the one starting point the model has seen at every scale. During training it saw images at every noise level, including total static, so it knows what to do from there. Starting from noise also gives variety for free: each seed is a different static pattern, the model resolves each into a different image, and the same seed always resolves the same way. This is the image version of [temperature in language models](https://protrailblazer.com/posts/llm-temperature-explained/): randomness is a design choice, and fixing the seed makes the output repeatable. Watch the right-hand pane again with this in mind. At high noise the model's best guess is a blurry blob, because fine detail is truly unrecoverable under that much static. As the noise drops, the blob sharpens: first the horizon, then the peaks, then the edges. The image is not assembled from parts. It is uncovered, coarse to fine. ## Why more steps look cleaner Each step is a jump from one noise level to a lower one, and the network's prediction is never perfect. Small jumps keep each error small and give the next step a chance to correct it. Big jumps carry more error per step, and the last jump's error lands directly in the final image. The demo adds an error at each step proportional to the jump size. With the default settings, 10 steps finishes with a residual error of `0.14` (root mean square, on a pixel scale from minus 1 to 1), 25 steps with `0.06`, and 50 steps with `0.03`. Ten steps is about 5 times worse, and you can see it as grain and blotches. Real samplers fight this with better step rules, and distilled "turbo" models are trained to produce good images in very few steps. The tradeoff is the same one the demo shows: fewer steps is faster and rougher. ## What the demo simplifies - **The noise predictor is an oracle. **A real model learns its prediction from billions of images. Here the true target image plays that role, blurred in proportion to the noise level the way a real denoiser's best guess is blurry at high noise. That keeps the process visible without a trained network. The demo says so on screen. - **The unprompted branch is an average. **A real unconditional model can draw plausible images on its own. Here it only knows the mean of the three targets, plus a smooth guessing error that changes with the seed. - **Pixels, not latents. **The demo denoises a 160 by 160 single-channel image directly. Stable Diffusion works in a compressed latent space and decodes at the end. - **Three images, not billions. **Switching the prompt mid-run snaps the final image to the new target. A real model mid-run is already committed to a layout and can only bend it. - The schedule, the forward equation, the DDIM update, the guidance formula, and the clamp are the real math. ## Frequently asked questions ### What is a diffusion model in simple terms? A neural network trained to look at a noisy image and predict the noise in it. Run that prediction in a loop, starting from pure static, and an image emerges one denoising step at a time. ### What does CFG scale do in Stable Diffusion? It controls how hard each step leans toward your prompt. The model predicts noise with and without the prompt, and the scale multiplies the difference. Around 7 is a common setting. Low values ignore the prompt; very high values oversaturate and clip. ### Why do AI images start as noise? The model is trained to remove noise at every level, including total static, so pure noise is a starting point it knows how to work from. Different noise (a different seed) gives a different image. ### What is a seed in AI image generation? The number that sets the starting noise. With the same seed, prompt, model, and settings, a generator produces the same image again. ### How many steps does a diffusion model need? Classic samplers use 20 to 50. Fewer steps are faster but leave more error, which shows up as grain or mush. Distilled models can work in one to four steps because they were trained for it. ### Is Midjourney a diffusion model? Midjourney has not published its architecture. It is widely described as diffusion-based, and it behaves like one (seeds, step-style refinement, guidance-like settings), but the company has not confirmed the details. ## Key takeaways - Training a diffusion model teaches one skill: given a noisy image and a noise level, predict the noise. Generation is that same skill run in a loop, starting from pure static. - At every step the model holds a full guess at the finished image. Early guesses are blurry averages, because heavy noise hides fine detail, and they sharpen as the noise drops. - The prompt does not draw anything. It conditions the noise prediction, and classifier-free guidance multiplies the gap between the prompted and unprompted predictions. Too little gives mush. Too much clips and overcooks. - Fewer steps means bigger jumps and more error per jump. In the demo, 10 steps leaves about 5 times the residual error of 50 steps (RMSE 0.14 vs 0.03). - The seed is the starting noise. Same seed, same prompt, same settings gives the same image, which is why tools let you lock it.