Course Content

Text-to-image: conditioning, guidance and the known weaknesses

Three components turn Section 8's unconditional denoiser into a text-to-image model: a text encoder, cross-attention, and a guidance scale. Understanding them is what lets you explain, in the second half of this lesson, why the model still gets a red cube and a blue sphere wrong.

The text encoder

The prompt is tokenised and passed through a text encoder, producing a sequence of embeddings — one per token — not a single vector. That distinction matters: keeping a sequence is what allows the image to attend to different words at different points.

Two families, with different strengths:

Contrastive text encoders — the text half of an image-text model trained as in the image captioning training lesson. Their representations are already aligned to images, which makes them excellent at nouns, styles, and visual concepts. They are usually trained on short captions and are correspondingly weak on long, syntactically complex prompts.
Large language-model encoders — a general text model used as a feature extractor. Far better at long prompts, clauses, and compositional structure, and not natively aligned to images, so the model has to learn the mapping.

Many systems use both, concatenating the two sets of embeddings, which is a straightforward way to get the visual grounding of one and the syntactic competence of the other. If asked to choose one, the honest answer is that longer and more compositional prompts push you towards the language-model encoder, and that this choice is an area where practice is still moving.

Cross-attention: how the text reaches the image

The denoising network from How diffusion works, slowly is modified by inserting cross-attention layers at several resolution levels of its U-shaped path. Transformer-based denoisers, now common in recent models, do the same job either with cross-attention in every block or by attending over image and text tokens jointly; the soft-blend behaviour described below applies either way.

At each such layer, the image features query the text embeddings and pull in what is relevant. This is exactly the mechanism from the image captioning model lesson running in the opposite direction: there, a generated word attended over image regions; here, an image region attends over prompt tokens.

Two properties worth stating, because they explain a great deal about the known weaknesses below:

It happens at every step. All 25 or 50 denoising steps consult the text. The prompt is not read once and stored; it is re-queried continuously, which is why guidance strength affects the whole trajectory rather than only the beginning.

It happens at multiple resolutions. Coarse levels, where global composition is decided, and fine levels, where texture is decided, both attend to the text. That is why a prompt can influence both layout and surface detail.

And the property that causes trouble: cross-attention produces a soft weighted blend over tokens for each image position. There is no mechanism that binds red to cube rather than to sphere, and no mechanism that counts. Attention decides how much each token matters at each position; it does not decide which object a word belongs to.

Classifier-free guidance and the guidance scale

The model is trained to predict noise both with the text conditioning and without it, by randomly dropping the prompt for a fraction of training examples (see the diffusion training lesson). At generation time, both predictions are computed at every step, and the final prediction pushes away from the unconditional one towards the conditional one:

prediction = unconditional + scale × (conditional − unconditional)

The guidance scale is how far to push. Scale 1 is the plain conditional prediction. Higher scales force stronger adherence to the prompt at the cost of image quality and diversity. Because both predictions are computed at every step, guidance doubles the number of network passes — the term most often missing from a candidate's cost estimate.

What the scale actually does

Illustrative behaviour for a typical model, described rather than measured:

Scale	What you get
1	Beautiful, varied, and frequently ignores the prompt
3	Loosely follows the prompt; high diversity
7	The usual default — good adherence, good quality
12	Strong adherence; colours saturate, contrast hardens, composition simplifies
20	Prompt-obedient and visibly degraded — burnt highlights, posterised colour, flat scenes

The curve is not monotone in usefulness. Alignment rises with scale and quality falls, and they cross somewhere around 6–9 for most models. That crossover is a per-model tuning job and it is worth exposing to advanced users while keeping a sensible default for everyone else.

One refinement worth knowing: guidance can be varied across the trajectory. High guidance early, when composition is being decided, and lower guidance late, when texture is being filled in, gives much of the adherence at less of the quality cost — and, as the diffusion serving lesson noted, dropping guidance entirely on late steps also recovers part of its cost.

The dashed arrows repeat at every block and every step — conditioning is continuous, not an initial condition.

The known weaknesses

Naming these honestly is what a strong candidate does, and they are the natural deep-dive questions. Each has a mechanism, and the mechanism is what makes the answer good.

Each traces back to the text encoder producing one blurred vector where the prompt had structure.

Text rendering inside images

Ask for a shop sign reading "OPEN" and you frequently get letter-shaped marks that are not letters.

Three compounding causes:

The text encoder does not see characters. It sees subword tokens. The token for OPEN carries the meaning of the word, not the shapes of four specific glyphs in order.
The autoencoder ceiling. As the latent diffusion lesson showed, small text often does not survive an encode-decode round trip. If the decoder cannot render it, the diffusion model cannot produce it. Run the round-trip diagnostic before blaming the generator.
The captions never transcribed it. Web alt-text almost never says what text appears in the image, so the model was never trained to associate a specific string with specific glyphs.

What has improved it, in order of impact: re-captioning that explicitly transcribes visible text, character-aware or byte-level text encoders, and less aggressive autoencoder compression. This is the weakness where the most progress has been made recently, and it is progress rather than a solution.

Counting

"Five apples" frequently produces four or six.

There is no counting mechanism anywhere in the architecture. Cross-attention distributes attention over tokens; nothing accumulates a tally. The model learns statistical associations between number words and typical quantities in training images, and those associations are weak because captions rarely state counts accurately.

Small numbers — one, two, three — work reasonably because they are common and visually distinct. Beyond about four, performance degrades sharply. Say that; it is more useful than "counting is hard".

Spatial relations

"A cube to the left of a sphere" produces the right objects in an arbitrary arrangement roughly as often as not.

Same root cause: attention says how much a token matters at a position, not where an object should go, and training captions rarely describe spatial layout. The model has learned which objects co-occur, not how they are arranged.

Attribute binding

The clearest single demonstration of the problem. "A red cube and a blue sphere" produces a blue cube and a red sphere often enough to be a standard test case.

The mechanism: the prompt's embeddings carry red, cube, blue, and sphere as a sequence, and cross-attention lets every image region attend to all of them. Nothing enforces that the region rendering the cube should attend to red and not to blue. Attributes leak across objects.

Compositional prompts

Combine the above — several objects, each with attributes, in specified relations, in specified quantities — and reliability falls off a cliff. Two objects with two attributes is often fine; four objects with relations between them is essentially unreliable.

These are unsolved, and saying so is the strong answer

There is no complete fix for compositional generation. The mitigations below help measurably and none of them closes the gap:

Better data (the re-captioning from the data lesson) — the largest single improvement, and it cannot teach what the captioner itself cannot describe.
Stronger text encoders — helps most on long and syntactically complex prompts.
Attention guidance at generation time — inspect the cross-attention maps mid-generation and nudge them so each object's tokens concentrate on a distinct region. Genuinely effective on two-object binding, adds compute, and does not scale to complex scenes.
Layout conditioning — have the user, or a language model, supply a bounding box per object and condition on it. This works well, because it removes the hard part from the model. The cost is that the interface is no longer a sentence.
Generate and verify — generate, ask a vision-language model whether the prompt is satisfied (the question-based check from the metrics lesson), and retry if not. Effective, and it multiplies cost by the average number of attempts, so budget it as such.

The candidate who says "this is a known open problem, here are five partial mitigations and what each costs" is giving a better answer than one who claims a solution.

Key Takeaway

The text encoder produces one embedding per token, and cross-attention at several resolutions lets image features query those tokens at every denoising step — the captioning mechanism reversed. Classifier-free guidance pushes the prediction away from the unconditional one; the scale trades adherence against quality, crosses over around 6–9, and doubles the network passes. Cross-attention is a soft blend that neither binds adjectives to nouns nor counts nor places objects, and training captions rarely described relations, which is why attribute binding, counting beyond about four and spatial relations fail; text rendering fails because the encoder sees subword tokens, the autoencoder loses small text and captions never transcribed it. Re-captioning, stronger encoders, attention guidance, layout conditioning and generate-and-verify all help, none is a fix, and saying so is the correct answer.