Transformer Architecture Q&A

Course Overview

Advanced

Free Course

For engineers preparing for ML, LLM and AI-engineering interviews who need to explain how Transformers work, not just use them. You will be able to answer questions on attention, masking, positional encodings, normalisation, feed-forward blocks, the KV cache and GPT-style models, working the key calculations by hand.

Instructor: MantraMindAI

Sections: 6

Course Content

Section 1: Foundations and Attention Basics

Prepares you to explain a Transformer forward pass end to end and to work scaled dot-product attention by hand, from Q, K and V to the final weighted sum.

1. Describe the Transformer architecture and how a forward pass produces an output?

2. Define a Large Language Model (LLM) in your words and outline how it generates text?

3. List the major building blocks of a Transformer and briefly state each role?

4. Explain self-attention and how a token attends to others within a Transformer layer?

5. What roles do Query, Key, Value play in attention, and how do they interact mathematically?

6. Why does multi-head attention exist? What advantages do multiple heads provide?

7. Compare "attention" vs "self-attention"—when are they the same and when do they differ?

8. Walk through Bahdanau attention and why it was a turning point in sequence modeling?

9. Describe scaled dot-product attention and why divide scores by √d_k?

10. Distinguish attention scores vs weights—when do scores become weights?

Section 2: Masking and Positional Encoding

Prepares you to explain how masks control what each token can see and how sinusoidal, learned, RoPE and ALiBi positions tell a Transformer about word order.

1. Define causal masking and explain when and why it is applied during attention?

2. Why does setting masked positions to −∞ (not 0) work correctly after softmax?

3. What is positional encoding, and why can a Transformer not function without position info?

4. Explain RoPE and where in the attention pipeline it is applied?

5. Contrast ALiBi vs RoPE vs learned positional embeddings—what trade-offs do they represent?

6. When training with packed documents, what issues do absolute positional embeddings introduce?

Section 3: Architecture Variants and Components

Prepares you to compare encoder, decoder and encoder-decoder models and to explain what residuals, normalisation and the feed-forward block (GELU, SwiGLU) each contribute inside a Transformer layer.

1. Compare encoder-only, decoder-only, encoder-decoder Transformers and their best use cases?

2. How does bidirectional encoder self-attention differ from causal decoder self-attention?

3. What does "autoregressive" mean in Transformers, and how does it differ in GPT vs BERT?

4. Explain role of residual connections in a Transformer block and what fails without them?

5. Describe the feed-forward network in a Transformer block—what it computes and why it exists?

6. FFN as “memory bank + generalizer”—what does that mean in Transformers?

7. Why has GELU largely replaced ReLU in Transformer FFNs?

8. What is SwiGLU, and why do modern LLMs prefer it over GELU/ReLU?

9. Why is layer norm used in Transformers instead of batch norm? Give a concrete reason?

10. Why does attention weight matrix have shape (seq_len × seq_len), not embedding size?

Section 4: Inference, KV Cache, and Scaling

Prepares you to explain why LLM inference is slow and memory-hungry — the KV cache, prefill versus decode, context limits — and how GQA, FlashAttention, sparse and ring attention, and mixture-of-experts reduce that cost.

1. What is the KV cache, and how does it speed up autoregressive decoding?

2. Differentiate prefill vs decode phase in LLM inference and what dominates cost in each?

3. What happens if output exceeds the model’s context window, functionally and in quality?

4. Explain Grouped-Query Attention (GQA) vs Multi-Head Attention—memory and quality differences?

5. Describe Flash Attention at a high level—why is it faster than naive attention?

6. What are sparse/local attention patterns, and how were they used in models like GPT-3?

7. Explain ring attention and how it enables very long contexts across multiple devices?

8. Describe MoE and how Mixtral routes tokens at inference time?

9. What separates dense Transformers from sparse (MoE) models in architecture and compute?

Section 5: GPT Internals and Attention Deep-Dive

Prepares you to explain what happens inside a GPT attention layer — heads and their sizes, Q/K/V and the output projection, masking, softmax and dropout — well enough to spot a bug in someone else's implementation.

1. What does GPT stand for, and what is the architectural intent behind it?

2. How are head dim, embedding dim, and number of heads related in multi-head attention?

3. Why is there an output projection (W_o) at end of multi-head attention, and what does it do?

4. Describe how causal mask is combined with attention scores and why order matters?

5. What is a "context vector" in self-attention, and why is it the key output?

6. Walk through how attention can resolve a pronoun to its antecedent with an example?

7. Why is dot product (not cosine or Euclidean) used as similarity in attention?

8. When visualizing attention weights, what patterns appear and what do they indicate?

9. Why is softmax used to normalize attention scores instead of dividing by their sum?

10. What roles do W_q, W_k, W_v play, and why must they be learned (not fixed)?

11. Difference between model "weight parameters" and "attention weights" at inference?

12. Why apply softmax along dim=-1 in attention, and what if along another dimension?

13. What purpose does dropout serve when applied within the attention mechanism?

14. Identify where dropout is typically applied in GPT architecture?

15. Explain combined QKV projection and why it’s faster than three separate projections?

Section 6: GPT Model Architecture End-to-End

Prepares you to trace a GPT model from token ids through embeddings, blocks and the LM head to generated text, and to do the parameter, shape and decoding reasoning interviewers ask for.

1. Trace the full GPT model from input tokens to output logits, naming each step?

2. What is GPT output tensor shape per forward pass, and why that shape?

3. Define weight tying in GPT-2 and why it’s 124M params vs ~163M raw?

4. Compare GPT-2 sizes—how do depth, width, and head count scale across models?

5. Architecturally, how does GPT-3 differ from GPT-2 beyond just scale?

6. Trace how vocab links tokenizer, embedding layer, and LM head into one system?

7. What is the embedding matrix, and how does its parameter count compare to the rest of GPT?

8. Contrast 2017 Transformer block vs Llama-3-style block—what concrete changes?

9. Walk through autoregressive text generation step by step, from first token to stop?

10. Define greedy decoding and discuss its main weaknesses?