Transformers in Language Tasks

A team needs to route support tickets into eight categories. Someone suggests a large generative model with a carefully engineered prompt. It works — 88% accuracy, roughly 900 ms per ticket, and a bill that scales with volume. A colleague fine-tunes a 110-million-parameter BERT on 4,000 labelled tickets instead: 94% accuracy, 12 ms per ticket on a single GPU, no per-call cost.

The generative model was not badly chosen because it is generative. It was badly chosen because the task is understanding, and a causal decoder deliberately throws away half the context of every token it reads. When you classify a ticket you have the whole document in hand — hiding its second half from its first half is a constraint with no benefit.

That is the thread running through this lesson. Three model families dominate language work, they differ in exactly one design decision — which tokens each token is allowed to see — and that one decision determines which tasks each family is good at.

Three shapes of language task

Make the constraint concrete. Take the sentence The bank raised rates again. To decide whether bank is a financial institution or a riverbank, the decisive word is rates, which appears afterwards. A causal model must commit to a representation of bank having seen only The. A bidirectional model sees the entire sentence when building every token's representation. On understanding tasks that difference is worth several points, consistently.

BERT: bidirectional encoding

The architecture

BERT is a transformer encoder stack and nothing else — no decoder, no cross-attention, no generation. Its self-attention is masked only for padding, never causally.

Input is wrapped in special tokens: [CLS] the bank raised rates . [SEP]. The [CLS] position has no word of its own, so through twelve layers of attention it becomes a learned summary of the whole sequence — that is the vector a classifier head reads. [SEP] separates two segments when a task needs a pair, such as a question and a passage.

The training objective, and why it had to be invented

You cannot train a bidirectional model with next-token prediction. If every token sees every other token, predicting token $i$ is trivial — token $i$ is right there in the input. The objective collapses.

BERT's answer is masked language modelling: corrupt 15% of tokens and predict the originals from both sides of context. The corruption is not uniform:

On a 512-token sequence: $0.15 \times 512 = 76.8$, so roughly 77 positions are selected. About 61 become [MASK], 8 become random tokens, 8 stay as they are.

Now notice the cost. BERT receives a gradient signal on 77 of 512 positions. A causal model receives one on all 512 — every position predicts the next. Per token of text consumed, causal language modelling delivers roughly $512/77 \approx 6.7$ times more prediction targets. That is a real and large efficiency gap, and it is the price of bidirectionality.

Masked language modelling buys bidirectional context at the cost of learning from only 15% of the tokens it reads. Causal language modelling learns from all of them but only ever sees the left half.

BERT also trained on next sentence prediction: given two segments, classify whether B actually followed A. Later work showed this contributed little or nothing, and RoBERTa dropped it entirely with better results.

From pre-training to a task

Fine-tuning attaches a small head and trains everything end to end. Different tasks read different positions:

The head is tiny — for 8-class classification it is $768 \times 8 + 8 = 6{,}152$ parameters against BERT's 110 million, about 0.006%. Nearly all the capability came from pre-training; fine-tuning is mostly adjusting what is already there.

The family

ELECTRA is worth pausing on, because it attacks exactly the inefficiency identified above: it recovers a full-token training signal while keeping bidirectional context.

GPT: causal decoding

The architecture

A GPT model is a transformer decoder stack with the cross-attention sub-layer deleted — so a stack of blocks with causal self-attention and a feed-forward network, nothing else. Position $i$ attends to positions $\le i$ only, enforced by adding $-\infty$ to future scores before the softmax.

The arithmetic of that mask, at position 3 of a four-token sequence with scaled scores $[2.0, 1.0, 3.0, 5.0]$:

Without the mask, 83% of this position's attention would go to the very token it is being trained to predict. The masked row renormalises over the three legal positions: $e^2 = 7.389$, $e^1 = 2.718$, $e^3 = 20.086$, sum $30.193$, giving $0.245$, $0.090$, $0.665$.

The objective

Causal language modelling maximises the likelihood of each token given everything before it:

Every position contributes. Any text at all is training data — no labels, no pairing, no annotation. That is the reason this family scaled: the data supply is effectively the entire internet.

Scale, and what appeared with it

Somewhere around the GPT-3 scale, a behaviour appeared that nobody trained for: the model performs new tasks from examples placed in the prompt, with no weight updates at all.

Translate English to French:sea otter => loutre de mercheese => fromagepeppermint =>

The model completes menthe poivrée. Nothing was fine-tuned. This is in-context learning, and the honest explanation is that a sufficiently large model trained on enough text has seen enough patterns of the form "demonstration, demonstration, query" that continuing the pattern is next-token prediction.

It is worth being precise about the trade-off, because "just prompt it" has become a default and is often the wrong call:

T5: everything is text-to-text

T5 takes a different position: rather than a different head per task, cast every task as text in, text out, and use one encoder-decoder for all of them.

Input:  "translate English to German: That is good."Output: "Das ist gut."Input:  "cola sentence: The course is jumping well."Output: "not acceptable"Input:  "stsb sentence1: The rhino grazed. sentence2: A rhino is grazing."Output: "4.4"Input:  "summarize: state authorities dispatched emergency crews ..."Output: "six people hospitalised after a storm in Attala County."

Note that the similarity score comes out as the literal string 4.4. Even regression becomes text generation. The gain is uniformity: one model, one loss, one decoding procedure, and multi-task training for free.

Because it is an encoder-decoder, T5 gets bidirectional attention over the input and causal generation of the output — the properties of both other families, at the cost of roughly twice the parameters for a given depth.

Its pre-training objective is span corruption. Drop contiguous spans (15% of tokens, mean span length 3) and have the decoder emit them:

Original:  Thank you for inviting me to your party last week.Encoder:   Thank you <X> me to your party <Y> week.Decoder:   <X> for inviting <Y> last <Z>

Corrupting spans rather than single tokens is what makes this harder than BERT's objective and, importantly, produces a target that is much shorter than the input — the decoder only writes the missing pieces. For a 100-token input, 15 tokens are corrupted across roughly 5 spans, and the target is about 20 tokens including sentinels.

Sizes run from T5-small (60 M) through T5-base (220 M), T5-large (770 M) and T5-3B to T5-11B.

Using them

1from transformers import pipeline23# Sentiment — an encoder model with a classification head4clf = pipeline("sentiment-analysis",5               model="distilbert-base-uncased-finetuned-sst-2-english")6print(clf("The battery lasts two days but the camera is disappointing."))7# [{'label': 'NEGATIVE', 'score': 0.9994}]89# Named entities — an encoder with a per-token head10ner = pipeline("ner", model="dslim/bert-base-NER", aggregation_strategy="simple")11print(ner("Ada Lovelace worked with Charles Babbage in London."))12# [{'entity_group': 'PER', 'word': 'Ada Lovelace', ...},13#  {'entity_group': 'PER', 'word': 'Charles Babbage', ...},14#  {'entity_group': 'LOC', 'word': 'London', ...}]

The aggregation_strategy="simple" argument in the NER call is easy to skip and important. NER labels are per sub-word token, so Lovelace might be split into Love and ##lace with two separate predictions. Without aggregation you get fragments and inconsistent labels within one word.

1import torch2from transformers import AutoTokenizer, AutoModelForQuestionAnswering34# Extractive QA — the head predicts a start and an end token index5name = "distilbert-base-cased-distilled-squad"6qa_tok = AutoTokenizer.from_pretrained(name)7qa = AutoModelForQuestionAnswering.from_pretrained(name)89enc = qa_tok("Where was the conference?",10             "The 2019 conference was held in Vancouver over four days.",11             return_tensors="pt")12with torch.no_grad():13    out = qa(**enc)14start, end = out.start_logits.argmax(), out.end_logits.argmax()15print(qa_tok.decode(enc["input_ids"][0, start:end + 1]))16# Vancouver

Extractive QA is written out by hand because recent transformers 5.x releases no longer ship a question-answering pipeline (v5.0 had already removed the translation and summarisation pipelines); the model classes remain. The two argmaxes are the whole head: one picks the most likely first token of the answer, the other the most likely last token. Production code also checks that end >= start and caps the span length.

1from transformers import AutoTokenizer, AutoModelForCausalLM, AutoModelForSeq2SeqLM23# Generation — a causal decoder4tok = AutoTokenizer.from_pretrained("gpt2")5gpt = AutoModelForCausalLM.from_pretrained("gpt2")67ids = tok("The three laws of robotics state that", return_tensors="pt")8out = gpt.generate(**ids, max_new_tokens=50, do_sample=True,9                   temperature=0.8, top_p=0.9,10                   pad_token_id=tok.eos_token_id)11print(tok.decode(out[0], skip_special_tokens=True))1213# Translation — an encoder-decoder14t5_tok = AutoTokenizer.from_pretrained("t5-small")15t5 = AutoModelForSeq2SeqLM.from_pretrained("t5-small")1617ids = t5_tok("translate English to German: The house is small.", return_tensors="pt")18print(t5_tok.decode(t5.generate(**ids, max_new_tokens=40)[0],19                    skip_special_tokens=True))20# Das Haus ist klein.

Two generation parameters decide almost everything about output quality. temperature divides the logits before softmax: below 1 sharpens the distribution towards the most likely token, above 1 flattens it. top_p=0.9 is nucleus sampling — keep the smallest set of tokens whose probabilities sum to 0.9 and renormalise over just those, which cuts off the long tail of implausible tokens that would otherwise be sampled occasionally. Greedy decoding (do_sample=False) is correct for translation, where there is roughly one right answer, and produces repetitive text for open-ended generation.

Tokenisation, which differs between the families

None of these models operate on words. Words are too many — English has hundreds of thousands, plus every typo and proper noun — and characters are too many positions. Sub-word tokenisation splits the difference: common words stay whole, rare words break into pieces.

1from transformers import AutoTokenizer23bert = AutoTokenizer.from_pretrained("bert-base-uncased")4gpt2 = AutoTokenizer.from_pretrained("gpt2")56print(bert.tokenize("unbelievable tokenization"))7# ['unbelievable', 'token', '##ization']89print(gpt2.tokenize(" unbelievable tokenization"))10# ['Ġunbelievable', 'Ġtoken', 'ization']

Unbelievable is common enough to be one token in both vocabularies; tokenization is not, so both split it. The ## prefix marks a continuation of the previous word in WordPiece. The Ġ in GPT-2 marks a preceding space — which means "hello" and " hello" are different tokens. Prompts that begin with a trailing space in the wrong place produce measurably worse completions for exactly this reason.

A rule of thumb worth internalising: for English, one token averages about four characters, so 1,000 tokens is roughly 750 words. Code, non-Latin scripts and rare technical vocabulary tokenise far less efficiently — the same 750 words of Japanese or of chemistry nomenclature can take two or three times as many tokens.

The hard rule: always use the tokeniser that shipped with the checkpoint. Token IDs are arbitrary indices into a specific embedding table. Feed BERT's IDs to GPT-2 and every embedding lookup returns the vector for a different word. Nothing errors. The output is confident nonsense.

The transfer learning pipeline

The economics of this are the whole reason these models are usable. Pre-training BERT-base cost on the order of thousands of dollars of compute. Fine-tuning it on 4,000 tickets costs a few minutes on one GPU. You are buying language understanding that someone else paid for, and adding a 6,000-parameter head.

What this means when you build something

Choose the family from the shape of the output, then choose the size from your latency and data budget.

Four failure modes account for most of the trouble people hit:

Using a generative model for classification. Slower, more expensive, and usually less accurate than a fine-tuned encoder once you have a few thousand labels. Prompting is the right answer when you have almost no labelled data or the label set changes constantly; it is the wrong answer for a stable, high-volume, narrow task.

Mismatched tokenisers. Load the tokeniser and the model from the same checkpoint name, every time. This produces no error and no warning.

Silent truncation at 512 tokens. BERT's position table has exactly 512 rows. A 2,000-token document is cut to 512 and the rest is discarded without comment — often losing the part that carried the answer. For long documents either chunk with overlap and pool, or use an encoder built for length such as ModernBERT (8,192 tokens).

Ignoring the domain gap. A model pre-trained on web text and Wikipedia has poor coverage of clinical notes or legal filings — its tokeniser fragments the vocabulary badly and its representations were never shaped by that language. Domain-adapted checkpoints exist for the big verticals and are usually worth several points over the general-purpose equivalent.