Harmful content: multimodal fusion and training under extreme imbalance

Harm frequently exists only in the combination of modalities, which is what makes this problem architecturally different from everything before it. The first half of this lesson is about where text, image, video and audio should meet inside the model.

The second half is training, which has three specific problems: an imbalance so extreme that ordinary loss functions fail, eight categories of wildly different frequency, and an evaluation that must not be reported in aggregate.

Why combination matters

Three invented examples that no single-modality model can catch:

• An ordinary photograph of a person, with text naming them and inviting others to find them. The image is benign. The text is benign in isolation. Together it is targeted harassment.
• A screenshot of a chart, with a caption making a dangerous medical claim the chart does not support. Neither element alone violates anything.
• A video of a public street, with audio containing a threat.

Run separate text and image classifiers and score each independently, and every one of these passes. The harm lives in the relationship between modalities, so the model must see the relationship.

Three fusion strategies

Early fusion. Convert each modality to tokens or patches and feed everything into one joint model from the start, so the modalities interact through every layer.

• Strengths: captures fine-grained cross-modal relationships. The only strategy that reliably catches the harassment example above.
• Weaknesses: expensive to train and serve; needs paired multimodal training data; a missing modality (text-only post) needs explicit handling; effectively impossible to explain a decision — you cannot tell a user which part of their post was the problem.

Late fusion. Run an independent model per modality, then combine their outputs — with a weighted average, or a small model over the per-modality scores.

• Strengths: each model is trained and debugged separately; per-modality data can be used independently; a missing modality degrades gracefully; a decision is explainable ("the image classifier scored 0.94"); each component can be updated on its own schedule.
• Weaknesses: cross-modal harm is invisible by construction. Combining 0.1 and 0.1 cannot produce 0.9 unless the combiner has seen the raw content, which it has not.

Hybrid fusion. Encode each modality separately with its own encoder, then combine the resulting embeddings — not the scores — in a joint model with cross-attention or concatenation, and predict from there.

• Strengths: encoders are pretrained and reusable; the joint layer sees enough to model cross-modal interaction; substantially cheaper than full early fusion; per-modality encoders are still separately debuggable.
• Weaknesses: less expressive than early fusion; still needs paired training data for the joint layer.

The recommendation: hybrid fusion. Pretrained per-modality encoders, a joint cross-attention layer, and eight output heads. It captures the cross-modal cases, keeps the encoders reusable across other systems on the platform, and preserves enough separability that a per-modality score can be surfaced to a reviewer.

Explainability is a real requirement here, not a nice-to-have. A reviewer deciding an appeal needs to know why the system flagged the post, and a user told their post was removed deserves better than "the model said so". Keeping per-modality scores available alongside the joint prediction is worth a small accuracy cost.

Training: the loss under extreme imbalance

With the architecture chosen, the first training problem is the base rate.

With positives at 0.2% and some categories at 0.001%, ordinary binary cross-entropy is dominated by the negative class exactly as in the Street View training lesson.

Three tools, and the answer is usually a combination:

Class weighting. Multiply positive examples' loss by a constant, often the inverse of the class frequency. Simple, and it makes gradients noisy, because a handful of positives per batch now carry enormous weight and one mislabelled positive can swing an update.

Negative downsampling. Keep every positive, sample negatives to 1:20. Training becomes cheap and the batch composition sane. It shifts the output probabilities, requiring the calibration correction described in the ad click data lesson.

Focal loss. As explained in the Street View training lesson: multiply each example's loss by (1 − p)^γ so confidently-correct examples contribute almost nothing. Here it does something specific and valuable — the vast majority of negatives are trivially benign, and focal loss removes them from the gradient without discarding them from the data.

The caution repeats: focal loss amplifies label noise, because a mislabelled example is indistinguishable from a hard one. With hate-speech labels at 71% inter-annotator agreement, that is a real risk. Use a moderate γ (around 1.5 rather than 2 or 3) on the low-agreement categories, and consider training those on soft labels, which naturally reduces the weight of contested examples.

Recommended combination: negative downsampling to roughly 1:20 for tractability, focal loss with a moderate γ for the residual imbalance, and per-category loss weights so rare categories are not ignored.

Multi-task learning across categories

Train one model with eight output heads rather than eight separate models.

Why it helps. The categories share structure. Recognising that an image contains a weapon helps with graphic violence, threats, and regulated goods. Recognising aggressive language helps with hate speech and harassment. A rare category with 4,000 positive examples cannot learn a good image representation alone; sharing a trunk with a category that has 400,000 lets it borrow one.

The gain is largest exactly where it is most needed — the rarest categories.

Where it hurts: negative transfer. If one category dominates the loss, the shared trunk optimises for it and the others degrade. With spam at 40× the volume of self-harm, the trunk will become a spam detector unless you intervene.

Mitigations, in the order to try them:

1. Weight the per-head losses inversely to category frequency, so no head dominates.
2. Sample batches to balance categories rather than sampling uniformly from the corpus.
3. Give each head capacity of its own — a couple of task-specific layers before the output — so category-specific patterns need not fit in the shared trunk.
4. Split the group if two categories genuinely conflict. Nothing requires all eight to share one trunk; two models over four categories each is a legitimate answer.

Watch for negative transfer by comparing each head against a single-task model trained on that category alone. If the multi-task head is worse, the sharing is hurting that category and the weighting needs work.

Evaluating per category

Aggregate metrics are actively misleading here. This case study has already said so once, in the metrics warning, and it bears repeating because it is the most common mistake in this problem.

An invented evaluation:

A volume-weighted aggregate over this table reports something around 0.75 and is a fiction. The system is decent at spam and nudity, weak at hate speech and harassment, and effectively untested on child safety.

Three consequences that belong in the design:

• Report every category separately, always. A release gate per category, not on the aggregate.
• Categories with too few positives to evaluate need a different approach — human review of everything above a very low threshold, plus hash matching, rather than a statistical claim.
• Report per region and per language too. A model trained mostly on English will perform worse on other languages, and the aggregate will not show it. This is a fairness requirement with a direct consequence: users posting in under-served languages get both more of their legitimate content removed and more harmful content left up.

Splits

Split by time (Step 5: training) — train on earlier months, test on later ones — because harm types evolve and a random split flatters the model on campaigns it has already memorised.

Add a split by author so the same account does not appear in both sets. Prolific violating accounts post hundreds of near-identical items, and an author leak inflates recall badly.