Course Content
AI Safety & Guardrails
5 sections · 50 lessons
What challenges do policymakers face when regulating AI?
What you need to know
The challenges
- Pace: a law drafted around one generation of technology may be outdated when it takes effect. The EU AI Act added general-purpose model rules late in drafting because large chatbots appeared mid-process.
- Defining the object: model-level rules miss harms that depend on use; application-level rules miss general-purpose systems. Compute thresholds (for example 10^25 FLOPs) are simple to apply but become less meaningful as training gets more efficient.
- Measurement: benchmarks are contaminated, easy to optimise for, and don't match real-world harm. Rules often fall back to process requirements (document, test, monitor) rather than outcome requirements.
- Jurisdiction: a model trained in one country is used everywhere, which makes unilateral rules partly unenforceable and creates regulatory arbitrage.
- Capacity: regulators rely on disclosures from the companies they regulate.
- Open weights: once released they cannot be recalled or patched, which breaks the normal product-safety model.
- Allocating responsibility: harm often comes from a combination — a base model, a fine-tune, an app and a user's request.
- Balancing innovation and protection: strict rules can entrench large companies that can afford compliance.
Common responses
| Response | Example |
|---|---|
| Risk-based tiers | EU AI Act |
| Voluntary frameworks and standards | NIST AI RMF, ISO/IEC 42001 |
| Regulatory sandboxes | Supervised testing space with regulators (EU member states must set these up under the AI Act) |
| Transparency and audit duties | Model documentation, bias audits for hiring tools |
| Duties on deployers, not just developers | Human oversight duties for users of high-risk systems |
| Sector regulators applying existing law | Financial regulators, data protection authorities, consumer protection |
| Government AI safety institutes | Pre-deployment testing of frontier models in the UK, US and elsewhere |
India's approach so far has leaned on existing laws, sector regulators and published guidelines rather than a single AI act.
Why an engineer should care
Rules will keep changing. The things that stay valuable across every regime are traceability (what ran, on what data, with what evidence), evaluations and human oversight you can demonstrate.
A real-life example
A health-tech company in India builds a symptom-checker used in India, the EU and the US. In India, no AI-specific law applies, but DPDP and medical-device rules might; in the EU, the AI Act and medical-device regulation may make it high-risk; in the US, it may fall under FDA oversight depending on its claims.
Rather than building three compliance tracks, the company builds one evidence base: versioned models and prompts, clinical evaluation reports, a risk-management file, decision logs and a documented human escalation path. When the EU's timeline shifts, the product plan does not, because the capabilities were built for all markets at once.
Follow-up questions to expect
- "Should AI be regulated at the model level or the application level?" — Mostly at the application level, where harm depends on context, with some duties on the most capable general models because their risks spread to every application.
- "Do voluntary frameworks work?" — They create a shared vocabulary and often become de facto requirements through procurement and contracts; enforcement is weaker than law.
- "What would you tell a regulator is measurable today?" — Process evidence (evals run, red-team results, incident logs) and specific outcome metrics like selection-rate ratios in hiring; not a general "safety score".