The Hard Work Happens Before the Prompt

A friend sent me Andrej Karpathy's neural networks course last week. We spent an evening working through the first few lectures together. What struck me wasn't the content itself, embeddings and tokenization are familiar territory for most people building AI systems. What struck me was that Karpathy built a course teaching something that gets far less attention than prompt engineering.

Everyone wants to know how to prompt models. Karpathy teaches how to structure problems before the model sees them.

That distinction matters. Creating this kind of work, rigorous, accessible, free, elevates the entire field. We need more people doing it. It changes what the next generation of builders understands about the systems they inherit.

The Core Mechanism

Every machine learning system converts raw reality into something the model can use. Turn words into vectors. Split text into chunks the network can learn. Organize attributes so patterns show up. These are not details. They decide whether a hard problem becomes solvable.

How you framed the problem before training began matters more than what happens during inference.

Good encodings convert expensive search into cheap lookup. Bad ones force every downstream system to work around structural defects that should never have existed.

Beyond Prompts

The newest AI research makes this concrete. Representation engineering is an emerging approach that works with the internal structures models form. Instead of tuning what you ask the model, researchers are exploring how to intervene on the representations themselves, changing how concepts get encoded internally rather than just adjusting the outputs.

Prompts tune outputs. Encodings decide what outputs become possible.

The Pattern Everywhere

Software follows the same principle. Microservices and monoliths each make certain changes easy and others difficult. The structure decides how much teams must coordinate, how much operational overhead you absorb, and where coupling appears.

API design follows it. How you encode state transfers, as resources, as queries, as events, shapes what clients can request and what servers must maintain.

Organizations follow it. How you structure teams, around what people know or around which customers they serve, decides how deep their skills get, how clear their accountability is, and how much they must coordinate.

The Decision This Changes

Most technical leaders allocate effort where problems become visible. They tune prompts when outputs degrade. They add indexes when queries slow down. They refactor when coupling makes changes expensive.

That treats symptoms. The expensive work is redesigning how the problem is framed so it does not recur.

A startup building an AI product can iterate endlessly on prompts, or it can reconsider how customer intent is encoded before it reaches the model. The first produces incremental improvements. The second changes what the system can learn.

An org struggling with velocity can add process, or it can ask whether the service boundaries reflect actual product needs or the team structure that existed when the code was written. Process coordinates around bad structures. Good structures reduce how much coordination you need.

Where This Applies

This matters most when you control the encoding. If you are calling an external API, the structure is fixed and prompt engineering is often the only lever. If you are building the model, training the system, or designing the stack, changing how the problem is framed is where leverage concentrates.

It also matters more as scale increases. A poorly designed system is annoying when small. At scale, it decides whether anything finishes. Technical debt that looks like nuisance in a demo becomes the constraint that shapes every subsequent decision in production.

Small systems tolerate bad encodings. Large ones expose them.

You can think about framing now, when you still control the constraints, or later, when changing them requires migrating production systems and retraining expensive models.

What I Took From It

Working through even part of Karpathy's course clarified something I had been doing by feel for years. When a system feels brittle, the instinct is to add guardrails, tune parameters, refactor the interface. Those help. But the deeper question is whether the structure itself matches the problem.

In one lecture, he walks through how different ways of breaking up text change what a language model can learn efficiently. Byte-pair encoding versus character-level versus word-level strategies are not interchangeable configurations. They are different hypotheses about what units of meaning the model should operate on. That choice cascades through training cost, how well it generalizes, and what the model can eventually express.

That same pattern shows up in database schemas, service boundaries, team structures. Hard problems get easier when the encoding matches the structure of the problem.

Invest time understanding how your system encodes reality. Most problems that look like tuning failures are framing failures.

The hard work happens before the prompt.