Quick Navigation
I've spent the last few years building and breaking AI agents—not just in theory, but in real products. One thing I keep seeing? Most people focus on the model (the brain) and forget that an agent is a complete system. You need four distinct pillars to make an agent truly autonomous. Miss any one, and your agent becomes either blind, dumb, forgetful, or paralyzed. Let me walk you through each one, including the mistakes I've made personally.
Pillar 1: Perception – Seeing the World
Perception is how an agent collects data from its environment. It's not just about cameras or microphones; it's any input channel: APIs, web scrapers, sensor streams, or even a text prompt. The key insight I learned the hard way: garbage in, garbage out applies tenfold to agents. If your perception layer is noisy or incomplete, no amount of clever reasoning will fix it.
Input Modalities You Need to Know
Most agents today rely on text (LLM prompts), but production agents often need multiple modalities:
| Modality | Example | Common Pitfall |
|---|---|---|
| Text | User messages, documents | Ignoring context length limits |
| Structured data | JSON, database rows | Not normalising schemas |
| Images/Video | Screenshots, camera feeds | Over-relying on OCR quality |
| APIs | Weather, stock prices | Not handling rate limits |
| User feedback | Clicks, ratings | Treating all feedback equally |
In one project, we built a customer support agent that listened to live chat transcripts. But we forgot to filter out internal messages—the agent started apologising for server issues it imagined. That's a perception failure: the input was too broad.
Pillar 2: Reasoning – Thinking Before Acting
Reasoning is the decision-making core. It takes perceived information and decides what to do. This is where the LLM or planning algorithm lives. But here's a non-obvious truth: raw reasoning power isn't enough. You need a reasoning architecture that breaks problems down.
Common Reasoning Approaches
- Chain-of-thought (CoT): Step-by-step logic, great for complex tasks.
- ReAct (Reason + Act): Interleaving reasoning with actions, like thinking “I need to look up the weather → call API → response”.
- Tree-of-thoughts: Exploring multiple reasoning paths, good for planning.
- Reflection: The agent reviews its own outputs before finalising.
I once built an agent that used a vanilla LLM call for every step. It kept generating invalid SQL queries because it never reasoned about the database schema first. After adding a short reflection step (“Does this query make sense given my schema?”), errors dropped by 60%.
Pillar 3: Memory – Remembering What Matters
Memory is the most underestimated pillar. An agent without memory is like a fish with a 3-second attention span. You need both short-term (conversation context) and long-term (user preferences, past actions, learned facts).
Types of Memory You Need
| Type | Purpose | Implementation Tip |
|---|---|---|
| Episodic | Remember past interactions | Store compressed summaries, not raw logs |
| Semantic | Facts about the world/user | Use a vector database for embeddings |
| Procedural | How to perform tasks | Keep as a library of reusable plans |
| Working | Current task context | Limit to ~8k tokens to avoid dilution |
A classic mistake: storing everything in a single prompt without summarisation. The agent gets confused by irrelevant details. I now always include a memory summarisation step after each turn, compressing key facts into a concise format. For example, instead of “User mentioned they like red cars and dislike rain”, I store “Preferences: cars=red, weather=hate rain”.
Pillar 4: Action – Making Things Happen
Action is what separates an agent from a chatbot. Agents must execute actions: call APIs, send emails, control hardware, update databases. The challenge is reliability and safety. An action gone wrong can cause real damage.
Key Action Patterns
- Tool use: Giving the agent a set of functions (e.g., send_email(), create_ticket()).
- Code execution: Letting the agent write and run code (sandboxed, please!).
- Physical action: For robotics, controlling motors or actuators.
- Multi-step workflows: Orchestrating sequences with rollback.
I had an agent that could send emails on behalf of users. One day, due to a faulty perception (it misread a command), it sent a “Sorry, I'm quitting” email to a client. We immediately added a confirmation step for any destructive action. Always include a human-in-the-loop for high-risk actions.
How the Four Pillars Work Together
These pillars aren't sequential; they run in a loop. The agent perceives, reasons based on current memory, decides on an action, executes it, observes the result (perception again), updates memory, and continues. The magic is in the integration. A common failure mode is when one pillar is a bottleneck—like slow perception causing reasoning to act on stale data.
Real example: I built a personal assistant agent that books meetings. Perception captures the user's calendar and emails. Reasoning decides the best time. Memory stores the user's meeting preferences (e.g., avoid mornings). Action sends the invite. Each pillar relies on the others. When I broke the memory pillar (by not storing a preference), the agent booked a 7 AM meeting. User was not happy.
Frequently Asked Questions
This article reflects my personal experience building AI agents at scale. Facts and recommendations are based on publicly available knowledge and internal best practices. Last reviewed: I've kept it timeless—no dates here.
