Architecting an Army of AI Agents: Thinking Like a Software Architect

AI agents are fascinating tools—they’re autonomous systems powered by large language models (LLMs) like Gemini that can execute tasks on your behalf. As someone who’s always exploring new ways to optimize productivity, I thought: why not leverage AI agents as assistants to make my work (and life) easier? Eventually, I realized that building useful agents would require more than just writing code—it meant thinking like an architect.

At first, I was eager to dive in and start building AI agents immediately. With platforms like CrewAI (an agent orchestration tool) and low-code/no-code options like Gumloop or Langflow, it’s never been easier to scaffold an agentic project. But as I took my first steps—writing some Python code and configuring agents—I realized that the coding part was simple. The real challenge was in the planning, discovery, and research phases.

The hard part: asking the right questions

Here’s the truth: building effective AI agents isn’t just about throwing together some tools and scripts. It’s about deeply understanding the problem you’re solving, why it’s worth solving, and how to approach it iteratively. Without a solid plan, you can quickly hit roadblocks. I found myself asking questions like:

• What exactly do I want these agents to accomplish?

• How will they add value—whether professionally or personally?

• What context do they need to perform as well as (or better than) me?

Answering these questions took far more time than writing YAML or Python files. It wasn’t just about solving tasks. It was about reducing ambiguity and ensuring every step added clarity and purpose to the project. I spent hours researching use cases, studying community projects, and even sketching out workflows to understand how agents could integrate into both personal and enterprise environments.

The process reminded me of software architecture: before you code, you need a blueprint. You need to think systematically, asking the right questions to build something sustainable and effective.

Questions to guide your AI agent projects

If you’re considering building your own AI agents, I recommend starting with a discovery phase. Think of it as a “workflow audit.” Here are some key questions to guide you:

1. What tasks do I want to delegate to agents? Focus on tasks that would free up your time, spark creativity, or make your work more impactful.

2. What’s the simplest version of this task an agent could replicate? Start small and build iteratively.

3. What steps do I normally take to complete this task manually? Map out your workflow to identify how agents can replicate—or improve—your process.

4. What responsibilities will each agent have? Define precise roles for your agents to avoid overlaps or confusion.

5. How many agents do I need? Some tasks may require collaboration between multiple agents, while others might be handled by one.

6. What context do agents need to perform well? Consider whether your agents need access to examples of your work, domain-specific knowledge, or other inputs to meet your expectations.

7. How can I structure prompts to ensure high-quality output? Prompt design is critical for achieving the right results.

8. Do agents need to “talk” to each other? 

For more complex workflows, agents may need to collaborate to produce cohesive results. These questions aren’t just theoretical—they’re a framework for turning vague ideas into actionable solutions. Your answers will help you build AI agents that aren’t just functional but genuinely useful.

Re-engineering workflow automation

One of the most insightful lessons I’ve learned in this journey is that human workflows don’t directly translate to agentic workflows. At a recent workshop on “Trustworthy, Responsible AI & Governance” at the Harvard Data Science Initiative, someone made a thought-provoking point:

This perspective changed how I approached building AI agents. Instead of trying to force agents into workflows designed for humans, I started thinking about how workflows could be reimagined to leverage the strengths of AI.

For example, I’ve been experimenting with creating an “army” of Developer Relations agents—each with specialized roles like content creation, community management, and performance analysis. I don’t want these agents to just execute tasks; I want them to measure and report on impact, helping me quantify value in ways that weren’t possible before.

Build big, think iteratively

The beauty of AI agents is their flexibility. You can start small—perhaps with a single agent automating a repetitive task—and gradually scale up to more ambitious projects. For me, this iterative approach has been key. I’m constantly refining my ideas, learning from experiments, and updating my vision of what an ideal system of agents could look like.

Here’s a diagram of what I initially came up with for my dream army of Developer Relations AI agents:

So, what would your ideal “army” of AI agents do for you? With you as the human manager in the loop, the possibilities are endless. But no matter how ambitious your vision is, remember to start with the fundamentals: think like an architect, ask the right questions, and design workflows that play to the strengths of agentic AI for your optimal benefit, however you see fit.