Building Enterprise Infrastructure for AI Agents

As the line blurs between large language models (LLMs) and AI agents, it’s a great time for enterprises to start thinking seriously and holistically about their AI infrastructure. If the last few years were defined by large AI labs delivering huge models to the world, the next few years will be defined by how well everyone else can take those models and run with them.

It is difficult to predict exactly how things will play out, but one safe bet is that AI agents and agentic workloads will take off. It’s already happening at the application level for workloads like coding, where agents execute increasingly complex tasks from simple prompts. Assuming a cooperative ecosystem, agents are set to proliferate on the web for everything from booking reservations to buying products directly from applications like ChatGPT.

Note: The Model Context Protocol (MCP) is a leading indicator of the agentic activity to come. Think of MCP as the AI analog of open APIs—it opens a new path for software products to communicate and for vendors to partner.

Startups and enterprises alike are moving toward a future where workflows won’t require a human catalyst. An event will trigger a function, which triggers a team of agents to parse documents, analyze data, or warn public safety officials of threats caught on sensors.

However, maximizing the potential of this new world requires a reinvestment in enterprise infrastructure and data architecture. At some point, the assumptions we’ve been relying on for decades have to break.

---

From Zero to Agents: The Shift to Inference

Before digging into the ideal infrastructure, let’s look at the recent history of AI to see how we got here. The evolution of AI capabilities has moved rapidly from simple classification to complex, autonomous behaviour.

The Rise of Inference-Time Scaling

Beyond the speed of evolution, the biggest shift in AI has been from a focus on training to a focus on inference.

• Training: Requires huge, predictable amounts of data and computing resources. Labs know exactly how much infrastructure they need and for how long.
• Inference: Far trickier from an infrastructure perspective. While individual tasks are small, they add up fast. Requests are often batched to increase GPU efficiency, creating a tradeoff between throughput and latency.

It is inference that drives the massive compute deals between AI labs and cloud providers. It is not just the volume of users straining the infrastructure; it is the type of workloads. Today’s reasoning models rely on inference-time scaling. They spend more computing resources testing possible options and generating detailed responses to mitigate the diminishing returns of pre-training scaling laws.

The Agentic Multiplier

Agentic workflows resemble reasoning models but involve more complexity:

1. External Calls: Frequent calls to APIs, MCP servers, and other agents.
2. Persistence: Agents must maintain context across long sessions.
3. Memory: Longer native context windows require storing larger token volumes in memory (KVcache), impacting bandwidth and GPU performance.

---

Challenges for Enterprise AI Systems

Most organizations lack an infrastructure foundation optimized for running AI agents at an operational scale. The problem is that AI agents were born into a world where infrastructure was designed for previous eras of computing.

Even cloud-hosted models have limitations regarding state maintenance and data freshness. Large enterprises, especially in government, regulated industries, or mission-critical sectors need to manage their own infrastructure to ensure:

• Low Latency: Crucial for real-time decision-making.
• Data Sovereignty: Protecting highly sensitive data.
• Cost Efficiency: Utilizing a Mixture of Experts (MoE) approach to intelligently route jobs to the most cost-effective model for the task, rather than always calling the most expensive frontier model.

The Multimodal Data Wrinkle

The advent of video-language models (VLMs) adds significant complexity.

• Traditional LLMs: Retrieval involves chunking text and storing lightweight embeddings in a vector database.
• VLMs: Each chunk requires multimodal embedding (textual cues, image features, video descriptors). This requires significantly more storage capacity and GPU resources.

Why Legacy Infrastructure Breaks

If expert predictions are correct, organisations may soon run millions of agents. Using legacy components to power these workloads results in “spaghetti” pipelines: disparate tools for event processing, stream processing, function calls, storage, and databases.

A DIY system comprised of legacy components often results in an unmanageable architecture:

The Risks of Complexity:

• Operational: The system risks “death by a thousand cuts” due to siloed components.
• Governance: Reproducibility of errors and traceability of interactions becomes nearly impossible.
• Performance: Reasoning is slowed down by forcing access to data through a collection of disjointed systems.

---

The Agentic Operating System

What does the ideal enterprise infrastructure for running AI agents look like? While the field is evolving, specific requirements for production-grade AI agent infrastructure are becoming clear.

Consider a public safety scenario: A system monitoring cameras to identify dangerous activity as it unfolds.

1. Ingest: Agents constantly extract and summarise video frames.
2. Act: Based on pre-defined policies, agents flag authorities immediately or request human oversight.
3. Audit: Every action is logged for troubleshooting and refinement.

To support this, infrastructure must support capabilities specifically for building, managing, and orchestrating AI agents. Providing these features natively allows users to deploy agents at a massive scale while ensuring they deliver on enterprise requirements around performance, governance, and security.

These agents won’t just be performing one-off coding jobs or making restaurant reservations. They will be integral parts of every enterprise workflow, always learning, remembering, and improving. The time to start investing in the foundation for this future is now.