Google Cloud says firms need AI infrastructure upgrades

Google Cloud has published a report on AI infrastructure based on a survey of more than 1,400 Senior IT Leaders. The findings suggest many organisations are reworking their systems to support agentic AI.

The report says 83% of organisations need infrastructure upgrades to support production use of agentic AI, which it describes as systems that do more than respond to prompts and instead carry out tasks and workflows.

Google Cloud argues that this shift is exposing weaknesses in older computing setups built for earlier forms of AI. In the survey, 62% of leaders said they were facing a significant "inference tax" linked to data egress fees, storage growth and underused specialised hardware, while 81% pointed to operational complexity as a hidden cost of scaling AI.

These findings form part of a broader push by large cloud providers to present infrastructure as the main constraint on wider use of newer AI systems. Google Cloud says the pressure comes from workloads in which a single prompt can trigger many downstream actions and require large amounts of context to remain available in memory.

Compute shift

Organisations are moving towards what Google Cloud calls "fluid compute", using different processors for different stages of AI work. It points to TPU 8t for training, TPU 8i for low-latency inference, and Arm-based Axion CPUs for orchestration and reinforcement learning simulations.

That mix reflects a wider market pattern as cloud providers and chipmakers try to match specialist silicon to distinct AI workloads rather than rely on a single computing approach. Google Cloud also uses the report to underline its own stack across processors, networking, storage and orchestration software.

Governance concerns

The survey also highlighted management and oversight as major concerns as businesses deploy larger numbers of autonomous software agents. Google Cloud says 79% of Technology Leaders identified security, governance and MLOps as their main challenge in scaling inference.

The report describes a risk of "agent sprawl", in which large numbers of autonomous systems operate across different tools and platforms without central oversight. To address that, organisations are seeking a central control plane for permissions, identity and workflows, along with stronger audit trails and approval checks for sensitive actions.

It also says 78% of organisations now source generative AI solutions from their main cloud provider, 30 percentage points higher than in 2025. That suggests buyers may be favouring integrated platforms over combinations of separate vendors as governance demands become more complex.

Data and deployment

Another theme in the report is the need to bring together fragmented corporate data so AI systems can work across it more effectively. Google Cloud says a unified data layer is becoming more important as agents run repeated queries across a business and need access to both structured and unstructured information.

The report also points to hybrid multicloud as the prevailing architecture choice. It found that 52% of organisations now use a hybrid multicloud model, while 48% are prioritising infrastructure with strict data residency controls.

This reflects growing pressure on companies to decide where data and AI models should sit, especially in regions with tighter sovereignty rules. Businesses increasingly want the option to run some AI work in public cloud environments while keeping other workloads on local or isolated systems.

Edge and energy

Edge deployment emerged as another strong theme in the survey. Google Cloud says 90% of organisations ranked edge deployment as important for AI initiatives, and 72% described it as extremely or very important.

Companies are looking to edge systems to reduce latency, maintain operations when connectivity is disrupted and limit the cost of running constant AI workloads in central cloud environments. The report cites uses in settings such as manufacturing plants, retail stores and hospitals.

Power consumption is also moving up the priority list for infrastructure buyers. According to the survey, 91% of leaders now factor energy use into hardware selection, with 61% calling it a primary or significant factor.

Google Cloud links this to limits on electricity supply, tighter regulation and the rising cost of cooling and facility upgrades needed for more power-hungry AI hardware. It argues that performance per watt is becoming a more important measure as businesses assess the economics of larger AI deployments.

Unified systems

Google Cloud uses the report to make the case for more tightly integrated infrastructure, arguing that businesses face high operational overhead when they have to stitch together computing, storage and networking layers manually. It positions its AI Hypercomputer architecture as an example of a system built with custom silicon, networking, storage and orchestration components designed to work together.

The report also connects these infrastructure trends to the development of AI systems that interact with the physical world, including robots trained in digital simulations before being deployed in real environments. It says organisations that succeed with agentic AI will be those that combine cost control, governance, edge resilience and infrastructure designed for autonomous systems.