Data center operators should evaluate power and cooling infrastructure in the context of an expanding range of chips designed to support AI workloads.
Like the auto industry, where gas-powered vehicles have long dominated, conventional data centers housing largely x86 CPU servers for general workloads still command the largest market share. However, both industries are now dealing with increasing diversification in response to innovation and customer demand.
Data center operators are adapting to diversification beyond traditional CPUs to a greater range of AI accelerators that deliver superior performance for AI applications. That diversification is only set to increase.
Hyperscalers, enterprises, cloud providers, and colocation data center operators are increasingly deploying AI servers powered by a wider range of processors to handle compute-intensive AI workloads, from model training to inference. This shift is driving the need for more diverse and future-ready digital infrastructure.
Multiple categories of AI chips are driving this requirement for infrastructure optimization and flexibility:
GPUs
GPUs deliver the highest performance for resource-intensive AI applications. Hyperscalers, neo-clouds, and the largest enterprises are at the forefront of adoption. However, some operators face procurement challenges due to supply constraints and high costs. Originally designed as graphics cards, GPUs offer massive parallel processing capabilities, substantial core counts, and high bandwidth memory, making them well-suited for training large language models (LLMs). GPUs are also versatile and can serve double duty as inference chips, running models once they're trained and reach production.
Custom silicon
ARM architecture has enabled large data center operators to build their own custom application-specific integrated circuits (ASICs). While they use AI accelerators from major chipmakers, hyperscalers like Amazon Web Services, Microsoft Azure, and Google Cloud are developing their own custom AI chips for both training and inference to give customers more options. For example, Google designed its Tensor Processing Unit (TPU) with both performance and power efficiency in mind, delivering strong performance-per-watt. Beyond cloud providers, AI-focused companies are developing custom AI chips for their own operations. Meta has built an AI chip for its internal data centers, while OpenAI, which currently uses GPUs, recently announced plans for its own custom AI processors.
Inference chips
While the AI training market continues to scale at gigawatt pace, AI inference represents a potentially even larger market. To capitalize on this opportunity, established chip manufacturers and startups alike have developed chips designed specifically for AI inference, which is usually a lighter task than training and doesn't always require the power and performance of GPUs. These processors offer more energy-efficient alternatives for running AI applications.
Established chipmakers like Qualcomm have entered this space by developing ASIC-based inference chips that prioritize performance-per-watt efficiency for enterprise deployments.
CPUs
Traditional processors continue to play an important role in AI infrastructure. In AI servers, CPUs serve as the primary processors for managing overall system operations, while AI accelerators handle high-performance AI computations. On their own, CPUs can also handle some AI inference workloads, particularly in smaller-scale deployments. The type of chip organizations use for inference depends on the size and complexity of the workload. Some workloads need only a CPU. Others require a specialized inference chip or GPU.
Emerging chip architectures
Wafer-scale processors represent another approach and are already available for AI training and inference. Quantum chips and neuromorphic chips (which simulate how the brain functions) may also eventually impact AI in the data center. For now, these AI accelerators—GPUs, custom silicon, inference chips, and wafer-scale processors—are driving the immediate infrastructure changes that data center operators must address.
In the data center semiconductor market, a recent Futurum Group survey of decision-makers focused on AI data centers found that 75% of total compute spending goes to GPUs, followed by 13% for XPUs (dedicated AI accelerators) and 12% for CPUs.
Impact on data center power and cooling
With AI adoption growing, large data center operators continue to build AI factories — highly dense, GPU-powered facilities dedicated to AI workloads. Others, including enterprise data center operators, may look to retrofit existing sites to support AI, deploying a mix of GPUs, CPUs, custom ASICs, and inference chips for diverse AI applications.
The impact on data center infrastructure varies significantly depending on the specific chips deployed and the requirements of the operator. For example, one startup has designed low-power inference chips that allow data centers to largely use conventional power and cooling infrastructure. Other options include AI processors that deliver superior performance-per-watt compared to GPUs but also pack a large volume of accelerators within a rack, consuming more total energy and requiring higher-density PDUs and liquid cooling.
Operators may take alternative approaches to dealing with silicon diversification. One option is a high degree of specialization with power and cooling infrastructure tuned to specific AI servers. The other is a more future-ready approach to infrastructure that reflects the ten-year plus lifespan of a facility versus the replacement cycle for AI servers and silicon innovation roadmaps.
For AI deployments that require infrastructure upgrades, power demands can be substantial. Rack densities have transformed from six to 10kW in the recent past, to 140kW racks to support today's models; quickly advancing towards 600 plus kW racks and beyond.
High-density deployments often require evolved power and thermal architectures including forms of direct liquid cooling and higher-voltage DC power distribution systems. On the cooling front, data center operators deploying AI can choose between direct-to-chip liquid cooling, immersion cooling, and hybrid systems that combine liquid cooling with air cooling.
A diverse silicon mix ahead
While GPUs dominate AI workloads and are likely to continue powering large-scale training and inference applications, the processor landscape is diversifying rapidly. CPUs will maintain their foundational role in AI infrastructure, while ASICs and energy-efficient inference chips are gaining traction as alternatives for different AI workload requirements.
Just as the auto industry evolved from requiring gas stations to electric charging and hydrogen refueling, data center operators are transforming the digital infrastructure stack from grid to chip and from chip to heat reuse, to keep pace with processor innovation. Navigating the road ahead will require continuous innovation and an integrated partner ecosystem.
Silicon diversification is one of the macro forces reshaping data center infrastructure identified in Vertiv Frontiers.
