AI's next bottleneck is not computing power: system integration and workflow orchestration become the new battlefield

Introduction

In the past few years, the core challenge for the AI industry has been whether it can rapidly build enough computing power to meet demand. The rise of large language models has triggered an unprecedented infrastructure race, with NVIDIA becoming the center of the AI ecosystem, hyperscale cloud providers expanding data centers at a pace rarely seen in history, and AI-native cloud providers serving organizations unable to access computing power through traditional channels. However, as AI deployment matures, the industry is facing a new bottleneck: shifting from acquiring AI infrastructure to effectively utilizing it.

This article will analyze the profound impact of this shift on the semiconductor supply chain, technology roadmaps, market competition dynamics, and regional supply chains.

Background

Tirias Research predicts that LLM inference volume will grow from 990 trillion tokens in 2024 to over 1 quadrillion tokens by 2030, while image and video generation is also surging. But the key point is that AI workloads are evolving from Wave 1 (conversational assistants) to Wave 2 (autonomous agents). Autonomous agents require reasoning, tool invocation, context maintenance, multi-step task execution, and often run continuously rather than as single responses. It is estimated that Wave 2 users consume an average of 40 times more tokens than Wave 1 users.

Computing power remains fundamental, but it is no longer sufficient. Success increasingly depends on systems that connect models, tools, data, and actions into reliable workflows. NVIDIA CEO Jensen Huang recently described agentic AI as another "turning point" the industry has reached.

Deep Dive

Technology Impact

The focus of technology roadmaps is shifting from single-model performance to system-level integration. AI cloud platforms are no longer just providing GPUs and storage but are beginning to encapsulate complete workflows—including models, search engines, vector databases, observability tools, agent frameworks, security controls, and orchestration layers. For example, Nebius launched "Agents Blueprint," extending the concept of infrastructure as code to AI workflows, allowing users to reuse validated system templates.

The technology barriers lie in: cross-layer integration capabilities, reliability of workflow orchestration, balancing low latency with high throughput, and security in multi-tenant environments. These soft skills will differentiate more than sheer computing density.

Supply Chain ImpactThis shift has a multi-dimensional impact on the supply chain: - Upstream chips: Continued demand for GPUs, but the demand pattern shifts from training bursts to sustained inference. Increased demand for high-performance networking chips (e.g., NVIDIA ConnectX, Broadcom Tomahawk) as agents frequently invoke external services. - Storage & memory: Continuously running agents need to maintain context, driving demand for high-bandwidth memory (HBM) and large-capacity storage. - Cooling & power supply: Sustained inference leads to more uniform power distribution in data centers, but total power consumption rises, placing higher demands on liquid cooling and power management chips. - Cloud platform software stack: Open-source projects (e.g., Kubernetes, Ray, LangChain) and commercial platforms (e.g., Nebius, Google Vertex AI) become core components, reshaping the software ecosystem.

Beneficiaries: Cloud service providers with full-stack optimization capabilities (e.g., AWS, Azure, GCP), system integrators, and startups offering observability and orchestration tools. At risk: Cloud service providers offering only bare-metal GPUs, and semiconductor companies overly reliant on training compute demand.

Competitive Landscape

• The competitive landscape is being reshaped:
• NVIDIA: Still at the core of AI computing, but agentic AI requires more complex networking and storage, which may weaken its single-GPU dominance. NVIDIA has laid out a full-stack strategy through acquisitions (e.g., Mellanox) and software (CUDA, AI Enterprise), but faces challenges from AMD, Intel, and custom ASICs (e.g., Google TPU, Amazon Trainium).
• Cloud giants: AWS, Microsoft Azure, and Google Cloud gain advantages with proprietary chips and platform services (e.g., Amazon Bedrock, Azure AI, Vertex AI). AI-native clouds (e.g., Nebius, CoreWeave) compete through workflow differentiation.
• Chip design: Agent AI requires lower latency for inference, driving the development of edge AI chips and inference-specific chips (e.g., Groq, Cerebras).
• EDA/IP: Increased system design complexity places new demands on simulation and verification capabilities of EDA tools (e.g., Synopsys, Cadence).

• Market share adjustment may accelerate: NVIDIA’s share in the inference market may be challenged, but its position in training remains solid; proprietary chip share on cloud platforms will grow; the AI cloud service market is concentrating among top players.- United States: Hyperscale cloud providers and NVIDIA dominate, but system integration capabilities rely on Silicon Valley's software ecosystem. AI-native clouds like Nebius (although registered in the Netherlands, its core team is in Israel/US) are expected to rise.
• China: Affected by export controls, domestic AI chips (Huawei Ascend, Cambricon) need breakthroughs in system software, while workflow platforms (e.g., Baidu AI Cloud, Alibaba Cloud PAI) are accelerating. However, global supply chain participation is declining.
• Taiwan, China: TSMC's advanced process demand continues, but the system integration trend has an indirect impact on wafer manufacturing. Design companies like MediaTek need to focus on the AP requirements of Agent AI.
• South Korea: Samsung has advantages in memory (HBM) and foundry, but system software capabilities are weak.
• Japan: Rapidus is laying out advanced packaging, but AI cloud platform construction lags behind.
• Europe: Companies represented by Nebius have made breakthroughs in AI cloud platform innovation, but the industry scale is limited. The EU AI Act may affect workflow compliance.
• Southeast Asia: Has become a hotspot for cloud data center locations, but local chip design capabilities are weak.

Investment Perspective

Long-term, capital markets focus on system-level value. After the GPU boom, investors are looking for the next growth point—software companies such as AI workflow platforms, observability tools, and Agent frameworks will have high valuations. On the hardware side, networking and storage chip companies benefit. Amid geopolitical risks, diversified supply chains are also favored.

Long-Term Outlook

• Within 3 years: AI cloud platforms shift from infrastructure to workflow-as-a-service, with large enterprises adopting Agent workflows. NVIDIA launches more system-level solutions.
• Within 5 years: Agent AI becomes mainstream, with token consumption growing exponentially. The demand structure for chips changes: the share of inference chips and networking chips increases.
• Within 10 years: AI system integration may spawn new chip architectures (such as in-memory computing, reconfigurable computing), and the software abstraction layer further sinks.

Industry Chain Analysis

Upstream: Chips and Equipment

Agent AI places balanced demands on computing, storage, and networking chips. GPUs remain the mainstay, but NPUs and LPUs focused on inference are emerging. Demand for 3nm/2nm advanced processes continues, but packaging technologies (such as CoWoS, 3D stacking) are crucial for memory bandwidth.

Midstream: Cloud Platforms and Software Stack

AI cloud service providers (AWS, Azure, GCP, Nebius) become core integrators. They provide full-stack solutions from model libraries to orchestration services. Workflow integration lowers the barrier for enterprise AI applications but increases reliance on platform stickiness.### Downstream: Industry Applications Enterprise users do not need to build their own AI systems; they can directly purchase workflow services. Differentiation relies on private data, domain fine-tuning, and security compliance. This accelerates AI adoption, but also shifts the technical barrier to data and industry knowledge.

Conclusion

The next bottleneck for AI is not computing power, but system integration and workflow orchestration capabilities. This shift will reshape the value distribution of the industrial chain: from hardware to software, from individual chips to system architecture. For chip companies, merely providing high-performance computing is no longer enough; they must deeply collaborate with cloud platforms to offer better total cost of ownership and integration efficiency. For investors, attention should be paid to AI cloud platforms with system-level solutions, network storage chips, and orchestration software. In the global competitive landscape, the United States still has a first-mover advantage, but Europe and China have opportunities in niche areas.