The AI Investment Thesis is Shifting: Training to Inference

TLDR:

The AI investment thesis is shifting from training to inference: from “who can train the best model” to “who serves tokens cheapest at scale”
The question now is who can produce tokens the cheapest and who can produce the most tokens.
Memory (HBM) and vertical integration are the biggest value chain pivots from training to inference. Advanced packaging, power, and networking still remain crucial as well.
Highest-beta plays show up in memory suppliers like SK Hynix, Micron, Samsung; custom silicon enablers like Broadcom; and hyperscalers with advanced vertical integration like Alphabet and Amazon.

The Shift: Training to Inference

Where the market has been

Over the past ~24 months, markets have been centered around the question: “How good can a model get?” — hence the hype around GPUs and Nvidia, that they’ve overlooked the point of training models — inference.

Most people only notice their interactions with the original and most basic form of AI — the “chatbot” — that they fail to understand where else “inference” shows up in their daily lives.

Where the trend is going

The AI trade is shifting: The question is no longer just “how good can a model get?”. Increasingly it’s about “How can we serve this intelligence at scale?”

This is the shift from training to inference:

Explosion of AI use cases: AI is becoming embedded into everything that we use — search, productivity, customer support, shopping, coding, with use cases continuing to proliferate.
Agentic workflows / “Test-time Compute”: Models are using exponentially more tokens than they did before
- Legacy models: user query → model response
- Current models: User query → model planning → multi-step reasoning → tool calls → verification → iteration → model response
User Growth: User count is growing rapidly, and real products require continuous serving, not just training.

Where’s the difference / What’s the mispricing?

In training, hyperscalers were constrained by power and thus we optimize for performance (FLOPS as a proxy) per watt, putting Nvidia’s GPUs in the spotlight. We were happy to pay Nvidia’s margins because that meant faster training time and better models.

Training is episodic and like CapEx → Spending more means better output.

Inference, however, is continuous. It’s like COGS, it’s driven by unit economics. There is no “spend more to get better output”. That doesn’t make sense because the thing we want to optimize for IS cost. A token is a token. The model’s capability is fixed. You’re not competing for “inference capability” — that’s not a thing. You’re competing on margins.

So now the investment thesis is moving towards:

Who can produce the most tokens
Who can produce tokens the cheapest
Who enables points 1 and 2

i.e. Who has highest beta to $/token and tokens/day

What are the investable opportunities?

Memory: LLM inference is bound by memory bandwidth, not compute. Longer contexts + more users + more use cases = more memory demand.

Advanced Packaging: Physically assembling the hardware by combining memory with logic dies. All “AI hardware” passes through this funnel.

Power / Thermal: How many MW can you deliver to a data center + how well can you remove heat at ultra-high power density.

Interconnect / Networking: How much data can you move and how fast can you move it? Longer contexts, more tokens, smart routing all drive greater need for more and better communication.

Vertical Integration / Custom Silicon: Inference is COGS-like so the name of the game becomes increasing margins. Hyperscalers have an incentive to vertically integrate wherever unit economics justify the investment. These show up in:

Custom ASICs (Application Specific Integrated Circuits): The hardware optimized for inference differs from the hardware designed for training.
Custom/More Interconnect: Custom silicon often increases the demand for networking (more disaggregation, more scale-out)
+ All original bottlenecks: Packaging, power, memory still apply

Opportunity	Constraint / Demand Rank	Market Underappreciation Rank
Memory	5/5	3/5 – widely discussed
Advanced Packaging	4/5	2.5/5 – widely discussed
Interconnect	3/5 – not the biggest constraint for inference	3/5 – not as discussed, but not a big opportunity
Power	5/5 – Increasingly more of a constraint	4/5 – Increasingly discussed but energy crisis not fully understood
Vertical Integration / Custom Silicon	5/5 – not a ‘first-order’ constraint, more of a solution	4/5 – some chatter but market is underestimating the pace

Some comments:

Power, Packaging, and Interconnect have been in the spotlight for a while now. They were bottlenecks in training and continue to be for inference as well.
Custom Silicon + Vertical Integration & Memory are the biggest pivots in the value chain from training to inference. Thus, these will be the focus for the rest of this post.
As a side note, Custom Silicon + Vertical Integration and Power are where I believe to be the least discussed relative to their growth profile, and thus continue to be where I’m most interested.

Who are the players?

Alphabet – Hyperscaler segment, advanced vertical integration, custom TPUs, not to mention their other businesses + leading AI model Gemini.
Amazon – Similar vertical integration story (Trainium, Inferentia). Not to mention their quiet dominance in physical AI – robotics, automation, distribution, etc.
Meta, Microsoft – Similar hyperscaler dynamics but less advanced custom silicon programs.
Broadcom, Marvell – Enablers of vertical integration / custom silicon. Partner with hyperscalers to design custom ASICs + Interconnect businesses.
SK Hynix, Micron – Memory manufacturers, direct exposure to THE binding constraint.

I’ve just mentioned a few names on my watchlist that are high-beta particularly to inference – memory & vertical integration players. There are plenty more names in these areas. I also haven’t managed any power, packaging, or interconnect names, but those have been in the spotlight for a while now.

Validation & Invalidation

Given our theses, what questions do we need to answer?

General Inference Demand Growth

Inference Capex Growth – Is inference capex actually outpacing training? How is the allocation shifting?
Are enterprises increasingly adopting AI? Are agentic workflows driving exponential token demand?

Vertical Integration

Custom ASIC CapEx Allocation – Are hyperscalers actually shifting CapEx to custom silicon?
Nvidia Inference Revenue Guidance – Is Nvidia losing share in inference?
Broadcom Custom ASIC Design TAM/Revenue – Are custom ASICs delivering material margin improvement for hyperscalers?

Memory

Memory has been in the spotlight for some time now but it’s important to continue looking at HBM pricing, lead times, supplier commentary, and developing technology that may shift memory demand.

Catalysts

Where do we find these answers?

Earnings

Hyperscalers (Google, Amazon, Meta, Microsoft):

AI CapEx guidance and composition (training vs inference vs networking)
Mentions of custom silicon
Commentary around unit economics – listen for repeated or emphasized metrics: e.g. $/token language that inference is increasingly a margin optimization problem

Nvidia:

Inference revenue and % of revenue mix
Revenue growth (signals if custom ASICs are taking share or not)
Gross margin guidance (compression = custom silicon taking share)

Custom Silicon Enablers (Broadcom, Marvell):

Revenue growth, hyperscaler partnerships/customers

Memory Suppliers (SK Hynix, Micron):

HBM pricing, lead times, demand allocation

Conferences

Nvidia GTC, Google Cloud Next, Computex, etc.

Roadmap updates, technology developments, partnerships/deals: Hints at what is in demand, what is the stack trying to optimize for next, is the value chain shifting

Risks

What non-insignificant events fundamentally shift the value chain?

High demand elasticity: Users/enterprises opt for “good enough” cheap models and unwilling to pay significantly more for advanced capability. Open-source (Llama, DeepSeek, Qwen) captures majority of inference workloads.

This is simultaneously a “China dumping” story. Although China doesn’t have the best models, they’re producing strong models for significantly cheaper, and have the potential to flood the market with said, cheap models. I fear this may be a possibility in 2026.

Step-function in model efficiency: Models become more efficient, lowering the expected demand for compute/memory.

China dumping hardware: Right now, China is doubling down on its domestic HBM and chip technology. If they make significant progress, this would not only lower China’s demand for US tech, but also poses a dumping risk.

Geopolitical Supply Disruption: Taiwan strait tensions, export control restrictions, tariffs, or other events that disrupt semiconductor supply chains.

TSMC concentration is the key vulnerability.

Power buildout lag: Delays in power infrastructure, restrictions on AI / Data Center buildouts.

Source: jan 23 2026 post.pdf (published Jan 23, 2026). Return Factor’s reference deep-dive — the house style in action: TLDR → the shift → the mispricing → opportunities (with the constraint/underappreciation table) → players → validation → catalysts → risks.