AI's bottleneck moves to chip packaging, optics and idle GPU capacity

For two years the short answer to "why can't we get more AI compute" was "not enough GPUs". This week's news makes that answer too simple. A Reuters analysis described advanced chip packaging as the next battleground, with spending on it heading towards about $28 billion. Marvell agreed to buy Lumentum, a maker of lasers and optical parts, at an implied enterprise value of $115 billion, a bet that light will replace copper for moving data between chips. And AWS made it possible to split one GPU into as many as seven isolated slices on SageMaker AI real-time endpoints.

These are one story at three levels: how chips are assembled, how they talk to each other, and how much of each chip you use. The first two shape what you can buy and when. The third you can act on on Monday.

What happened

Reuters: advanced packaging becomes the next battleground (6 November)

On 6 November Reuters published an analysis from Taipei by Wen-Yee Lee on how the AI boom is pushing money into advanced chip packaging, because demand for AI chips exceeds the capacity to assemble them. The article cites Yole Group, which forecasts capital spending on advanced packaging near $28 billion, and which expects advanced packaging revenue to overtake traditional packaging for the first time in 2026.

Key points from the report:

• TSMC dominates CoWoS packaging, used by Nvidia, AMD and Broadcom, and its capacity limits are pushing customers to other suppliers.
• Nvidia is reported to be financing CoWoS expansion at SPIL, a unit of ASE Technology, with billions of dollars. Nvidia did not comment.
• Powertech plans a $400 million panel-level packaging joint venture with Broadcom in Singapore, with production possibly from 2028. TSMC has a pilot line for its own panel technology, CoPoS.
• Former Intel chief executive Pat Gelsinger estimated packaging could be as much as half of a chip product's value. Some chips now integrate 10 or more dies, sometimes up to 20.

Marvell agrees to acquire Lumentum (2 November)

On 2 November Marvell and Lumentum announced a definitive agreement under which Marvell will acquire Lumentum in an all-stock transaction. The companies say:

• Implied enterprise value for Lumentum: $115 billion.
• Fixed exchange ratio: 4.500 Marvell shares for each Lumentum share, about $1,132 per Lumentum share.
• Premium: about 23.9%, based on unaffected closing prices on 28 October 2026.
• Ownership after the deal, fully diluted: about 67% Marvell stockholders, about 33% Lumentum stockholders.
• Expected close: first half of calendar 2028, subject to both sets of stockholders and regulators.
• More than $1 billion in annual run-rate cost synergies, and a combined addressable market the companies expect to reach $550 billion.

The stated rationale is that AI bandwidth demand and the limits of copper are pushing the industry towards optics. Marvell brings electrical and optical interconnect, switching and custom accelerators. Lumentum brings lasers, transceivers, optical circuit switching and compound semiconductor manufacturing.

Because the ratio is fixed, the value moves with Marvell's share price. The Motley Fool calculated roughly $81.2 billion at midday on 2 November, after both stocks fell.

AWS adds GPU partitioning to SageMaker AI real-time endpoints (30 October)

On 30 October AWS announced that SageMaker AI inference can split a GPU into isolated partitions using NVIDIA Multi-Instance GPU (MIG), so several models can share one GPU. According to AWS:

• A GPU can be divided into up to seven partitions, each with dedicated memory and compute.
• The target is small and mid-size models such as embeddings, rerankers and classifiers.
• Supported instances are P4d (A100 40 GB), P5 (H100 80 GB), G7 (RTX PRO 4500, 32 GB) and G7e (RTX PRO 6000, 96 GB).
• The documentation lists seven Regions: us-east-1, us-east-2, us-west-2, ap-southeast-1, ap-northeast-1, eu-central-1 and eu-west-1.
• You set the partition layout on the endpoint's production variant and a partition type on each inference component. Invocation does not change.

AWS added MIG to SageMaker HyperPod in November 2025. The new part is managed real-time endpoints.

How the hardware bottleneck works

Why packaging, not just chipmaking

A modern AI accelerator is not one chip. It is one or more compute dies, several stacks of high-bandwidth memory (HBM), and a layer underneath that wires them together. "Packaging" is building that assembly and connecting it to the board.

Think of a kitchen. You can buy the best oven in the world, but if the fridge is in the garage, the cook spends the evening walking back and forth. The fix is to put the fridge next to the oven with a very wide door. In a GPU, the compute die is the oven and HBM is the fridge. Large models need to read huge amounts of weights and cached values for every token, so the distance and width of that connection decide speed.

Three ideas make this work:

• HBM stacking. HBM is DRAM dies stacked on top of each other, like floors in a building, connected by vertical wires drilled through the silicon (through-silicon vias). Stacking gives a lot of memory in a small footprint with a very wide interface.
• The interposer. Ordinary circuit boards cannot carry thousands of fine wires between the GPU and its memory. So both sit on a thin slice of silicon, the interposer, which can be patterned with very dense wiring. It is a dedicated motorway between oven and fridge.
• CoWoS. TSMC's Chip-on-Wafer-on-Substrate places the dies on an interposer (chip on wafer), then mounts that assembly on a package substrate. It is the most widely used way to build these high-end assemblies.

Why this becomes the bottleneck: a single die cannot grow past the size limit of the lithography tools, so chipmakers add performance by putting more dies and HBM stacks side by side. That means bigger interposers, more steps and more yield risk. One bad bond can waste an expensive assembly. Packaging lines are specialised and slow to build, so they lag demand.

Panel-level packaging uses large rectangular panels instead of round wafers. Cutting rectangles from a round wafer wastes the edges, like cutting square biscuits from round dough. Panels waste less, but the dates in the Reuters report run to 2028.

Why optics matter for scale-out

Once a single accelerator is built, you need many of them to act as one. Training and large-scale inference split work across GPUs with tensor parallelism and other schemes, and those GPUs exchange data constantly. The network between them becomes part of the computer.

Copper is cheap and reliable over short distances, but as speeds rise its signals fade faster and need more power. At current rates it works inside a rack and struggles beyond it. Light in fibre loses very little over distance, but converting between electrical signals and light costs power, money and space.

Think of copper as a bicycle courier: fast across the office, hopeless across the city. Optics is a train line: excellent for distance, but it needs stations at each end. Today those stations are pluggable transceivers at the edge of a switch or network card. The direction of travel is to move them closer to the chip, eventually into the package (co-packaged optics).

That is why the Marvell and Lumentum announcement talks about connectivity from data centres down to "inside advanced semiconductor packages". Packaging and optics meet at the package. Optical circuit switching, one of Lumentum's lines, goes further by redirecting light with mirrors instead of converting it at every hop.

The practical lesson: cluster size is limited by power, packaging and interconnect, not only by GPU shipments, and large-cluster prices and lead times reflect all three.

How GPU sharing works

Three ways to share one GPU

Most production inference does not fill a modern GPU. An embedding model or a classifier might use a few gigabytes of memory and a fraction of the compute on an NVIDIA H100. Running one such model per GPU is like renting a whole bus to carry three passengers. There are three common ways to fill the seats.

Time-slicing. The GPU switches between processes in turns, like one chef cooking for several tables by rotating between them. It works on almost any GPU and needs no special setup. But there is no memory isolation, and each process gets the whole GPU in turn, so one heavy job makes everyone else wait. Latency becomes unpredictable.

MPS (Multi-Process Service). NVIDIA's MPS lets kernels from several processes run on the GPU at the same time, instead of taking turns. It is like letting several cooks share one kitchen at once. Throughput improves, but isolation is weak: processes share memory and a misbehaving one can affect the others.

MIG (Multi-Instance GPU). On supported data-centre GPUs such as the A100 and H100, MIG cuts the GPU into hardware-separated instances. Each instance gets its own compute slices, its own memory and its own memory bandwidth path. It is like building walls in the kitchen and giving each cook their own stove and fridge. A busy neighbour cannot steal your memory or slow your kernels beyond the slice you were given. The cost is rigidity: the slices come in fixed sizes, and you cannot resize a slice while work runs on it.

A short comparison:

How MIG profiles read

MIG profile names look cryptic but follow a pattern. In mig-1g.10gb, "1g" means one compute slice out of seven, and "10gb" means 10 GB of memory. On SageMaker's P5 (H100 80 GB), AWS lists profiles from mig-1g.10gb (seven per GPU) up to mig-7g.80gb (one per GPU, the whole card). On P4d (A100 40 GB) the smallest is mig-1g.5gb, also seven per GPU. On G7e the smallest is mig-1g.24gb with four per GPU, and on G7 it is mig-1g.16gb with two per GPU. So "up to seven" applies to P4d and P5. To get the count per instance, multiply by the number of GPUs: an ml.p4d.24xlarge has eight.

Utilisation economics in plain numbers

You pay for a GPU by the hour whether it works or not. The useful figure is not the hourly price but cost per useful unit of work: per thousand requests, per million tokens, per thousand embeddings.

A simple illustration, not a benchmark. Suppose you run seven small models, each on its own GPU, and each keeps its GPU about 10% busy. You pay for seven GPUs and get the work of less than one. If all seven fit on one GPU split into seven slices, and each still meets its latency target, you pay for one GPU. Your cost per request falls by up to seven times, minus the overheads of shared CPU, networking and host memory, which MIG does not partition.

The catch is the word "fit". A model that uses 10% of a GPU on average may need more than one seventh at peak. Averages hide peaks, so right-sizing must be measured under realistic traffic. The site's guide on cost and latency budgets covers how to set those targets first.

What it means in practice

Large-cluster supply stays tight for structural reasons. Packaging and optical capacity take years to add, and several dates in this week's news point to 2028. Plan big training runs and reserved capacity with that lag in mind.

The newest chips carry a packaging premium. Multi-die parts such as the NVIDIA B200 and its successors will stay expensive and allocated. For many inference workloads, an older GPU used well beats a new one used badly.

Interconnect is a design input. If your workload spans many GPUs, ask what the network is and at what bandwidth. Two clusters with the same GPU count can perform very differently.

Inference efficiency is where most teams can win now. You cannot speed up TSMC, but you can stop paying for idle silicon. The SageMaker change makes hardware-isolated sharing a configuration option rather than a Kubernetes project, though MIG has long been available on self-managed clusters.

Sharing changes observability. AWS states that partitioned endpoints do not publish GPU, CPU or memory utilisation metrics, so you monitor invocations and latency in CloudWatch. Autoscaling and alerts built on GPU dashboards need a new signal; see model monitoring.

Batching compounds with partitioning. For large language models, engines such as vLLM already raise utilisation with continuous batching and careful KV cache management. Partitioning helps most with the small models around the LLM: embedders, rerankers, classifiers and guardrails.

What to do now

1. Measure real utilisation. For each GPU service, record average and peak memory, compute use and requests per second over a normal week.
2. Sort models into bins. Whole GPU, a half or third, or a seventh. Embeddings, rerankers and classifiers usually land in the small bins.
3. Pick sharing by isolation need. MIG for production models with latency targets and different owners. MPS or time-slicing for development, batch jobs or one team's cooperative processes.
4. Load-test inside the slice. Run peak traffic on the slice you plan to use and check p95 and p99 latency, not just the mean.
5. Plan the layout up front. On SageMaker, a running endpoint's layout cannot change; an update with a different layout is rejected. Plan a blue-green swap to a new endpoint instead.
6. Move alerts to service metrics. Base autoscaling and alerts on invocations, latency and errors before you switch.
7. Check drivers. On P4d, mig-1g.10gb needs InferenceAmiVersion set to al2023-ami-sagemaker-inference-gpu-4-1, or partitioning can fail with an unclear error.
8. Shrink models before buying more. Quantisation, knowledge distillation and better batching can move a model into a smaller bin.
9. Ask about interconnect when buying capacity. Get topology and bandwidth in writing, not just GPU model and count.
10. Revisit capacity plans quarterly, with a plan B on a second instance family or provider.

Risks and open questions

Forecasts are forecasts. The $28 billion figure and the revenue crossover come from Yole Group via Reuters. Nvidia financing SPIL is reported, not confirmed.

The deal is not done. Marvell and Lumentum need stockholder and regulatory approval, with close expected in the first half of 2028. The $550 billion market and synergy figures are company projections.

Optics has limits. Lasers use power and fail. Co-packaged optics puts that failure inside an expensive package, and whether operators accept the trade at scale is open.

MIG is not free performance. A slice gets a fraction of bandwidth as well as compute, and host CPU, memory, storage and networking stay shared. AWS has not published latency figures for partitioned endpoints, and the missing utilisation metrics make capacity planning harder.

What to watch. TSMC's CoWoS and CoPoS capacity updates, regulatory filings on the deal, per-partition metrics from AWS, and whether other managed services follow.

Frequently asked questions

What is advanced packaging in one sentence?

Advanced packaging places several chips, such as compute dies and HBM stacks, very close together on a shared base with dense wiring, so they behave like one larger, faster chip. TSMC's CoWoS is the best-known example.

Why does memory need to sit so close to the GPU?

Large models read enormous amounts of data for every step. If the path to memory is long or narrow, compute units sit idle waiting. Stacked HBM on an interposer gives a wide, short path, which is why modern accelerators are fast and hard to build.

Will the Marvell and Lumentum deal change anything for my team soon?

Probably not. Close is expected in the first half of 2028. The trend it reflects, more optics in data-centre networks, is already shaping how providers build and price large clusters.

How is MIG different from just running two models on one GPU?

Without MIG, two models share memory and compute with no walls, so one can starve or crash the other. MIG gives each partition its own compute, memory and bandwidth in hardware. Latency is predictable, but slice sizes are fixed in advance.

Which models are good candidates for GPU partitioning?

AWS names embeddings, rerankers and classifiers: models that fit easily in a slice's memory with modest compute and steady traffic. Large language models usually do better on whole GPUs with an engine that batches well.

Does partitioning always save money?

No. It saves money when models were leaving most of a GPU idle and still meet latency targets inside a slice. If a model needs more than a slice at peak, or shared host resources become the limit, you lose more than you gain. Measure before and after.

Summary

The AI compute bottleneck is spreading outwards from the chip. Reuters reported that advanced packaging, led by TSMC's CoWoS, is drawing about $28 billion of investment according to Yole Group. Marvell agreed to buy Lumentum at an implied $115 billion, betting on optics replacing copper, with close expected in the first half of 2028. AWS made MIG partitioning, up to seven slices per GPU on P4d and P5, an option on SageMaker AI real-time endpoints. The first two shape what you can buy in coming years; the third shapes what you get from what you have. The fastest gain for most teams is to measure utilisation, pack small models onto shared GPUs with the right isolation, and monitor by latency. More is in the MLOps and AI infrastructure topic.

Sources

• Lumentum (SEC Form 8-K exhibit): Marvell and Lumentum to combine, creating a category leader in AI connectivity
• Marvell Investor Relations: Marvell and Lumentum to combine, creating a category leader in AI connectivity
• AWS What's New: Amazon SageMaker AI inference now supports GPU partitioning for real-time endpoints
• AWS Documentation: GPU partitioning for inference components
• AWS What's New: Amazon SageMaker HyperPod now supports NVIDIA Multi-Instance GPU (MIG) for generative AI tasks
• Reuters (via SRN News): AI boom turns chip packaging into the next battleground for semiconductors
• Light Reading: Marvell strikes $115B deal for Lumentum
• The Motley Fool: Why Lumentum stock fell 14.9% today