Arm (ARM) and Microsoft (MSFT) back Gimlet's $300 million bet on cheaper AI

Does splitting inference across chips pay?

Answering a prompt has two phases. Prefill reads the prompt, and it is heavy on raw compute. Decode writes the answer one token at a time, and generating each token means reading the model's weights, which makes decode sensitive to memory bandwidth. Gimlet's argument is that different chips may be better suited to each phase, and that running both on the same GPU leaves part of the chip idle in each phase.

In an analysis it published in October 2025, Gimlet benchmarked individual H100, B200 and Gaudi 3 accelerators and fed those measurements into performance models. It then modeled a configuration pairing an Nvidia (NVDA) B200 for prefill with an Intel (INTC) Gaudi 3 for decode, and compared it with running both phases on H100s and with running both on B200s.

The headline figure was a 1.7x improvement in modeled total cost of ownership. For a prompt-heavy workload of 4,096 input tokens and 512 output tokens, the paper reports a 3x cost benefit against the H100 baseline and says the mixed pair also beats the B200-only configuration. For an answer-heavy workload of 512 in and 4,096 out, it reports as high as 4x against H100s, while B200s alone modeled at about 2.5x on the same workload. Some of the gain against H100s reflects newer hardware, so the B200-only figure is the closer comparison, and the difference between 2.5x and 4x is what the model attributes to using Gaudi 3 for decode.

Those results come with qualifications. It's a model, not a deployed system, and the paper says its goal was to compare disaggregation strategies, so its results are relative multipliers rather than dollars per token. The split also has a cost of its own: after prefill finishes, the working memory called the KV cache has to move from one chip to the other. Gimlet's paper puts the format translation at 20 to 50 microseconds and the optimized transfer at 5 to 10 milliseconds, and it says "it's challenging to disaggregate the same workload across multivendor accelerator types" because the software stacks differ by vendor.

Some accelerators Gimlet is exploring for decode have limited onboard memory. In a March post on chips from Cerebras and d-Matrix, which keep model weights in fast on-chip SRAM instead of external memory, the company notes that even Cerebras's wafer-scale chip holds 44 gigabytes, "so weights for large models need to be distributed across multiple chips." Gimlet's answer is to use those chips for smaller draft models paired with a frontier model on GPUs. In the Chipstrat interview, the team said hardware amortization runs to about 70 percent of an inference provider's annual costs, and listed the hard parts as KV cache transfer between different systems, network latency between racks, and scheduling across several hardware platforms at once. The company lists Nvidia, AMD (AMD), Intel, Arm, Cerebras and d-Matrix as supported chip partners.

This is different from a router that sends whole requests to whichever machine is free, the design we described in Nvidia's PAIR tool last week. Gimlet's software divides one request between chips by phase and moves the request's state between them. The gain it claims comes from matching each phase to hardware built for it, not from concurrency.

What contracted revenue means

Contracted revenue generally refers to future business under customer agreements, but Gimlet hasn't disclosed its calculation. It cannot automatically be equated with accounting revenue, remaining performance obligations or another company's backlog. As an illustration only, a three-year contract worth $300 million would count as $300 million of contracted revenue on the day it is signed and, assuming service is delivered evenly over three years, as $100 million of revenue a year. We covered the same distinction when Nscale pitched $103 billion in contracted revenue ahead of its IPO.

Gimlet's own disclosures give the scale of the gap. When the company came out of stealth in October 2025, TechCrunch's Julie Bort reported it had eight-figure revenue, meaning at least $10 million, and 30 employees. In the May interview with Chipstrat, the team put annualized revenue at $10 million as of the March raise. The March release said the customer base had tripled and included "one of the top three frontier labs" and "one of the top three hyperscalers," neither named.

The contracted figure is at least 100 times Gimlet's last disclosed annualized revenue figure. That comparison illustrates scale rather than revenue growth: one is a yearly run rate, while the other spans undisclosed contract terms. Gimlet says the funding will support its cloud operations and team expansion.

Three things the announcement doesn't say: how long the contracts run, whether they carry cancellation rights, and how much of the contracted value depends on Gimlet delivering the hundreds of megawatts of capacity it says it is scaling toward.