Back to Blog
AILocal InferenceBenchmarksNVIDIA DGX Spark

Inkling 2 Mini on DGX Spark: More Agents with an FP8 KV Cache

5 min read

Inkling 2 Mini · Local inference

I switched Inkling 2 Mini’s KV cache from BF16 to FP8. The same memory budget held 47% more cached tokens, long-context decoding was faster, and I found no drop in accuracy on the tasks I tested.

Three-quarter view of the NVIDIA DGX Spark Founders Edition.
NVIDIA DGX Spark Founders Edition · Photo by Daniel Lu · CC BY-SA 4.0
+47%
Cached tokens at the same memory setting
48 → 61
Output tokens per second at the 256K tier, four requests at once
1 each
Wrong retrieval answer out of 8,200 questions per setting

An agent can start with a short request and end up carrying a stack of documents, tool results and earlier decisions. Run several such agents at once and a Spark that comfortably loads the model can still run short of room for their active context. I wanted to know whether a smaller cache would give me more room and whether the model would still find the right facts when I needed them.

The same memory budget held 47% more context

The key-value cache, or KV cache, stores attention data from tokens the model has already processed. Active requests draw on that working memory. For Inkling 2 Mini, the cache is especially relevant because its middle layers run three passes with shared weights, and each pass has its own cache.

How many tokens fit in the KV cache?

Measured vLLM cache capacity · same 0.40 memory fraction

BF16395,170
FP8580,576
That is room for 185,406 additional cached tokens. At 262,144 tokens per sequence, the pool goes from fitting one full sequence to fitting two by token count.

Both servers used the same checkpoint, including Inkling’s native mixed-precision weights. The change was what vLLM stored in the cache. FP8 uses half as many bits per stored value as BF16. The 47% gain is what vLLM actually reported for the cache pool.

The long answers came back faster

The FP8 decoding advantage grew from about 11% at the 64K tier to about 28% at the 256K tier. With four requests in flight at the longest tier, the server generated roughly 61 output tokens per second across the four responses, up from 48.

Output at the longest context tier

Aggregate decode throughput · output tokens/s

BF1648
FP861
Four concurrent retrieval requests at the 256K tier. The longest actual prompts in the growing-context test were around 240K tokens.

First-attempt misses at 256K

Strict scoring · share of retrieval questions

BF166.5%
FP84.4%
Strict scoring counts an answer that hits a limit as a miss. Timeouts accounted for the gap at this tier.

Reading the prompt was close. BF16 was about 4–7% faster at reading uncached input in these runs. The FP8 advantage showed up when generating the answer, particularly once the context was long.

Did the smaller cache cost accuracy?

I ran the same 8,200 retrieval questions on each setting. The prompts grew across ten synthetic operations logs. Each new block introduced facts among similar-looking decoys. After appending a block, I asked about older facts again to see whether they got harder to retrieve as the logs grew.

One wrong answer, on the same question

1 / 8,200BF16 · wrong answer
1 / 8,200FP8 · wrong answer
When a correct retry counted, the overall miss rate, including limits, was 1.1% for BF16 and 0.9% for FP8. The paired difference was statistically inconclusive (exact McNemar p = 0.26).

I also ran a 750-item text and vision suite. Both settings received the same generated items, two at a time, with exact answers scored by code. No task category showed a statistically significant difference. The suite covered objectively scored work such as reading, counting, ordering, maths and code. Open-ended writing quality was outside the test.

For an agent under a deadline, a correct answer that arrives too late is still a failed step. At the longest context tier, more FP8 answers arrived within the time limit. The wrong-answer tally stayed the same.

Benchmark setup and scoring

I ran one setting per NVIDIA DGX Spark (GB10), in parallel, with matched driver, kernel, Docker and container versions. Both served thinkingmachines/Inkling-2-Mini at revision ed40e7da with the same weights, vLLM’s inkling-2-mini image, DSpark speculative decoding and a 262,144-token limit. The baseline used --kv-cache-dtype auto, which resolved to BF16. The comparison used --kv-cache-dtype fp8.

Sampling used temperature zero with reasoning enabled. Retrieval used four in-flight requests and the task suite used two. A wrong answer, a timeout and a token-limit stop were recorded separately. Strict scoring used the first attempt. Retry-adjusted scoring credited a correct retry on a stopped answer.

Each configuration ran on its own Spark, so different machines could account for some of the timing difference. With only one wrong retrieval answer per setting, the tests leave room for smaller accuracy effects and for effects on other workloads.

What I’d run on one Spark

I’d use --kv-cache-dtype fp8 for this Inkling 2 Mini setup. For agents that may all reach the context ceiling at once, I’d size the cache from the sum of their peak contexts and leave room for the rest of the serving stack.

FP8 sizing estimates for fully occupied 256K contexts
Concurrent agentsEstimated memory budgetvLLM fraction
4 agentsAbout 74 GiBAbout 0.61
6 agentsAbout 97 GiBAbout 0.80

Those are planning estimates for one Spark. The four-request throughput run used a 0.40 memory fraction and asked questions about growing logs. Four requests in flight in that benchmark and four independent conversations that each fill a 256K context place different demands on the cache.

In this setup, FP8 gives me room to serve more active context and gets long answers out faster. The retrieval and task-suite results give me enough confidence to switch.

Niël Malan

The Practical Futurist