DSpark: DeepSeek's new speculative decoding work (Video + Blog)

English Video Link: https://www.youtube.com/watch?v=iluBkgFl53E
Chinese Video Link: https://www.youtube.com/watch?v=diorl2YLuO8

0. What is speculative decoding

• Drafting: a small drafter model proposes $\gamma$ tokens quickly
• Verification: all $\gamma$ tokens are fed into the target model in one forward pass (similar to chunked prefill)

1. Autoregressive vs Parallel Drafting

Autoregressive

• Representative works: Deepseek MTP, Eagle1/2/3/3.1

DeepSeek MTP (Dec 2024): introduced in DeepSeek-V3, originally for training

Eagle1 (Jan 2024)

• inference optimization
• next-feature prediction + static tree

Parallel

• Representative works: Medusa, DFlash

Medusa (2024)

DFlash

• Inputs: the embedding of a bonus (anchor) token followed by $\gamma$ mask token embeddings

• Output: logits for all mask positions

• Context Feature Fusion: hidden states uniformly sampled from the target model → fuse them using projection

• KV injection: features from the target model are injected directly into the Key/Value projections of every draft model layer and stored in the KV cache

• Bidirectional attention within a block

2. Speedup formula

Per-token latency:

$$
\frac{T_\text{draft}+T_\text{verify}}{\tau} (\text{token/s})
$$

DSpark has 3 levers:

• T_draft: parallel drafting
• $\tau$: lightweight sequential module
• T_verify: dynamic verification

3. DSpark architecture

• Inputs: prompt tokens ABC + bonus (anchor) token D
• D as the input → a heavy parallel backbone and a lightweight sequential head → draft tokens EFGH + confidence scores c1 - c4
• Hardware-Aware Prefix Scheduler: evaluates c1 - c4 scores to retain the prefix EFG and drop HThe
• The target model verifies the scheduled prefix in parallel

4. Semi-Autoregressive Generation

Parallel stage

• Use DFlash backbone
• A minor modification: treat the anchor itself as the first prediction position, so $\gamma$ input tokens (anchor + $\gamma-1$ masks) yield 𝛾 draft logits

Sequential stage

• Prefix-dependent transition bias: $B_k(x_0, x_{<k}, x_k)$
  • x0: anchor token
  • x_{<k}: previous draft
  • x_k: current draft
• probability distribution

• $U_k(v)$: base logit vector produced by the parallel backbone at position k

Two instantiations of the sequential block

• Markov head
  • first-order transition $B(x_{k-1},x_k)$
  • In principle, this is a full $V\times V$ matrix $B$
  • In practice, we use a low-rank factorization $B = W_1W_2$
    • W1 shape = [V, r], W2 shape = [r, V]. By default, r=256
    • W1: embedding lookup table
    • W2: logits projection
• RNN head: maintaining a recurrent state that accumulates the full prefix history within a block

5. Confidence-Scheduled Verification

• The confidence head outputs a score c_k for each draft position k
• c_k is the conditional probability that draft 1 to k-1 are accepted
• How to generate c_k? linear projection → sigmod
• Inputs
  • backbone hidden state at position k
  • Markov Embedding from the previous draft token
• Supervise c_k using the analytical acceptance

Sequential Temperature Scaling (STC)

• Neural confidence estimates are often overconfident
• Calibrate the joint probability consecutively from left to right
  • At each position 𝑘 ∈ {1, . . . , 𝛾 }, we perform a simple 1D grid search to find the optimal temperature scalar that minimizes the Expected Calibration Error (ECE) of the cumulative product, keeping the already-calibrated scores of all preceding positions fixed

6. Hardware-Aware Prefix Scheduler

• Existing work: a static confidence threshold
• T_verify depends on system loads (batch size, context length, etc.)
• Formulate as a global throughput maximization problem

7. Training

• cross-entropy loss to predict the correct next token
• distribution matching loss between target and draft distributions
• confidence loss

All three losses are position-weighted (emphasize earlier block positions)

8. Why Can Parallel be Better?

Architecture difference

• Eagle3: one layer due to slow drafting
• DSpark & DFlash: five layers

9. Drafter Depth

Drafter length

• more layers = larger $\tau$
• 1L to 2L has the largest gain
• 2-layer DSpark outperforms 5-layer DFlash

10. $\tau$ vs Proposal Length

DSpark vs DFlash: performance gap steadily widens as 𝛾 increases

11. $T_{draft}$ vs Proposal Length