Robot learning · Lesson 9 of 12

Action chunking and diffusion policies: ACT and beyond

Why robot policies predict action chunks, how ACT and diffusion policies work, training them with LeRobot, and reporting success rates with a Wilson interval.

advanced1 hUpdated Fact-checked Free

What you will learn

  • Explain how predicting chunks of k actions reduces compounding errors and smooths motion
  • Describe the ACT architecture: CVAE encoder, transformer encoder-decoder and temporal ensembling
  • Implement temporal ensembling with exponential weights and show its effect in NumPy
  • Explain why diffusion policies represent multimodal action distributions that MSE regression averages away
  • Evaluate a policy over N trials and report the success rate with a Wilson score interval

Before you start

  • Imitation learning (lesson 8)
  • Deep learning for robot perception (lesson 7)
  • Basic probability: Bernoulli trials and the normal distribution

After this lesson you can explain the two policy classes most often trained on low-cost arms today, ACT and diffusion policies, and the ideas behind them: predicting chunks of future actions, and modelling a distribution of actions instead of a single mean. You will implement temporal ensembling and a Wilson confidence interval in NumPy, and set up an evaluation that gives success rates you can defend.

Why predict chunks of actions

A single-step policy maps the current observation to the next action, π(at∣ot)\pi(a_t \mid o_t). Lesson 8 showed that its errors compound over the horizon TT. On an arm controlled at 30 to 50 Hz, a 10 s task is 300 to 500 decisions, each a chance to drift.

An action-chunking policy predicts the next kk actions at once, π(at:t+k∣ot)\pi(a_{t:t+k} \mid o_t), and executes them before it looks again. The ACT paper (Zhao et al., RSS 2023) describes three effects:

  • Shorter effective horizon. With one query every kk steps, a TT-step task becomes T/kT/k decisions, a kk-fold reduction.
  • Temporal consistency. The kk actions come out of one forward pass, so they form a coherent motion instead of independent guesses.
  • Pauses inside a chunk. Human demonstrations contain pauses and hesitations that depend on time, not on the observation. A single-step policy struggles to model them; inside a chunk the problem is mitigated.

The paper’s ablation, without temporal ensembling and averaged over four simulated settings, shows success rising from 1 % at k=1k = 1 to 44 % at k=100k = 100, then dipping slightly towards fully open-loop control at kk = 200 and 400. Chunking is not free: while a chunk plays, the policy ignores new observations, and switching chunks every kk steps causes jumps.

ACT: Action Chunking with Transformers

ACT was designed for low-cost teleoperated arms with a leader-follower setup, the same idea as the SO-101 pair. The paper’s tasks used four cameras and two arms, 14 joint values in total. Actions are absolute target joint positions; the authors report worse results with joint-position deltas.

A conditional VAE

Human demonstrations of the same situation differ. To model that variation, ACT is trained as a conditional variational autoencoder (CVAE) with a latent “style” variable zz.

  • The CVAE encoder is a transformer that reads a learned [CLS] token, the current joint positions and the demonstrated chunk at:t+ka_{t:t+k}. It outputs the mean and variance of a diagonal Gaussian over zz. Images are left out of the encoder to speed up training.
  • The CVAE decoder is the policy. It takes the images, the joint positions and zz, and predicts the kk-step chunk.
  • The loss is reconstruction plus β\beta times the KL divergence between the encoder’s Gaussian and N(0,I)\mathcal{N}(0, I). The text of the paper states an L1 reconstruction loss, which the authors found more precise than L2.
  • At test time the encoder is discarded and zz is set to the prior mean, zero.

Transformer encoder and decoder

Each camera image goes through a ResNet-18 backbone, which turns a 480×640480 \times 640 image into a 15×20×51215 \times 20 \times 512 feature map. Flattened, that is 300 tokens of dimension 512 per camera, 1200 for four cameras. With one token each for the joint positions and zz, the transformer encoder sees 1202 tokens and mixes information across cameras. The transformer decoder uses kk fixed position embeddings as queries, attends to the encoder output through cross-attention, and an MLP maps the k×512k \times 512 result to kk joint-position vectors.

The paper’s hyperparameters include k=100k = 100 and β=10\beta = 10. The model has about 80 M parameters, trained in about 5 hours on one RTX 2080 Ti, with about 0.01 s inference time on the same GPU.

Temporal ensembling

To keep chunking’s consistency without its jumps, ACT queries the policy at every step. Chunks then overlap: at time tt there are up to kk predictions for ata_t, made at times t−k+1,…,tt - k + 1, \dots, t. Algorithm 2 of the paper averages them with exponential weights:

at=∑iwiAt[i]∑iwi,wi=exp⁡(−m i)a_t = \frac{\sum_i w_i A_t[i]}{\sum_i w_i}, \qquad w_i = \exp(-m\, i)

where At[i]A_t[i] is the ii-th prediction for time tt and i=0i = 0 is the oldest. For m>0m > 0 the oldest prediction has the largest weight. The paper notes that a smaller mm means faster incorporation of new observations, and that the ensemble adds inference cost but no training cost. For 5 predictions and m=0.1m = 0.1 the normalised weights are 0.242,0.219,0.198,0.179,0.1620.242, 0.219, 0.198, 0.179, 0.162.

The implementation keeps a buffer of predictions per future time step:

class TemporalEnsembler:
    def __init__(self, m: float) -> None:
        self.m = m
        self.buffer: defaultdict[int, list[NDArray[np.float64]]] = defaultdict(list)

    def step(self, t: int, chunk: ArrayLike) -> NDArray[np.float64]:
        c = np.atleast_2d(np.asarray(chunk, dtype=float))
        for j, a in enumerate(c):
            self.buffer[t + j].append(a)  # appended in time order, so index 0 is the oldest
        preds = np.array(self.buffer.pop(t))
        return ensemble_weights(len(preds), self.m) @ preds

To see the effect, chunking.py replaces the network with a stand-in: each chunk is a target joint trajectory plus one random offset per chunk (standard deviation 0.03 rad) and small per-step noise (0.005 rad). This mimics a policy whose predictions are right in shape but slightly wrong in level each time it is queried.

execution                  RMS error  mean step  max step
target                        0.0000     0.0100    0.0157
open-loop chunks, k = 20      0.0258     0.0125    0.0962
ensemble, m = 0.01            0.0070     0.0100    0.0245
ensemble, m = 0.1             0.0079     0.0103    0.0239

Open-loop chunks jump by up to 0.096 rad at chunk boundaries, six times the largest step of the target. The ensemble averages up to 20 independent offsets, which cuts the RMS error by a factor of about 3.7 and removes the jumps.

Joint command in radians against time step from 20 to 120. A dashed target curve rises to 0.5 and falls. The amber open-loop chunk curve follows it with an offset that changes abruptly every 20 steps, with jumps up to about 0.1 rad at steps 60, 80 and 100. The blue temporal-ensemble curve lies almost on the target with no jumps.
Executing noisy 20-step action chunks open loop and with temporal ensembling (m = 0.01). Each open-loop chunk carries its own offset and switches abruptly at the chunk boundaries, marked by vertical lines; the ensemble averages overlapping predictions for each step and follows the target smoothly.

The toy flatters the ensemble in one way. Its offsets are independent between queries and older predictions are as accurate as new ones. On a real robot, a prediction made 99 steps ago used an old observation, so after a disturbance the ensemble reacts slowly; that is the role of mm. In LeRobot’s ACT configuration, temporal ensembling is controlled by temporal_ensemble_coeff. It defaults to off, requires n_action_steps = 1, and a comment in LeRobot’s code states that the original ACT implementation used 0.01. By default LeRobot’s ACT uses chunk_size = 100 and n_action_steps = 100, so it plays whole chunks.

In the paper’s ablation, temporal ensembling gave ACT a 3.3 % gain in success rate, much less than chunking itself.

Download chunking.py Download test_chunking.py

Multimodal actions: why the mean is wrong

Lesson 8 argued that mean-squared-error regression averages the modes of a multimodal action distribution. bimodal.py shows it with numbers. The observation xx is an obstacle size between 0 and 1; half the demonstrations pass on the left and half on the right, with lateral offset y=±(0.5+0.5x)y = \pm(0.5 + 0.5x) plus noise of 0.05. No demonstration comes closer than 0.3 to the centre line.

  • A cubic polynomial fitted by least squares predicts values between −0.06-0.06 and 0.080.08 everywhere. Every prediction lies inside the obstacle band.
  • A mixture of two linear-Gaussian components, fitted by expectation maximisation, recovers weights 0.49 and 0.51, slopes −0.49-0.49 and 0.510.51, intercepts −0.52-0.52 and 0.490.49, and spreads of 0.05. Of 1100 samples drawn from it, none falls in the band.
Scatter plot of action y against observation x. Grey demonstration points form two bands, one rising from 0.5 to 1.0 and one falling from -0.5 to -1.0, with an empty shaded band between -0.3 and 0.3 labelled as the obstacle. An amber MSE regression curve wiggles near zero inside the empty band. Blue samples from a two-component mixture lie on top of the two demonstration bands.
Bimodal demonstrations of passing an obstacle on either side. Mean-squared-error regression predicts the conditional mean, which runs through the obstacle where no demonstration went; sampling from a two-component mixture model produces actions in one mode or the other.
Download bimodal.py Download test_bimodal.py

A mixture with a fixed number of Gaussians is one fix. It needs the number of modes in advance and scales poorly to long, high-dimensional action chunks. ACT’s latent zz is another, although ACT decodes with z=0z = 0 at test time. Diffusion policies take a third route.

Diffusion policies

A diffusion policy (Chi et al., RSS 2023; extended version in IJRR, 2025) generates an action sequence by starting from noise and removing it step by step, conditioned on the observation.

Denoising as iterative refinement

Write AtkA_t^k for the action sequence after kk remaining denoising steps. Sampling starts with AtKA_t^K drawn from a standard Gaussian and applies, for k=K,…,1k = K, \dots, 1:

Atk−1=α(Atk−γ εθ(Ot,Atk,k)+N(0,σ2I))A_t^{k-1} = \alpha \left( A_t^k - \gamma\, \varepsilon_\theta(O_t, A_t^k, k) + \mathcal{N}(0, \sigma^2 I) \right)

The network εθ\varepsilon_\theta predicts the noise in its input. Subtracting it is a small gradient step towards actions that look like the demonstrations; the paper interprets εθ\varepsilon_\theta as predicting the gradient of an energy function. α\alpha, γ\gamma and σ\sigma depend on kk and form the noise schedule.

Training is simple regression on the noise. Take a demonstrated action sequence At0A_t^0 and its observation OtO_t, pick a random step kk, add noise εk\varepsilon^k of the matching size, and minimise

L=MSE(εk,  εθ(Ot,At0+εk,k))L = \text{MSE}\left(\varepsilon^k,\; \varepsilon_\theta(O_t, A_t^0 + \varepsilon^k, k)\right)

This is mean squared error again, but on the noise, not on the action. The network learns a direction field that points towards the demonstrations from anywhere in action space. Each sample starts from different random noise and slides into one of the modes. The paper attributes the multimodality to exactly these two sources, the random initialisation and the noise added during sampling, and shows a planar pushing task where the diffusion policy commits to going left or right in each rollout, while the baselines either favour one side or switch between them.

The design choices that make it a policy

  • Visual conditioning. The observation is an input to εθ\varepsilon_\theta, not part of what is generated, so the image encoder runs once per action sequence, not once per denoising step.
  • Receding horizon. The policy uses the last ToT_o observations, predicts TpT_p future actions, executes the first TaT_a of them, and then predicts again. This is the same chunking idea as ACT, with replanning built in.
  • Network. A 1D temporal CNN conditioned on the observation by FiLM layers, or a transformer. The paper recommends starting with the CNN and switching to the transformer for tasks with fast, sharp action changes.
  • Vision encoder. ResNet-18 without pretraining, with spatial softmax pooling instead of global average pooling and GroupNorm instead of BatchNorm, trained end to end.
  • Speed. Many denoising steps are slow. With DDIM, the paper trains with 100 steps and runs inference with 10, for 0.1 s latency on an RTX 3080 in its real-world experiments.

The paper also reports that position-control actions worked better than velocity actions with this policy class.

ACT or diffusion

Both predict chunks and both handle multimodal data better than plain regression. ACT needs one forward pass per query and is the policy the LeRobot documentation recommends starting with, citing fast training and good results with about 50 demonstrations. A diffusion policy needs several network evaluations per action sequence, but it represents multimodal action distributions directly. A reasonable plan is to train ACT first, look at the failures, and try a diffusion policy if they come from averaged modes.

Training with LeRobot

The commands below are from the LeRobot documentation and repository (v0.7.0). They were not run here. Training needs a GPU; the documentation shows --policy.device=cuda for NVIDIA cards and mentions mps for Apple silicon.

# From the LeRobot documentation, not run here.
lerobot-train \
  --dataset.repo_id=${HF_USER}/record-test_YYYYMMDD_HHMMSS \
  --policy.type=act \
  --output_dir=outputs/train/act_so101_test \
  --job_name=act_so101_test \
  --policy.device=cuda \
  --policy.repo_id=${HF_USER}/my_policy

Without --steps, training runs for the default 100 000 steps; the documentation notes that imitation learning typically converges within 5 to 10 epochs over the data, so a small dataset needs fewer. Checkpoints go to outputs/train/act_so101_test/checkpoints. The diffusion policy is registered in LeRobot under the type name diffusion, and its extra dependencies are installed with pip install 'lerobot[diffusion]'. Its configuration defaults in v0.7.0 are 2 observation steps, a prediction horizon of 64, 32 executed action steps and 100 DDPM training steps. Switch policies with --policy.type=diffusion and keep the rest of the command.

To evaluate on the arm and record every evaluation episode, the documentation’s episodic rollout strategy runs the policy for a fixed number of episodes with reset phases in between:

# From the LeRobot documentation (SO-100 example; use so101_follower and so101_leader), not run here.
lerobot-rollout \
    --strategy.type=episodic \
    --policy.path=${HF_USER}/my_policy \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --teleop.type=so100_leader \
    --teleop.port=/dev/ttyACM1 \
    --dataset.repo_id=${HF_USER}/rollout_eval_data \
    --dataset.num_episodes=20 \
    --dataset.episode_time_s=30 \
    --dataset.reset_time_s=10 \
    --dataset.single_task="Pick up the red cube"

Use the same --robot.id, camera names and task string as in the training data. In simulation, lerobot-eval with --policy.path, --env.type and --eval.n_episodes does the same job.

Evaluating a policy

A success rate from a handful of trials is a noisy number. Treat evaluation as a measurement with a protocol.

The protocol

  1. Define success before testing, in observable terms: “cube inside the bin and gripper open within 30 s”.
  2. Fix the initial conditions. Use a grid of object positions, for example 5 positions times 4 trials, and use the same grid for every policy you compare.
  3. Fix everything else. Same checkpoint, lighting, camera mounts and episode time limit. Randomise the order of trials so that slow drift (warming servos, changing daylight) does not favour one policy.
  4. Record every trial, including video, and log the outcome before looking at the next one.
  5. Log failure modes in fixed categories, for example: missed grasp, dropped object, collision, wrong object, timeout, frozen or oscillating. A table of failures by initial condition tells you what to fix: more data at a position, a camera change, or a different policy class.

The ACT paper used 50 trials per seed over 3 seeds in simulation and 25 trials on each real task.

A confidence interval for a success rate

nn independent trials with ss successes give the estimate p^=s/n\hat{p} = s/n. The textbook (Wald) interval p^±zp^(1−p^)/n\hat{p} \pm z\sqrt{\hat{p}(1-\hat{p})/n} fails at the sample sizes robot experiments use: it can extend past 1, and at 20 out of 20 it collapses to a single point. Use the Wilson score interval instead:

p^+z22n1+z2n  ±  z1+z2np^(1−p^)n+z24n2\frac{\hat{p} + \dfrac{z^2}{2n}}{1 + \dfrac{z^2}{n}} \;\pm\; \frac{z}{1 + \dfrac{z^2}{n}} \sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}

with z=1.960z = 1.960 for 95 %.

Worked example, 18 successes in 20 trials: p^=0.9\hat{p} = 0.9, z2=3.8416z^2 = 3.8416, z2/n=0.1921z^2/n = 0.1921.

  • Centre: (0.9+0.0960)/1.1921=0.8356(0.9 + 0.0960) / 1.1921 = 0.8356.
  • Half-width: (1.960/1.1921)0.0045+0.0024=1.6442×0.0831=0.1366(1.960 / 1.1921) \sqrt{0.0045 + 0.0024} = 1.6442 \times 0.0831 = 0.1366.
  • Interval: [0.699,0.972][0.699, 0.972].

The Wald interval for the same data is [0.769,1.031][0.769, 1.031]. For 20 out of 20, Wilson gives [0.839,1.000][0.839, 1.000]: a perfect run of 20 is still consistent with a true rate as low as 84 %. ACT’s reported 88 % on one real task, from 25 trials (22 successes), has a Wilson interval of [0.700,0.958][0.700, 0.958].

def wilson_interval(successes: int, n: int, confidence: float = 0.95) -> tuple[float, float]:
    _check(successes, n)
    z = z_value(confidence)
    p = successes / n
    denom = 1.0 + z**2 / n
    centre = (p + z**2 / (2 * n)) / denom
    half = (z / denom) * math.sqrt(p * (1 - p) / n + z**2 / (4 * n**2))
    return max(0.0, centre - half), min(1.0, centre + half)

The test file checks the function against scipy.stats.binomtest(...).proportion_ci(method="wilson") and confirms by simulation that it covers the true rate close to 95 % of the time for n=20n = 20, p=0.8p = 0.8.

Download stats.py Download test_stats.py
cd public/code/robotics-ai/act-and-diffusion
/Volumes/M2-Dev/conda/envs/toptech/bin/python -m pytest -q   # 18 passed

To plan the number of trials, use the approximate half-width zp(1−p)/nz\sqrt{p(1-p)/n}. For a rate near 80 % and a half-width of 0.1 you need about 1.962×0.16/0.01≈621.96^2 \times 0.16 / 0.01 \approx 62 trials; for 0.05, about 246. Comparisons between two policies need even more, which is why “18/20 versus 15/20” settles nothing (Wilson intervals [0.699,0.972][0.699, 0.972] and [0.531,0.888][0.531, 0.888]).

Exercise

Weights and intervals

  1. At some time step three predictions exist for one joint, oldest first: 0.30, 0.34 and 0.40 rad. Compute the ensembled action for m=0.01m = 0.01 and for m=1m = 1. Which one follows the newest prediction more closely?
  2. Compute the 95 % Wilson intervals for 7 of 10 and 35 of 50 successes. Both estimates are 0.7. What changed?
Show solution
  1. For m=0.01m = 0.01 the raw weights are 1,0.990,0.9801, 0.990, 0.980, normalised to 0.337,0.333,0.3300.337, 0.333, 0.330. The action is 0.337⋅0.30+0.333⋅0.34+0.330⋅0.40=0.3460.337 \cdot 0.30 + 0.333 \cdot 0.34 + 0.330 \cdot 0.40 = 0.346 rad. For m=1m = 1 the raw weights are 1,0.368,0.1351, 0.368, 0.135, normalised to 0.665,0.245,0.0900.665, 0.245, 0.090, giving 0.3190.319 rad. The small mm is closer to the newest value 0.40: it incorporates new observations faster, as the ACT paper states.
  2. 7/10 gives [0.397,0.892][0.397, 0.892] and 35/50 gives [0.562,0.809][0.562, 0.809]. Five times as many trials shrank the width from 0.495 to 0.247, roughly by 5\sqrt{5}. Check with wilson_interval(7, 10) and wilson_interval(35, 50).

Summary

  • Predicting a chunk of kk actions reduces the effective horizon kk-fold and gives temporally consistent motion; the ACT paper’s ablation went from 1 % success at k=1k = 1 to 44 % at k=100k = 100.
  • ACT is a CVAE: a transformer encoder of joints and the action chunk produces a style variable for training, and a ResNet-plus-transformer decoder predicts the chunk from images, joints and z=0z = 0 at test time.
  • Temporal ensembling queries every step and averages overlapping predictions with weights exp⁡(−mi)\exp(-m i), oldest first; smaller mm reacts faster to new observations.
  • MSE regression averages modes. A diffusion policy learns to denoise action sequences conditioned on the observation, and each sample commits to one mode; DDIM keeps inference fast enough for control.
  • Evaluate with a fixed protocol, log failure modes, and report successes with a Wilson interval: 18/20 is [0.699,0.972][0.699, 0.972], not “90 %”.

Check your understanding

Pick an answer for each question, then check. Answers stay hidden until you submit.

1. ACT predicts chunks of k = 100 actions and executes them without temporal ensembling. How many policy queries does a 400-step episode need?
2. In ACT's temporal ensemble the weights are w_i = exp(-m i), with w_0 for the oldest prediction. What does a larger m do?
3. What does the ACT CVAE encoder receive, and is it used at test time?
4. A policy succeeds in 18 of 20 trials. Which 95 % interval is the Wilson interval?
5. Policy A succeeds 18/20 times, policy B 15/20 times. What can you conclude?

Further reading

Found a mistake or something unclear?Report a correction.