Robot learning · Lesson 10 of 12

Reinforcement learning for robots: Q-learning to PPO

MDPs and the Bellman equation, tabular Q-learning in NumPy, PPO's clipped objective, reward design for reaching, and why most robot RL runs in simulation.

intermediate55 minUpdated Fact-checked Free

What you will learn

  • Write a robot task as a Markov decision process with states, actions, rewards and a discount factor
  • Derive the Bellman optimality equation and solve a small MDP with value iteration
  • Implement tabular Q-learning with epsilon-greedy exploration and check it against value iteration
  • Explain the policy-gradient idea and the PPO clipped surrogate objective
  • Design a reaching reward, spot reward hacking, and decide between RL, imitation learning or both

Before you start

  • Python and NumPy
  • Probability: expectations and conditional probability
  • Imitation learning (lesson 8)

After this lesson you can write a robot task as a Markov decision process, solve a small one exactly with value iteration, and learn the same solution from trial and error with tabular Q-learning. You will also know how the method scales to neural networks and continuous arm commands through policy gradients and PPO, why a reaching reward is easy to get wrong, and why almost all robot reinforcement learning (RL) runs in simulation.

Learning from reward instead of demonstrations

In imitation learning the robot copies recorded actions. It can never do better than the demonstrator, and it has no signal for states the demonstrator never visited. RL replaces the demonstrations with a scalar reward that scores each step. The robot tries actions, observes the reward, and adjusts its behaviour to collect more reward over time. The upside is that RL can find behaviour nobody demonstrated. The cost is that it needs many trials, and the reward must describe the task exactly.

Markov decision processes

RL problems are written as a Markov decision process (MDP), a tuple (S,A,p,r,γ)(\mathcal{S}, \mathcal{A}, p, r, \gamma):

  • States s∈Ss \in \mathcal{S}: everything the robot needs to choose an action. For the SO-101 reaching a point, that could be the six joint angles, their velocities and the target position.
  • Actions a∈Aa \in \mathcal{A}: what the policy outputs, for example six goal positions for the servos.
  • Transition model p(s′∣s,a)p(s' \mid s, a): the probability of the next state. The physics of the arm defines it, and the learner usually does not know it.
  • Reward r(s,a,s′)r(s, a, s'): a number for each transition.
  • Discount γ∈[0,1)\gamma \in [0, 1): how much a reward one step later is worth compared with now.

“Markov” means the next state depends only on the current state and action, not on the earlier history. If the robot only sees joint angles and not velocities, a fast-moving and a resting arm look the same and the Markov property fails. That is why velocities, or a short stack of past frames, often go into the state.

The agent maximises the return, the discounted sum of future rewards:

Gt=rt+1+γrt+2+γ2rt+3+⋯=∑k=0∞γkrt+k+1G_t = r_{t+1} + \gamma r_{t+2} + \gamma^2 r_{t+3} + \dots = \sum_{k=0}^{\infty} \gamma^k r_{t+k+1}

With γ=0.95\gamma = 0.95, a reward 7 steps away is weighted by 0.957≈0.700.95^7 \approx 0.70. The effective horizon is about 1/(1−γ)=201/(1-\gamma) = 20 steps. A policy π(a∣s)\pi(a \mid s) maps states to (probabilities of) actions.

Value functions and the Bellman equation

The state value of a policy is its expected return from state ss, and the action value is the expected return after taking action aa in ss and following π\pi afterwards:

Vπ(s)=Eπ[Gt∣st=s],Qπ(s,a)=Eπ[Gt∣st=s,at=a]V^\pi(s) = \mathbb{E}_\pi\left[G_t \mid s_t = s\right], \qquad Q^\pi(s, a) = \mathbb{E}_\pi\left[G_t \mid s_t = s, a_t = a\right]

Split the return into the first reward plus the discounted rest, Gt=rt+1+γGt+1G_t = r_{t+1} + \gamma G_{t+1}, and the definition becomes recursive. This is the Bellman expectation equation:

Vπ(s)=∑aπ(a∣s)∑s′p(s′∣s,a)[r(s,a,s′)+γVπ(s′)]V^\pi(s) = \sum_a \pi(a \mid s) \sum_{s'} p(s' \mid s, a)\left[r(s, a, s') + \gamma V^\pi(s')\right]

The optimal policy picks the best action in every state, which replaces the average over actions with a maximum. This is the Bellman optimality equation:

V∗(s)=max⁡a∑s′p(s′∣s,a)[r(s,a,s′)+γV∗(s′)],Q∗(s,a)=∑s′p(s′∣s,a)[r+γmax⁡a′Q∗(s′,a′)]V^*(s) = \max_a \sum_{s'} p(s' \mid s, a)\left[r(s, a, s') + \gamma V^*(s')\right], \qquad Q^*(s, a) = \sum_{s'} p(s' \mid s, a)\left[r + \gamma \max_{a'} Q^*(s', a')\right]

Once you have Q∗Q^*, the optimal policy is greedy: π∗(s)=arg⁡max⁡aQ∗(s,a)\pi^*(s) = \arg\max_a Q^*(s, a).

Value iteration turns the optimality equation into an update. Start with V=0V = 0 and apply the right-hand side to every state until nothing changes. For γ<1\gamma \lt 1 the update is a contraction: each sweep shrinks the largest error by at least a factor γ\gamma, so it converges to V∗V^* from any start. Value iteration needs the model pp, which you have for a grid world and do not have for a real arm.

A grid world as a reaching task

A 5 by 5 grid gives a discrete version of reaching. The gripper starts at S and must reach the target G. H cells are obstacles that end the episode with a penalty, and # cells are walls. Each move costs −1-1, reaching G pays +10+10, entering H pays −10-10, and moves into a wall or the edge leave the agent where it is.

S....
.##.H
...#.
.H...
...#G
Download gridworld.py Download value_iteration.py Download test_gridworld.py Download test_value_iteration.py

value_iteration.py computes V∗V^* with γ=0.95\gamma = 0.95 and converges in 13 sweeps. Its greedy policy, printed as arrows, is:

v<<<<
v##^H
>>v#v
^H>>v
>>^#G

The shortest safe path takes 8 moves, so its return is seven step penalties and the goal reward. Check V∗(S)V^*(S) by hand:

V∗(S)=−∑k=060.95k+0.957⋅10=−1−0.9570.05+6.9834=−6.0333+6.9834=0.9501V^*(S) = -\sum_{k=0}^{6} 0.95^k + 0.95^7 \cdot 10 = -\frac{1 - 0.95^7}{0.05} + 6.9834 = -6.0333 + 6.9834 = 0.9501

which matches the program’s output of 0.9501. The top row is a trap: the route along it ends next to the hazard with a wall below, so the arrows there point back towards the start.

Tabular Q-learning

Q-learning learns Q∗Q^* without the model. After each real transition (s,a,r,s′)(s, a, r, s') it moves Q(s,a)Q(s, a) a step α\alpha towards a one-sample estimate of the Bellman right-hand side:

Q(s,a)←Q(s,a)+α[r+γmax⁡a′Q(s′,a′)⏟TD target−Q(s,a)]Q(s, a) \leftarrow Q(s, a) + \alpha\Big[\underbrace{r + \gamma \max_{a'} Q(s', a')}_{\text{TD target}} - Q(s, a)\Big]

The bracket is the temporal-difference (TD) error. At a terminal state the target is just rr. Q-learning is off-policy: the target uses the greedy next action, so it learns the values of the greedy policy while the robot behaves with an exploring one. The inner loop of qlearning.py is short:

for t in range(max_steps):
    a = epsilon_greedy(q[s], eps, rng)
    s2, r, done = env.step(s, a, rng)
    target = r if done else r + gamma * q[s2].max()
    q[s, a] += alpha * (target - q[s, a])      # temporal-difference update
    total += r
    s = s2
    if done:
        break
Download qlearning.py Download test_qlearning.py

The learner only calls env.step, never env.transitions. The test file trains with 5000 episodes, α=0.5\alpha = 0.5 and ε\varepsilon falling from 1.0 to 0.2, and checks five seeds against value iteration. Every non-terminal state must satisfy ∣max⁡aQ(s,a)−V∗(s)∣<10−3|\max_a Q(s,a) - V^*(s)| \lt 10^{-3}, every entry of QQ must be within 10−210^{-2} of Q∗Q^*, and every greedy action must be optimal. In a deterministic world with a constant α\alpha, Q-learning converges exactly, provided every state-action pair keeps being tried. With random transitions, α\alpha must decay for the estimates to settle. Run all tests from the download folder:

cd public/code/robotics-ai/reinforcement-learning
/Volumes/M2-Dev/conda/envs/toptech/bin/python -m pytest -q

Exploration

A purely greedy learner exploits whatever looks best so far and never discovers better routes. Epsilon-greedy exploration takes a uniformly random action with probability ε\varepsilon and the greedy action otherwise. With four actions and ε=0.2\varepsilon = 0.2, the greedy action is chosen with probability 0.8+0.2/4=0.850.8 + 0.2/4 = 0.85, and each other action with 0.050.05. A unit test checks those frequencies.

Two line charts against training episode, 0 to 3000. Left: training return per episode, averaged over 20 seeds. Constant epsilon 0.05 rises to about 2 within 100 episodes. Constant epsilon 0.3 levels off near minus 3.6. Epsilon decaying from 1.0 to 0.05 starts near minus 35 and reaches about 2 by episode 1500. A dotted line marks the optimal return of 3. Right: log-scale error between learned values and value iteration. Constant 0.05 stays near 6, constant 0.3 near 2.5, and the decaying schedule falls to about 0.001 by episode 700.
Q-learning on the 5 by 5 grid with three exploration settings. Low constant exploration finds the shortest path quickly but never learns values for states off that path. High constant exploration keeps paying for random moves. A decaying schedule explores the whole grid early and ends with near-optimal behaviour and values.

The figure shows the trade-off. With ε=0.05\varepsilon = 0.05 the agent finds the 8-step path within about 100 episodes. Cells away from that path, such as the bottom-left corner, are then almost never visited, and their values stay wrong by up to about 6. With ε=0.3\varepsilon = 0.3 the values improve, but the training return levels off near −3.6-3.6 because random moves walk into hazards. The decaying schedule pays early, with returns near −35-35, and ends with both a good policy and an accurate QQ. On a robot, every one of those early random moves is a real motion. That cost is the subject of a later section.

From tables to function approximation

A table needs one entry per state-action pair. Six joint angles discretised into 100 bins each give 101210^{12} states before velocities are added. Real robot RL therefore replaces the table with a parameterised function Qθ(s,a)Q_\theta(s, a), usually a neural network, and turns the update into regression on the TD target:

L(θ)=E[(r+γmax⁡a′Qθˉ(s′,a′)−Qθ(s,a))2]L(\theta) = \mathbb{E}\left[\left(r + \gamma \max_{a'} Q_{\bar\theta}(s', a') - Q_\theta(s, a)\right)^2\right]

Here θˉ\bar\theta is a slowly updated copy of the weights, so the target does not move with every gradient step. Transitions are stored in a replay buffer and sampled in random minibatches to break the correlation between consecutive steps. Two difficulties appear that the table did not have. Bootstrapping from your own estimates, with a function approximator, on off-policy data can diverge. And max⁡a′\max_{a'} over a continuous six-dimensional action space is itself an optimisation problem. For arm control the usual answer is to learn the policy directly.

Policy gradients and PPO

A policy-gradient method parameterises a stochastic policy πθ(a∣s)\pi_\theta(a \mid s), for example a Gaussian whose mean is a network output, and climbs the gradient of the expected return:

∇θJ(θ)=Eπθ[∇θlog⁡πθ(at∣st) A^t]\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\left[\nabla_\theta \log \pi_\theta(a_t \mid s_t)\, \hat{A}_t\right]

The advantage A^t\hat{A}_t estimates how much better action ata_t was than the policy’s average in sts_t, usually as the observed return minus a learned value V(st)V(s_t). Actions that did better than expected become more likely. The estimate is noisy, and one large step can wreck a policy that took hours to train.

Proximal Policy Optimization (PPO, Schulman et al., 2017) limits how far each update moves the policy. It defines the probability ratio between the new and the data-collecting policy:

rt(θ)=πθ(at∣st)πθold(at∣st)r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_\text{old}}(a_t \mid s_t)}

and maximises the clipped surrogate objective

LCLIP(θ)=E^t[min⁡(rt(θ)A^t,  clip⁡(rt(θ),1−ϵ,1+ϵ)A^t)]L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t\left[\min\left(r_t(\theta)\hat{A}_t,\; \operatorname{clip}\left(r_t(\theta), 1-\epsilon, 1+\epsilon\right)\hat{A}_t\right)\right]

For a positive advantage, the objective grows with rtr_t only up to 1+ϵ1+\epsilon. Beyond that it is flat, so there is no gradient pushing the action’s probability further. For a negative advantage, the same happens below 1−ϵ1-\epsilon. Taking the minimum makes the objective a pessimistic lower bound on the unclipped one. This lets PPO run several epochs of minibatch gradient steps on each batch of collected data without the policy running away. The paper uses ϵ=0.2\epsilon = 0.2; in its comparison on continuous-control benchmarks, 0.2 scored better than 0.1 and 0.3. In practice the loss also includes a value-function error term and an entropy bonus, and the advantages come from generalised advantage estimation.

PPO is on-policy: data collected with πθold\pi_{\theta_\text{old}} is used for a few epochs and then discarded. That is simple and stable, but it needs a lot of interaction. Off-policy actor-critic methods such as Soft Actor-Critic (SAC) reuse a replay buffer and need fewer samples, which matters when the samples come from a real arm.

Reward design and reward hacking

For a reaching task with gripper position pp and target gg, three common rewards are:

  • Sparse: +1+1 when ∥p−g∥<1\lVert p - g \rVert \lt 1 cm, else 0. It describes the task exactly, but random exploration almost never finds the reward in a large workspace.
  • Dense: −∥p−g∥-\lVert p - g \rVert every step. It gives a gradient everywhere, but it does not say “stop” or “be gentle”.
  • Shaped: the dense term plus a success bonus, minus penalties on action size ∥a∥2\lVert a \rVert^2, joint speed and contact force, with episode termination on success or on leaving the workspace.

The optimiser will maximise the number you give it, not the intent behind it. Reward hacking is the result. A per-step bonus for being near the target teaches the arm to hover and never finish, because touching ends the stream of reward. A reward measured on an object’s position can be collected by knocking the object towards the goal instead of grasping it. In simulation, a policy can find contact-solver errors that produce impossible forces. A missing action penalty gives jerky, bang-bang commands that wear the servos.

Two habits catch most of these. First, check the sign of the per-step reward: positive per-step rewards make survival attractive, negative ones make finishing attractive. Second, watch rollouts, not only the return curve. A rising return with strange motion is the classic symptom. Shaping that takes the form F=γΦ(s′)−Φ(s)F = \gamma \Phi(s') - \Phi(s) for some potential Φ\Phi, for example Φ=−∥p−g∥\Phi = -\lVert p - g \rVert, with Φ=0\Phi = 0 at terminal states, adds guidance without changing which policy is optimal.

Exercise

Reward hacking in the grid world

Change the step reward from −1-1 to +1+1 and run value iteration again (GridWorld(step_reward=1.0), γ=0.95\gamma = 0.95). Predict V∗(S)V^*(S) and the optimal behaviour before you run it. Then find the largest constant step reward for which reaching G is still optimal from S.

Show solution

With +1+1 per step, never finishing is worth ∑k0.95k=1/(1−0.95)=20\sum_k 0.95^k = 1/(1-0.95) = 20. Reaching the goal in 8 moves is worth ∑k=060.95k+0.957⋅10=6.03+6.98=13.02\sum_{k=0}^{6} 0.95^k + 0.95^7 \cdot 10 = 6.03 + 6.98 = 13.02, which is less. Value iteration returns V∗(S)=20.0V^*(S) = 20.0, and the greedy policy points every cell away from G and H: the arm “reaches” forever and never arrives.

For a step reward cc, finishing in 8 moves is worth c⋅6.0333+6.9834c \cdot 6.0333 + 6.9834, and never finishing is worth 20c20c. Finishing is optimal while 6.0333c+6.9834≥20c6.0333c + 6.9834 \ge 20c, that is c≤6.9834/13.9667≈0.50c \le 6.9834 / 13.9667 \approx 0.50. A small positive “alive” bonus is already enough to break the task. Use a negative step reward or a zero step reward with a goal bonus.

Sample cost on real hardware

The tabular agent above used 3000 episodes and 36,673 steps (seed 0) to learn a 21-state problem. At an example rate of 10 actions per second, that is about an hour of continuous robot motion, without counting the time to reset the scene after each episode. Robot tasks have far larger, continuous state spaces, and deep RL on them needs many more steps than this toy problem.

On a real arm each step has costs that a simulator does not:

  • Time: steps happen in real time, one robot at a time, and resets are slow or need a person.
  • Wear and damage: random exploration hits joint limits, the table and the robot itself. Small bus servos heat up under sustained load.
  • Safety: an exploring policy is by design unpredictable, so the workspace must be fenced and a person must watch.
  • Reward measurement: a simulator knows the exact gripper and object positions. On the real robot, the reward has to come from cameras or a trained success classifier, which adds noise and its own failure modes.

Simulation removes most of these costs. Physics engines step faster than real time, simulated robots never wear out, and GPU-accelerated frameworks such as Isaac Lab speed up the physics further. The price is the gap between the simulated and the real robot, which the next lesson, sim-to-real transfer and VLAs, covers.

Real-robot RL is possible when the task is short and the setup limits exploration. The LeRobot documentation describes a human-in-the-loop workflow (HIL-SERL) for arms such as the SO-101. It starts from a few teleoperated demonstrations, trains a vision-based reward classifier, and runs an off-policy SAC learner while a person can take over the arm at any time. It also bounds the end-effector workspace so that exploration stays in a safe box.

RL, imitation learning, or both

Imitation learningReinforcement learning
Training signalDemonstrated actionsScalar reward
Data sourceTeleoperation, fixed datasetThe policy’s own trials
Can exceed the demonstratorNoYes
Main riskCompounding errors off the demonstrated statesSample cost, reward hacking, unsafe exploration
Typical settingReal robot, tens to hundreds of episodesSimulation, or short real tasks with safeguards

The two combine well:

  • Pretrain, then fine-tune. Behaviour cloning gives a policy that already succeeds sometimes, and RL improves it. This avoids the hopeless random start of sparse-reward tasks.
  • Demonstrations in the replay buffer. An off-policy learner can mix demonstration transitions with its own, as the HIL-SERL workflow does.
  • Human interventions as data. When a person takes over, the corrected actions are recorded and the policy learns from them.
  • RL in simulation as a teacher. A policy trained with privileged simulator state (exact object poses) can generate demonstrations for a camera-based student policy trained by imitation.

For a new manipulation task on the SO-101, start with imitation learning (ACT and diffusion policies) and reach for RL when you need to go beyond what the demonstrations show, or when the task is defined better by an outcome than by a motion.

Exercise

One Q-learning update by hand

In the grid world with γ=0.95\gamma = 0.95 and α=0.5\alpha = 0.5, the agent is in the cell directly above G and all QQ values are still 0. It moves down, enters G and receives +10+10. What is the new QQ for that state and action? In a later episode, before the cell above G is visited again, the agent is in the cell to the left of it, moves right and receives −1-1. What is the update now, and what value does it converge to?

Show solution

G is terminal, so the target is just the reward: Q←0+0.5(10−0)=5Q \leftarrow 0 + 0.5(10 - 0) = 5. Each later visit halves the gap, giving 7.5, 8.75 and so on towards 10.

For the move into the cell above G, the target is −1+0.95max⁡a′Q(s′,a′)=−1+0.95⋅5=3.75-1 + 0.95 \max_{a'} Q(s', a') = -1 + 0.95 \cdot 5 = 3.75, so Q←0+0.5⋅3.75=1.875Q \leftarrow 0 + 0.5 \cdot 3.75 = 1.875. As the value above G approaches 10, the target approaches −1+9.5=8.5-1 + 9.5 = 8.5. Value iteration gives the same 8.5 for that state, two steps from the goal.

Summary

  • An MDP has states, actions, a transition model, a reward and a discount γ\gamma. The agent maximises the discounted return, with an effective horizon of about 1/(1−γ)1/(1-\gamma) steps.
  • The Bellman optimality equation defines V∗V^* and Q∗Q^*. Value iteration solves it when the model is known, and the greedy policy on Q∗Q^* is optimal.
  • Tabular Q-learning learns Q∗Q^* from sampled transitions with the TD update. It only learns what it visits, so exploration such as decaying epsilon-greedy decides how good the values become.
  • Large and continuous problems need function approximation. Policy-gradient methods such as PPO learn the policy directly, and PPO’s clipped ratio keeps each update close to the data-collecting policy.
  • Reward design is the hardest part of robot RL: positive per-step rewards, measurable proxies and missing penalties all invite reward hacking, so watch the rollouts.
  • Real-robot samples are slow, costly and risky, so most robot RL runs in simulation. Combining RL with demonstrations and human interventions cuts the number of real trials.

Check your understanding

Pick an answer for each question, then check. Answers stay hidden until you submit.

1. With a discount factor of 0.95, what is the present value of a reward of 10 received 7 steps in the future?
2. Q-learning is called off-policy. What does that mean?
3. A Q-learning agent with constant epsilon = 0.05 reaches the goal by the shortest path, yet its Q-table is far from Q* in one corner of the grid. What is the most likely cause?
4. In PPO with epsilon = 0.2, the advantage of an action is positive and the probability ratio r is already 1.35. What gradient does the clipped term give for increasing r further?
5. A reaching reward pays +1 every step the gripper is within 5 cm of the target and ends the episode when the gripper touches it. What behaviour should you expect?

Further reading

Found a mistake or something unclear?Report a correction.