Robot Air Hockey Challenge 2023

The problem: a game that punishes slow planners

Air hockey is a hard robotics benchmark for an unglamorous reason: the puck is faster than your control loop would like. A puck struck at ~1.5 m/s crosses the half-table in under 300 ms. A 7-DOF arm at 50 Hz gets roughly 15 control steps to observe the trajectory, plan an interception, and drive the mallet into position — with joint-velocity and workspace constraints the whole time.

This project addresses the Air Hockey Challenge 2023 (University of Freiburg), a standardised competition where a KUKA LBR iiwa14 robot arm must play three tasks in MuJoCo simulation:

Task	Goal
Defend	Prevent the puck from entering the agent’s goal
Hit	Strike the puck into the opponent’s goal
Prepare	Push a stationary puck into a target preparation zone

The constraints that make each task hard:

Contact dynamics — mallet-puck impact is brief and stiff; approach angle matters.
Workspace — the mallet must stay on the table plane ( $z = 0.1645$ m) inside $x \in [0.65, 1.32]$ m, $y \in [-0.40, 0.40]$ m.
Joint limits — the KUKA has per-joint position and velocity bounds that cannot be violated without hardware damage.
Latency — a trajectory planned for where the puck was is wrong by the time it executes.

Architecture overview

Rather than choosing between model-based and learned control, the project tests two hybrids that share the same high-level policy but differ in the low-level trajectory executor.

Observation (23-dim)
        │
        ▼
  TQC Policy (2D action)      ← same in both variants
        │
        ▼
 desired (x, y) on table
        │
   ┌────┴────┐
   │         │
  IK        MPC
   │         │
   ▼         ▼
joint pos  joint vel commands → robot (50 Hz)

Shared component — TQC actor: the RL policy outputs a 2D normalised target position on the table. It learns what to aim for given the puck and arm state.

IK variant: converts the 2D target to joint angles in ~1 ms using the closed-loop IK solver built into MuJoCo.

MPC variant: solves a constrained nonlinear optimisation over a 5-step horizon in ~20 ms, explicitly encoding joint-velocity limits and workspace bounds into the trajectory.

Background: Truncated Quantile Critics (TQC)

TQC is a distributional extension of Soft Actor-Critic. Instead of maintaining a single scalar $Q(s, a)$ , it maintains an ensemble of critics each outputting a set of quantile estimates — a full return distribution. Over-optimism is controlled by trimming the top quantiles during target computation.

Policy representation

The actor is a Gaussian policy with a tanh squashing transformation:

\pi(a \mid s) = \text{tanh}(\epsilon), \quad \epsilon \sim \mathcal{N}(\mu_\theta(s),\, \sigma_\theta(s))

Because tanh squashes $\epsilon \in \mathbb{R}$ to $a \in (-1, 1)$ , the log-probability requires a Jacobian correction:

\log \pi(a \mid s) = \log \phi(\epsilon) - \sum_{i} \log(1 - \tanh^2(\epsilon_i))

where $\phi$ is the standard normal density. The second term is numerically stabilised as:

\log(1 - \tanh^2(\epsilon)) = 2\log 2 - 2\,\text{softplus}(2\epsilon) + 2\epsilon

Distributional critic

Each of $N_{\text{nets}} = 5$ critic networks outputs $N_q = 25$ quantile estimates $z_1 < z_2 < \cdots < z_{N_q}$ that collectively represent $\hat{F}_{Q}(s,a)$ , the CDF of returns. The quantile targets are $\tau_i = \frac{2i - 1}{2N_q}$ for $i = 1, \ldots, N_q$ .

Bellman target with top-quantile dropping: to reduce over-estimation bias, the top $d = 2$ quantiles per network are discarded before computing targets, leaving $N_{\text{nets}} \cdot N_q - d \cdot N_{\text{nets}} = 115$ target quantiles. Sorted and filtered:

\mathbf{z}'_{\text{target}} = \text{sort-and-drop-top-}d\!\left(\{z'_{k,j}\}_{k=1,j=1}^{N_{\text{nets}}, N_q}\right)

The Bellman target for each surviving quantile is:

y_j = r + \gamma(1 - d)\!\left[z'_{\text{target},j} - \alpha \log \pi(a' \mid s')\right], \quad a' \sim \pi(\cdot \mid s')

Quantile Huber loss

The critic is trained with asymmetric Huber regression. Let $\delta_{ij} = y_j - z_i(s, a)$ be the TD error between target quantile $j$ and predicted quantile $i$ :

\rho_{\tau_i}(\delta) = \left|\tau_i - \mathbf{1}[\delta < 0]\right| \cdot L_\kappa(\delta)

where $L_\kappa$ is the Huber loss with threshold $\kappa = 1$ :

L_\kappa(\delta) = \begin{cases} \tfrac{1}{2}\delta^2 & \text{if } |\delta| \leq \kappa \\ \kappa\!\left(|\delta| - \tfrac{1}{2}\kappa\right) & \text{otherwise} \end{cases}

Total critic loss, averaged over the batch and summed over predicted/target quantile pairs:

\mathcal{L}_{\text{critic}} = \mathbb{E}_{(s,a,r,s') \sim \mathcal{B}}\!\left[\frac{1}{N_q N_{\text{targets}}} \sum_{i=1}^{N_q} \sum_{j=1}^{N_{\text{targets}}} \rho_{\tau_i}(\delta_{ij})\right]

The asymmetric weight $|\tau_i - \mathbf{1}[\delta < 0]|$ enforces that quantile $i$ underestimates at rate $\tau_i$ and overestimates at rate $1 - \tau_i$ , which is the defining property of quantile regression.

Actor loss and automatic entropy tuning

The actor maximises the expected soft return:

\mathcal{L}_{\text{actor}} = \mathbb{E}_{s \sim \mathcal{B},\, a \sim \pi}\!\left[\alpha \log \pi(a \mid s) - \bar{Q}(s, a)\right]

where $\bar{Q}$ is the mean over all critic quantiles. The temperature $\alpha$ is tuned automatically to enforce a minimum entropy target $\mathcal{H}^* = -\dim(\mathcal{A}) = -2$ :

\mathcal{L}_\alpha = -\log\alpha \cdot \mathbb{E}_{a \sim \pi}\!\left[\log \pi(a \mid s) + \mathcal{H}^*\right]

Target critic weights are updated via Polyak averaging at each step:

\theta_{\text{target}} \leftarrow \tau\,\theta + (1 - \tau)\,\theta_{\text{target}}, \quad \tau = 0.005

Variant 1: TQC + Inverse Kinematics

Inverse kinematics answers the question: given a desired end-effector (EE) position, what joint configuration $q$ achieves it?

The robot’s forward kinematics defines the EE position as:

p_{\text{EE}} = f(q) \in \mathbb{R}^3

The IK problem is to find $q^*$ satisfying $f(q^*) = p_{\text{des}}$ . MuJoCo’s built-in CLIK (Closed-Loop Inverse Kinematics) solver iterates:

\Delta q = J^+(q)\,(p_{\text{des}} - f(q))

where $J^+(q) = J^\top(JJ^\top + \lambda I)^{-1}$ is the damped pseudo-inverse of the $3 \times 7$ Jacobian $J(q) = \nabla_q f(q)$ , and $\lambda$ is a small damping constant.

Execution flow per step:

TQC outputs $(x_{\text{norm}}, y_{\text{norm}}) \in [-1,1]^2$ .
Rescale to physical coordinates: $x = 0.65 + 0.335(x_{\text{norm}} + 1)$ , $y = 0.40\,y_{\text{norm}}$ .
Set $z = 0.1645$ m (table surface height).
Solve IK → joint configuration $q^*$ .
Send $q^*$ as the position command to the robot.

Why IK is fast: the CLIK solve converges in a few iterations with no inequality constraints. Typical wall-clock time is ~1 ms per step at 50 Hz — well within budget.

Where IK falls short: IK finds a pose, not a trajectory. It does not consider joint velocity limits between the current configuration and $q^*$ , nor does it look ahead more than one step. In practice this can produce abrupt torques or step into configurations that are hard to escape.

Variant 2: TQC + Model Predictive Control

MPC replaces the single-step IK solve with a receding-horizon optimisation that explicitly enforces constraints over $N = 5$ future time steps.

Why a horizon helps

At each 20 ms control step, MPC plans a short trajectory:

now      +20ms    +40ms    +60ms    +80ms    +100ms
 │         │        │        │        │         │
 s₀ ──u₀──► s₁ ──u₁──► s₂ ──u₂──► s₃ ──u₃──► s₄ ──u₄──► s₅

All five joint-velocity commands $u_0, \ldots, u_4$ and all five EE positions $s_0, \ldots, s_5$ are optimised simultaneously. Only $u_0$ is executed; the problem is re-solved next step with updated state. This means:

Joint-velocity limits are respected across every future step, not just the current one.
Workspace bounds are enforced as hard constraints on every $s_k$ .
Anticipatory smoothing — the planner avoids configurations that would force large corrections later.

System dynamics

The EE velocity is related to joint velocities by the Jacobian:

\dot{s} = J(q)\,\omega, \quad J(q) \in \mathbb{R}^{3 \times 7}

$J(q)$ is computed numerically at the current joint configuration via MuJoCo’s mj_jacBody. This linearisation is re-computed at every MPC call, making the approach “linearise-and-optimise” rather than globally nonlinear.

RK4 integration

Euler integration $s_{k+1} \approx s_k + h\,J\omega$ would accumulate drift. The MPC uses fourth-order Runge-Kutta for accurate rollout within the horizon:

\begin{aligned} k_1 &= f(s_k,\; u_k) \\ k_2 &= f\!\left(s_k + \tfrac{h}{2}k_1,\; u_k\right) \\ k_3 &= f\!\left(s_k + \tfrac{h}{2}k_2,\; u_k\right) \\ k_4 &= f(s_k + h\,k_3,\; u_k) \\ s_{k+1} &= s_k + \frac{h}{6}\left(k_1 + 2k_2 + 2k_3 + k_4\right) \end{aligned}

where $f(s, u) = J(q)\,u$ and $h = 0.02$ s. RK4 is fourth-order accurate in $h$ , meaning integration error is $O(h^5)$ per step — negligible over a 5-step horizon.

Nonlinear programme (NLP) formulation

The MPC solves, at each step:

\min_{u_0,\ldots,u_{N-1}} \;\sum_{k=0}^{N-1} \|s_k - s_{\text{des}}\|_2^2 + \|s_N - s_{\text{des}}\|_2^2

\text{subject to:} \quad s_{k+1} = \Phi(s_k, u_k) \quad [\text{RK4 dynamics}]

\omega_{\min} \leq u_k \leq \omega_{\max} \quad [\text{joint velocity limits}]

s_k \in \mathcal{W} \quad [\text{workspace bounds}]

where $s_{\text{des}}$ is the 3D target provided by TQC, and $\Phi$ denotes one RK4 step. The workspace set is $\mathcal{W} = \{s : x \in [0.5, 1.5],\, y \in [-0.5, 0.5],\, z = 0.165\}$ m.

KUKA LBR iiwa14 joint velocity limits (rad/s):

Joint	$\omega_{\min}$	$\omega_{\max}$
1	−1.484	+1.484
2	−1.484	+1.484
3	−1.745	+1.745
4	−1.309	+1.309
5	−2.269	+2.269
6	−2.356	+2.356
7	−2.356	+2.356

NLP variable layout. CasADi assembles the decision vector $w$ by interleaving state and control nodes:

w = [\underbrace{s_0}_{3},\; \underbrace{u_0}_{7},\; \underbrace{s_1}_{3},\; \underbrace{u_1}_{7},\; \ldots,\; \underbrace{s_N}_{3}] \in \mathbb{R}^{N(3+7) + 3} = \mathbb{R}^{53}

for $N = 5$ this gives 53 decision variables. The problem is solved by IPOPT (Interior Point OPTimizer) in typically 10–20 iterations. The first control action $u_0$ is extracted and applied; joint positions are integrated forward:

q_{\text{next}} = q_{\text{current}} + u_0 \cdot \Delta t

Why MPC outperforms IK

The table below captures the key trade-offs:

Property	IK	MPC
Planning horizon	1 step (greedy)	5 steps (0.1 s)
Joint velocity constraints	Implicit — may saturate	Explicit — never violated
Workspace constraints	None	Hard inequality constraints
Trajectory smoothness	Potentially jerky	Smooth by construction
Per-step compute	~1 ms	~20 ms
Solver success rate	~100 %	~99 % (IPOPT convergence)

The key insight is predictive constraint satisfaction. IK generates the nearest joint configuration that reaches the target — if the required motion exceeds a joint velocity limit, the robot simply saturates and falls short. MPC, by contrast, solves for the entire trajectory simultaneously. If the target requires fast motion in a direction where a joint is near its limit, the optimiser redistributes the workload across the other six joints over the full horizon.

Receding-horizon planning prevents local traps. A greedy controller (IK) can move the arm into a configuration that is excellent for the current step but constrains every subsequent step — for example, parking a joint near its limit to intercept one puck, then being unable to recover for the next shot. MPC sees 100 ms ahead and implicitly penalises ending the horizon in a poor posture.

Smoothness emerges from the cost, not from filtering. IK outputs are sometimes post-processed with filters to reduce jitter. MPC produces smooth trajectories without post-processing because continuity is encoded in the dynamics constraints — $s_{k+1}$ must be reachable from $s_k$ under the RK4 model, which already limits how fast the EE can jump.

The cost for this is the ~20× higher per-step compute (~20 ms vs ~1 ms). On the defend task this is affordable at 50 Hz, but would become a bottleneck in faster tasks or on hardware with tighter real-time guarantees.

Reward design

The reward functions are designed to be sparse at the terminal level (large bonus/penalty at episode end) and dense during execution (continuous shaping terms that guide early learning).

Defend reward

Phase 1 — Approach (before first mallet-puck contact):

r_{\text{approach}} = -\|p_{\text{EE}} - p_{\text{intercept}}\|_2

where $p_{\text{intercept}} = (-0.6, y_{\text{puck}})$ is the point on the defensive line with the same $y$ -coordinate as the puck.

Phase 2 — Post-contact (after the mallet touches the puck, if puck moves toward agent):

r_{\text{contact}} = 0.3 - |v_{x,\text{puck}}|

This rewards slowing the puck’s horizontal velocity.

Terminal rewards:

r_{\text{terminal}} = \begin{cases} -100 & \text{puck enters agent's goal} \\ +150 & \text{puck velocity} < 0.1 \text{ m/s (successfully stopped)} \end{cases}

Constraint violation penalty (soft, applied every step):

r_{\text{constraint}} = -\frac{1}{10}\sum_j \max(0,\, c_j(q, \dot{q}, p_{\text{EE}}))

summed over all joint position, velocity, and workspace violations. This soft penalty encourages feasibility during training without hard resets.

Hit reward

The hit task requires striking the puck toward the opponent’s goal, so the reward encodes both approach angle and post-contact velocity:

r_{\text{approach}} = -\frac{\|p_{\text{EE}} - p_{\text{puck}}\|_2}{2} + (\cos\theta_{\text{opt}} - 1) \cdot 0.5

where $\cos\theta_{\text{opt}} = \max(\cos\theta_{\text{goal}},\, \cos\theta_{\text{bank}})$ — the agent may aim directly or use a bank shot.

Post-contact:

r_{\text{hit}} = \min\!\left(1,\; 0.3 \cdot \max(v_{x,\text{puck}}, 0)\right) - 0.01\,\|q - q_{\text{init}}\|_2

The posture penalty $-0.01\|q - q_{\text{init}}\|_2$ discourages drifting into awkward configurations after the strike. Terminal: $+100$ if the puck scores.

Prepare reward

The agent must push the puck into the preparation zone $\mathcal{Z} = \{(x,y) : x \in [-0.8, -0.2],\, |y| < 0.3\}$ . A symmetric approach-angle reward guides the mallet to contact the puck from a direction pointing into $\mathcal{Z}$ . Terminal: $+10$ if puck is in $\mathcal{Z}$ at episode end, $-10$ otherwise.

Training setup

Both agents share the same hyperparameters and training pipeline.

Hyperparameter	Value
Algorithm	TQC
Total timesteps	50 M
Discount $\gamma$	0.99
Polyak $\tau$	0.005
Batch size	256
Actor/critic LR	$3 \times 10^{-3}$
Temperature LR	$3 \times 10^{-4}$
Critic networks $N_{\text{nets}}$	5
Quantiles per net $N_q$	25
Top quantiles dropped $d$	2
Max episode steps	500 (10 s)
Evaluation frequency	every 3 000 steps

Replay buffer: standard FIFO with capacity $10^6$ transitions. Each transition stores $(s, a, s', r, \text{done})$ where $s \in \mathbb{R}^{23}$ : puck position (3), puck velocity (3), joint positions (7), joint velocities (7), opponent EE position (3).

Gradient updates: one update per environment step, sampling a fresh batch each time. Critic and actor losses are back-propagated separately; target critics are updated with Polyak averaging after each step.

Environment

Simulation: MuJoCo (stiff contact dynamics, required for accurate mallet-puck collisions).

Control frequency: 50 Hz ( $\Delta t = 0.02$ s).

Robot: KUKA LBR iiwa14, 7 degrees of freedom.

Table geometry:

Parameter	Value
Length	1.948 m
Width	1.038 m
Goal width	0.25 m
Surface height	0.1645 m

Observation space (23 dimensions):

Component	Dim	Description
Puck position	3	$(x, y, z)$ in world frame
Puck velocity	3	$(\dot{x}, \dot{y}, \dot{z})$
Joint positions	7	$q_1, \ldots, q_7$ (rad)
Joint velocities	7	$\dot{q}_1, \ldots, \dot{q}_7$ (rad/s)
Opponent EE	3	$(x, y, z)$ of opponent mallet

Action space: 2D Cartesian target $(x, y)$ normalised to $[-1, 1]$ .