Slicing Return Distributions: A Practical Route to Multivariate Distributional RL
Sliced divergences make it practical to compare multivariate return distributions. They offer an attractive solution for multi-objective distributional RL and beyond. Their use in TD learning still leaves some open questions.
Distributional RL models the distribution over returns instead of their expectation [1]. In the usual case of a single reward dimension, this is already useful. The mean alone does not show how much returns vary, whether their distribution is skewed, or whether it has several peaks. This also allows risk-aware action selection, for example by giving more weight to poor outcomes [19]. Distributional RL was also an important part of Rainbow's improvements over DQN [16].
In the multivariate case, for example when the reward has several dimensions, there is more to learn : the joint distribution tells us how the return components vary together [6]. Multi-objective RL is an obvious example where the reward can have several, possibly competing, components [17]. Other examples include successor features [14] and value-gradient methods [15]. Both use temporal difference updates with targets that have several dimensions.
But comparing multivariate distributions can be much more expensive than comparing 1D distributions. In this work, we use a trick from the optimal transport literature called slicing [7] [8] [9]. The core idea is to compare high-dimensional distributions by looking at their 1D marginals along different directions. We propose combining slicing with TD learning to learn return distributions. As we discuss below, this leaves some questions open.
Table of contents
- Why multivariate distributional RL ?
- What makes the multivariate case difficult ?
- The slicing trick
- Which 1D divergence should we use slicing with ?
- From sliced divergences to a TD update
- Uniform slicing vs max slicing
- What the theory says
- The TD sampling catch
- Experiments
- Practical takeaway
- Open question
- Bibliography
Why multivariate distributional RL ?
For a fixed policy $\pi$, the usual scalar return is a discounted sum of rewards,
With several reward components, we would like to predict their joint return distribution. Learning each component separately would not tell us how they vary together. For example, two objectives might both have high returns on some trajectories, or improving one might come at the expense of the other.
We can write a multivariate return as
Here $R_t$ is a reward vector. The discount $\Gamma(s,a)$ can be $\gamma I_d$, which applies the same discount to every component. A more general matrix can apply different discounts or mix the components.
The corresponding distributional Bellman update is
As in scalar RL, we combine the immediate reward with a discounted prediction at the next state. The difference is that the prediction is now a joint distribution over several return components.
What makes the multivariate case difficult ?
Why not just add more outputs to the usual 1D distributional critic ? Each distributional RL method combines two choices : 1) how to represent the distribution and 2) how to compare distributions. It turns out that both the representations and the losses do not extend so easily from 1D to multivariate distributions.
C51 models the distribution over returns by placing probabilities on a fixed grid [1]. Extending that grid to $d$ dimensions would require $m^d$ grid points for $m$ values per dimension. Quantile methods instead rely on ordering the returns [3]. Scalar samples can be sorted from smallest to largest, but multivariate samples have no obvious equivalent ordering.
What about simply using the Wasserstein distance ? In 1D, its empirical estimator is cheap to compute because it relies on sorting. In several dimensions, solving the optimal transport problem is much more expensive [6]. With $n$ samples, the costs can differ substantially.
| Problem | Typical empirical cost |
|---|---|
| 1D Wasserstein by sorting | $\mathcal O(n \log n)$ |
| exact multivariate OT | often around $\mathcal O(n^3 \log n)$ |
We would pay that cost at every TD update. Among other issues, the Wasserstein distance suffers from the curse of dimensionality. In general, maintaining the same estimation accuracy can require exponentially more samples as the dimension increases [20].
The slicing trick
The slicing trick is simple : instead of comparing the multivariate distributions directly, look at them from many directions, each time projecting them onto a line [7] [8] [9]. The figure below shows two projections of the same pair of distributions.
Choose a unit direction,
and project each vector $x \in \mathbb R^d$ onto it,
For distributions $\mu$ and $\nu$, the notation $(P_\theta)_\#\mu$ and $(P_\theta)_\#\nu$ means their distributions after projection. Both are in 1D, so we can compare them with a 1D divergence $\Delta$.
Uniform slicing averages these comparisons over all directions. We write the powered objective as
Here $\sigma$ is the uniform distribution over directions. In practice, we draw $L$ random directions and average their contributions,
With this estimator, we avoid computing an expensive multivariate transport plan, as we would with the Wasserstein distance. Instead, we compute the divergence between 1D marginals along many directions. The whole point is that each of these 1D comparisons can be very cheap. As we discuss below, we can compare those projections in several ways, depending on the variant of slicing we choose.
Which 1D divergence should we use slicing with ?
Many divergences can be computed at a reasonable cost in 1D. Here we consider the Wasserstein distance, the Cramér distance and the Maximum Mean Discrepancy (MMD) [11]. To see how they are computed, suppose our prediction and target are each represented by $n$ particles,
Each particle is one possible return vector of the distribution we seek to model. For a direction $\theta$, simply project the particles,
Then compute the 1D divergence between these bags of projected particles, and finally average over the random directions.
Wasserstein. Let $u_i=P_\theta z_i$ and $v_i=P_\theta \hat z_i$ be the projected particles. Sort both bags and match the smallest values, then the next smallest, and so on. Writing the sorted values as $u_{(i)}$ and $v_{(i)}$, the empirical estimator is
Averaging over $L$ directions gives
Sorting costs $\mathcal O(n\log n)$ per projection. The loss is cheap to compute, but its gradient can be biased when used in combination with TD learning [2] [4].
Cramér. This divergence only compares cumulative distribution functions, or CDFs [2]. A CDF $F_\mu(t)$ gives the probability of a value at or below $t$. It integrates the squared difference between the two CDFs,
To compute it from particles, merge both bags of projected particles and sort again. Between consecutive values $t_k$ and $t_{k+1}$, the empirical CDF difference is constant. Calling that difference $\Delta_k$, we get
The sliced objective averages these values over directions,
Again, sorting is the main cost, so each projection takes $\mathcal O(n\log n)$. Unlike Wasserstein, the squared Cramér objective has the unbiased gradients needed for the stochastic TD updates considered here [2].
MMD. Maximum Mean Discrepancy uses a kernel $k$ to compare pairs of samples [11]. It is the only divergence considered here that remains practical to compute directly in several dimensions. It compares kernel values within each distribution with those between the two distributions,
With projected particles, the standard biased estimator is
Sliced MMD averages the kernel loss after projection [9],
Evaluating this estimator requires $\mathcal O(n^2)$ kernel evaluations per projection, just as when comparing the two bags of samples in their original space. Interestingly, choosing the kernel gives some flexibility. We discuss which kernels can be used in this setting and with non-sliced MMD in our later work [21].
To summarize, the costs for one projection are
| Divergence | Time complexity |
|---|---|
| Wasserstein $\mathbf W_p$ | $\mathcal O(n\log n)$ |
| Cramér $\mathbf C_2$ | $\mathcal O(n\log n)$ |
| MMD$_k$ | $\mathcal O(n^2)$ |
MMD is also an existing baseline for distributional RL. Nguyen-Tang, Gupta and Venkatesh introduced it through moment matching [12]. Zhang et al. used it for multivariate rewards [5], and Wiltzer et al. studied its multivariate contraction properties [6]. We compare against that existing multivariate method as well as applying MMD after projection.
From sliced divergences to a TD update
How do we use these losses to train a critic ?
For each state-action pair $(s,a)$, the critic predicts $N$ return particles,
We sample an environment transition $(s,a,r,s')$ and a next action $a'\sim\pi(\cdot\mid s')$. Adding the reward to the discounted next-state particles gives the target,
Here $\phi$ denotes the critic's parameters and $\phi^-$ the parameters used to construct the target. We minimize a sliced divergence between the predicted and target particles, treating the target as fixed during the gradient step.
sample transition (s, a, r, s')
sample next action a' ~ pi(. | s')
for i = 1,...,N:
z_i = Z_phi(s, a, i)
zhat_i = r + Gamma(s,a) Z_phi_minus(s', a', i)
draw projection directions theta_1,...,theta_L
project z_i and zhat_i on each theta_l
compute the 1D divergence on each projection
average over directions
update phi by gradient descent
This is one-sample TD. The name refers to sampling one environment transition for the update, not to using just one critic particle. We can have many target particles, but they all come from the same sampled successor and reward. They do not represent all the outcomes that could have followed $(s,a)$.
Uniform slicing vs max slicing
So far, we have averaged over random directions. Another option is to search for the direction that best distinguishes the two distributions [10]. This gives max slicing,
In practice, we approximate the supremum by optimizing the direction on the unit sphere. This costs more than drawing random directions, but it can give a contraction guarantee that uniform slicing does not have. The difference appears when the discount is a general matrix rather than the same scalar for every return component.
What the theory says
With an exact Bellman update, we would like two different return predictions to move closer together. A contraction bound makes that precise. It tells us how much the distance between them shrinks after applying the operator.
Write $\eta(s,a)$ for a predicted return distribution. The operator is
The form of $\Gamma(s,a)$ determines which contraction results we can obtain.
Classical distributional RL. There is one return component and one discount factor,
Multivariate returns with a shared discount. Every component uses the same discount factor,
Suppose the base divergence satisfies the required translation, scaling and mixture conditions. Uniform slicing then gives
The overline takes the largest divergence across state-action pairs, and $c(\gamma)<1$ is the contraction factor.
Result 1. Uniform slicing preserves contraction with a shared scalar discount. For $\Gamma=\gamma I_d$, the sliced divergence inherits the contraction guarantee of its one-dimensional base divergence, under those conditions.
A fixed discount matrix. We can also use
For example, we might predict returns over different horizons by assigning a different discount factor to each component [18],
A discount matrix that depends on the state and action. More generally,
The off-diagonal entries allow one return component to contribute to the update of another. To bound the effect of these matrices, define
The operator norm measures the largest factor by which a matrix can stretch a vector. The contraction bound we would like to obtain depends only on $\bar L$,
We want $c(\bar L)<1$ whenever $\bar L<1$.
Result 2. Uniform slicing need not satisfy this bound for general discount matrices. Even when $\bar L<1$, we cannot generally bound its contraction using $\bar L$ alone. Some MMD constructions fail this bound as well.
The failure concerns this particular bound, not every possible contraction result.
Max slicing instead considers the most discriminative direction,
Result 3. Max slicing recovers the desired bound. Under the assumptions on the base divergence, it gives a contraction factor controlled by $\bar L$ for general discount matrices.
These results concern the Bellman operator applied to whole distributions. Our algorithm only sees samples, so we still need to check what happens to its gradients.
The TD sampling catch
An essential property for stochastic TD is that the gradient from a sampled target agrees, on average, with the gradient for the full target. A contraction guarantee alone does not establish this [2] [4].
The full Bellman target mixes the possible next states and actions according to their probabilities,
One-sample TD uses the distribution associated with the sampled successor instead,
Why does that matter ? Comparing a prediction with each outcome in turn need not have the same effect as comparing it with their mixture. The expected gradient can point in a different direction.
For an empirical target $\hat\mu_m$ built from independent samples $X_{1:m}$ of $\mu$, the unbiased sample-gradient property is
Here $\nu_\phi$ is the prediction being trained. In one-sample TD, the target particles share a sampled successor rather than being independent draws from the full mixture. More particles from that one successor do not remove the mismatch.
Uniform slicing preserves unbiased gradients. If $\Delta^p$ has this property, so does $\mathbf S\Delta_p^p$. Averaging losses over independently sampled directions preserves the equality.
Max slicing does not preserve it in general. The direction is chosen using the same sample on which the loss is evaluated. This selection can bias the gradient, even if the loss for every fixed direction has unbiased gradients.
The base divergence matters too. Wasserstein has biased sample gradients [2]. The squared Cramér and squared MMD objectives have the required property, and uniform slicing preserves it [12] [13].
| Objective | Unbiased |
|---|---|
| $\mathbf{SW}_p$ | ✗ |
| $\mathbf{SC}_2^2$ | ✓ |
| $\mathbf{SMMD}_{k}^{2}$ | ✓ |
| $\mathbf{MSW}_p$ | ✗ |
| $\mathbf{MSC}_2$ | ✗ |
| $\mathbf{MSMMD}_{k}$ | ✗ |
| $\mathbf W_p$ | ✗ |
| $\mathbf{MMD}_{k}^{2}$ | ✓ |
For Cramér and MMD, the square matters. The property applies to the squared objective used for training, not automatically to its square root.
Experiments
Does the predicted distribution match the returns we actually observe ? And does the critic help choose good actions ? We test these questions separately.
In policy evaluation, the policy stays fixed while the critic learns its return distribution. In control, the critic is also used to choose actions. A critic can do well at one without doing well at the other.
Chain environment.
The first experiment is policy evaluation on a small chain of states. The policy always takes the forward action. From $s_i$, it reaches $s_{i+1}$ with probability $0.9$ and resets to $s_0$ with probability $0.1$. The reward has one component per state, with a value of one for the state just entered and zero elsewhere.
Because we know the two possible successors and their probabilities, we can compare two ways of building the target.
With one-sample TD, we observe one transition and form the target from that successor's predicted return distribution. With true distributional TD, we would instead use the full mixture of both possible outcomes.
We approximate the latter by applying the reward and discount to both successor distributions, then mixing them with weights $0.9$ and $0.1$. We resample particles from this mixture to keep the same particle budget. This is the near-exact update. It still uses a finite approximation, but no longer conditions the entire target on one sampled successor.
The bars measure empirical $\widehat{\mathbf W}_2$ between the critic's particles at the start state and returns collected from Monte Carlo rollouts. Lower values mean a closer match. The bars show medians over 25 random seeds, with 95% bootstrap confidence intervals.
With the near-exact update, the objectives give much more similar results. With one-sample TD, Wasserstein and max-sliced objectives perform poorly. Changing how we construct the target changes whether these losses can learn the distribution.
The plots below show the distributions themselves, using two return coordinates. Orange points are rollout returns and blue diamonds are critic particles. The crosses mark their means. Each column uses a different objective, with one-sample TD on the top row and the near-exact update below.
For some objectives, the predicted particles collapse under one-sample TD even though their mean remains close to the rollout mean. Getting the mean roughly right is not the same as learning the distribution.
Maze from pixels.
The maze experiments also evaluate a fixed policy, but the critic now learns from images [5]. The policy chooses actions uniformly at random. We use ordinary one-sample TD, without explicitly building the mixture of possible outcomes as we did in the chain.
Each observation is an $84\times84\times3$ RGB image. The grey triangle is the agent, while grey blocks and black walls are obstacles. The agent can move up, down, left or right. Moves into an obstacle have no effect. Each colored square gives a stochastic reward in its corresponding component, and can be collected only once per episode.
Colored walls restrict which way the agent can pass. Yellow walls allow passage from the top or left, and red walls from the bottom or right. The three layouts below change which rewards can be reached along the same trajectory. The critic has to learn how the resulting return components vary together, not just how large each one is on average.
The bars use the same comparison with Monte Carlo returns as in the chain. They show medians over five random seeds, with 95% bootstrap confidence intervals. Sliced Cramér and sliced MMD learn the distributions accurately. Objectives with biased gradients again give worse matches.
The plots below show two return coordinates at the start state. Rows correspond to maze variants and columns to training objectives. As before, orange points are rollout returns and blue diamonds are critic particles. Look at whether the blue particles cover the same regions as the orange returns, rather than only whether their averages are close.
Atari control.
The Atari experiments test action selection rather than distributional accuracy. They use games whose rewards can be decomposed into several components [5]. The plot aggregates normalized evaluation scores. Higher is better here.
Sliced Cramér performs strongly, but so does sliced Wasserstein, despite its poor distributional estimates in the previous experiments. How can that happen ?
The policy chooses actions using expected returns. A critic can provide useful estimates of those expectations even when its predicted distribution has collapsed. The chain plots show how these two things can happen together. Good control performance therefore does not establish that the critic has learned the return distribution [4].
Practical takeaway
Sliced Cramér gives accurate multivariate return predictions with one-sample TD in these experiments. It combines efficient estimation with the unbiased gradients needed for these updates.
Sliced MMD gives more freedom through the choice of kernel. In these experiments, though, it brings no clear advantage over sliced Cramér, while requiring quadratic time in the number of particles.
Sliced Wasserstein is not appropriate as a distributional learning objective under one-sample stochastic TD. Its good Atari scores do not change that conclusion. A collapsed prediction can still give useful estimates for choosing actions, without representing the return distribution accurately.
Max slicing gives the contraction guarantee we want for general discount matrices. But choosing its direction from the same sampled target can bias the gradient. It is therefore not generally appropriate for one-sample stochastic TD either.
Open question
Can we find an objective that combines all of the following ?
- A contraction bound controlled only by the discount matrix norm.
- Unbiased gradients for one-sample TD.
- Efficient empirical estimators.
- Estimation rates that do not deteriorate sharply as the dimension grows.
The dream is a divergence $\mathcal D$ satisfying
while remaining practical to estimate and optimize from sampled transitions.
We left this as an open question.
Bibliography
- Bellemare, Dabney, and Munos, 2017. A Distributional Perspective on Reinforcement Learning. arXiv.
- Bellemare, Danihelka, Dabney, Mohamed, Lakshminarayanan, Hoyer, and Munos, 2017. The Cramér Distance as a Solution to Biased Wasserstein Gradients. arXiv.
- Dabney, Rowland, Bellemare, and Munos, 2018. Distributional Reinforcement Learning with Quantile Regression. arXiv.
- Rowland, Dadashi, Kumar, Munos, Bellemare, and Dabney, 2019. Statistics and Samples in Distributional Reinforcement Learning. arXiv.
- Zhang, Chen, Zhao, Xiong, Qin, and Liu, 2021. Distributional Reinforcement Learning for Multi-Dimensional Reward Functions. OpenReview.
- Wiltzer, Farebrother, Gretton, and Rowland, 2024. Foundations of Multivariate Distributional Reinforcement Learning. arXiv.
- Rabin, Peyré, Delon, and Bernot, 2011. Wasserstein Barycenter and its Application to Texture Mixing. Springer.
- Bonneel, Rabin, Peyré, and Pfister, 2015. Sliced and Radon Wasserstein Barycenters of Measures. Springer.
- Nadjahi, Durmus, Chizat, Kolouri, Shahrampour, and Simsekli, 2020. Statistical and Topological Properties of Sliced Probability Divergences. arXiv.
- Deshpande et al., 2019. Max-Sliced Wasserstein Distance and Its Use for GANs. arXiv.
- Gretton, Borgwardt, Rasch, Schölkopf, and Smola, 2012. A Kernel Two-Sample Test. JMLR.
- Nguyen-Tang, Gupta, and Venkatesh, 2021. Distributional Reinforcement Learning via Moment Matching. arXiv.
- Killingberg and Langseth, 2023. The Multiquadric Kernel for Moment-Matching Distributional Reinforcement Learning. OpenReview.
- Barreto et al., 2017. Successor Features for Transfer in Reinforcement Learning. arXiv.
- Debes and Tuytelaars, 2026. Distributional Value Gradients for Stochastic Environments. arXiv.
- Hessel, Modayil, van Hasselt, Schaul, Ostrovski, Dabney, Horgan, Piot, Azar, and Silver, 2018. Rainbow: Combining Improvements in Deep Reinforcement Learning. arXiv.
- Felten, Alegre, Nowé, Bazzan, Talbi, Danoy, and da Silva, 2023. A Toolkit for Reliable Benchmarking and Research in Multi-Objective Reinforcement Learning. Proceedings of NeurIPS 2023. OpenReview.
- Fedus, Gelada, Bengio, Bellemare, and Larochelle, 2019. Hyperbolic Discounting and Learning Over Multiple Horizons. arXiv.
- Dabney, Ostrovski, Silver, and Munos, 2018. Implicit Quantile Networks for Distributional Reinforcement Learning. arXiv.
- Fournier and Guillin, 2015. On the Rate of Convergence in Wasserstein Distance of the Empirical Measure. arXiv.
- Debes and Tuytelaars, 2026. Bellman Contraction under MMD: A Unified Framework. arXiv.