Policy Gradients — Section 8: Decision Making Under Uncertainty

Q-learning learns values, then acts greedily. Policy gradients optimize the policy $\pi_\theta$ directly via gradient ascent on expected return.

REINFORCE

The simplest policy gradient. For a parameterized policy $\pi_\theta(a | s)$ , the gradient of the expected return $J(\theta)$ is:

\nabla J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left[\sum_t \nabla \log \pi_\theta(a_t | s_t) \cdot G_t\right]

where $G_t = \sum_{k \geq t} \gamma^{k-t} r_k$ is the discounted return from time $t$ . Update: $\theta \leftarrow \theta + \alpha \nabla J(\theta)$ , estimated from one or more trajectories.

The intuition: increase the log-probability of actions that led to high return; decrease the log-probability of those that led to low return.

Variance is the bottleneck

REINFORCE's gradient estimate has very high variance — the return $G_t$ depends on the entire future trajectory, much of which is unrelated to the action at time $t$ . Two standard variance reductions:

### Baselines

Subtract a state-dependent baseline $b(s)$ from $G_t$ . Doesn't change the gradient in expectation but reduces variance:

\nabla J(\theta) = \mathbb{E}\left[\sum_t \nabla \log \pi_\theta(a_t | s_t) \cdot (G_t - b(s_t))\right]

The natural baseline is $V^\pi(s)$ . Approximate it with a learned critic.

### Actor-critic

Train two networks: actor $\pi_\theta$ for the policy, critic $V_\phi$ for the baseline. Use the advantage $A_t = Q(s_t, a_t) - V(s_t)$ in place of $G_t - b(s_t)$ . Generalized Advantage Estimation (GAE) interpolates between high-bias short returns and high-variance Monte-Carlo returns.

PPO (Proximal Policy Optimization)

The workhorse of modern RL. Optimizes a clipped surrogate objective that prevents the policy from changing too much per update:

L^{\text{CLIP}}(\theta) = \mathbb{E}\left[\min(r_t(\theta) A_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) A_t)\right]

where $r_t(\theta) = \pi_\theta(a_t | s_t) / \pi_{\text{old}}(a_t | s_t)$ . The clip prevents huge policy jumps on high-advantage transitions.

PPO is the default RL algorithm in 2024-2026 for continuous-control and language-model alignment. Stable, reasonably hyperparameter-robust, well-understood.

When policy gradients beat Q-learning

Continuous action spaces — Q-learning needs argmax over actions, which is intractable continuous. Policy gradients output actions directly.
Stochastic policies — exploration is built in; you sample from $\pi_\theta$ .
Easier to incorporate priors via policy structure.

When Q-learning wins

Discrete actions, off-policy learning desired.
Large replay buffers / sample efficiency matters.
Multi-task learning with shared value function.

Modern landscape

DDPG, TD3, SAC for continuous control. PPO for everything else by default. AlphaZero and successors for game-like discrete domains. Many production RL systems are actor-critic variants of PPO with domain-specific reward engineering.