Reinforcement Learning Fundamentals

Train agents to make sequential decisions through trial and error. Covers Markov decision processes, Q-learning, policy gradients, reward shaping, and the patterns that let AI systems learn optimal behavior from interaction with an environment.

Reinforcement learning (RL) is a family of algorithms for training agents to make sequential decisions in environments where the dynamics are unknown and feedback arrives as sparse, delayed rewards. It matters when the problem structure cannot be captured by static classification or regression: when an agent must explore unknown state spaces, discover long-term credit assignment, and optimize a cumulative return signal. This reference page covers the fundamental workflow from problem formulation to training, evaluation, and the practical failure modes that appear once the theory leaves the textbook.

Formulate the Problem as an MDP

Every RL problem reduces to a Markov decision process: a tuple (S, A, P, R, γ) where S is the state space, A the action space, P(s'|s,a) the transition kernel, R(s,a,s') the reward function, and γ ∈ [0,1] the discount factor. Concrete implementation begins with defining the environment and its spaces.

Using gymnasium (the maintained successor to OpenAI Gym) provides a standardized interface for this:

import gymnasium as gym


env = gym.make("CartPole-v1", max_episode_steps=500)

# Reset with a deterministic seed for reproducible starts
obs, info = env.reset(seed=42)

# Inspect the spaces that the agent must operate within
print("Observation space:", env.observation_space)
# Discrete(4) – four continuous values representing pole angle, cart position, etc.
print("Action space:", env.action_space)
# Discrete(2) – 0 = push left, 1 = push right

The observation space defines what the agent can sense; the action space defines what it can do. In practice, S is often a Box (multivariate continuous) or Discrete (finite categories), and A follows the same classification. The transition kernel P is implicit: each env.step(a) returns (next_obs, reward, terminated, truncated, info), where terminated marks a goal or failure state and truncated marks step‑limit or timeout termination.

A common immediate failure is mis‑specifying the environment ID: gymnasium.error.UnrecognizedEnvID is raised if the string does not match a registered ID. Similarly, passing incompatible max_episode_steps for environments that enforce a hard limit can cause the episode to end prematurely without a non‑zero reward, producing an apparently empty learning signal.

Represent a Policy

The policy π(a|s) maps observations to action selection probabilities. In practice this is a parametric function, most commonly a neural network. Below is a minimal PyTorch policy that outputs a categorical distribution over actions:

import torch
import torch.nn as nn
import torch.distributions as dist

class PolicyNet(nn.Module):
    def __init__(self, obs_dim, n_act):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(obs_dim, 64),
            nn.ReLU(),
            nn.Linear(64, 64),
            nn.ReLU(),
            nn.Linear(64, n_act)
        )

    def forward(self, x):
        # Outputs raw logits; callers apply softmax or construct a distribution
        return self.net(x)

# Initialise with dimensions from the environment
policy = PolicyNet(env.observation_space.shape[0], env.action_space.n)

The network takes the observation vector as input and emits n_act logits. These logits are not probabilities yet; they are passed to torch.distributions.Categorical to sample actions and compute log‑probabilities. This pattern—network → logits → distribution → sample → log‑prob—is the backbone of policy‑gradient methods, actor‑critic architectures, and Q‑learning with function approximation.

If the observation space is Box (continuous), the first Linear layer automatically accepts the full vector size. If it is Discrete, the observation is typically one‑hot or embedded prior to the network. The key invariant is that the policy’s output dimension must equal env.action_space.n for discrete actions or env.action_space.shape[0] for continuous action spaces.

Train the Policy

Policy‑gradient methods maximize the expected cumulative return J(θ) = E[Σ γᵗ rₜ] by estimating the gradient ∇θ J(θ). A REINFORCE‑style update uses sampled returns as the learning signal. The following loop demonstrates a complete training iteration with a single episode:

import numpy as np

gamma = 0.99          # discount factor
lr = 3e-4             # learning rate for the policy optimizer
optimizer = torch.optim.Adam(policy.parameters(), lr=lr)

for episode in range(300):
    obs, terminated = env.reset()
    log_probs = []
    rewards = []
    steps = 0

    while not terminated and steps < 500:
        obs_tensor = torch.as_tensor(obs, dtype=torch.float32)
        logits = policy(obs_tensor)
        action_dist = dist.Categorical(logits=logits)
        action = action_dist.sample()
        log_prob = action_dist.log_prob(action)

        next_obs, reward, terminated, truncated, info = env.step(action.item())

        log_probs.append(log_prob)
        rewards.append(reward)
        obs = next_obs
        steps += 1

    # Compute the discounted return G for the entire episode
    G = 0.0
    for r in reversed(rewards):
        G = r + gamma * G

    # REINFORCE loss: negative return weighted by log‑probability
    # Here we use the same return G for every step in the episode (monte‑carlo)
    loss = -sum(G * lp for lp in log_probs) / len(log_probs)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

The loop collects log‑probabilities and rewards per step, computes a single discounted return G for the episode, and performs a gradient step that pushes probability mass toward actions that contributed to higher returns. In practice, multiple episodes are collected into a batch before the optimizer step, which reduces variance, but the core computation remains identical.

A subtle sharp edge: if the episode terminates before any non‑zero reward is received (common in sparse‑reward domains), G will be zero for every step, and the gradient update becomes a no‑op. The policy never receives a non‑zero signal, and learning stalls silently. This is not a code bug; it is the expected behavior when the reward function does not provide intermediate shaping.

Evaluate the Agent

After training, the agent must be evaluated in a fresh environment to estimate the policy’s actual performance. The evaluation loop disables exploration (deterministic action selection) and records cumulative return per episode:

def evaluate(env, policy, episodes=10, deterministic=True):
    returns = []
    for _ in range(episodes):
        obs, terminated = env.reset()
        episode_return = 0.0
        while not terminated:
            obs_tensor = torch.as_tensor(obs, dtype=torch.float32)
            logits = policy(obs_tensor)
            if deterministic:
                action = torch.argmax(logits).item()   # greedy selection
            else:
                action_dist = dist.Categorical(logits=logits)
                action = action_dist.sample().item()
            obs, reward, terminated, truncated, info = env.step(action)
            episode_return += reward
        returns.append(episode_return)
    return sum(returns) / len(returns)

evaluate returns the mean episode return across the requested number of rollouts. Running this after every N training episodes gives a learning curve without requiring a separate validation set.

Sharp Edges What Silently Fails

Integration and Compute Considerations

RL training is inherently iterative and sample‑intensive. In production‑adjacent settings, the training loop is often wrapped in a job scheduler or containerized runtime. A typical pattern uses docker to encapsulate the Python environment, systemctl (or an equivalent service manager) to restart training jobs after preemptions, and kubectl to submit distributed training pods when the algorithm scales beyond a single machine (e.g., PPO with multiple workers). The configuration keys that control these systems—docker run --gpus all, systemctl restart rl‑trainer, kubectl apply -f job.yaml—are platform‑specific; the patterns shown here map directly to any container‑orchestrated environment without invoking vendor‑internal mechanisms.

When the training job finishes, the resulting policy parameters (state‑dict checkpoint) are persisted to object storage or a version‑controlled artifact registry. Loading a checkpoint for inference follows the same import‑and‑forward pattern shown earlier, with the added step of policy.load_state_dict(torch.load("policy.pt")).

Summary

Reinforcement learning fundamentals reduce to four repetitive steps: (1) reset the environment and observe the initial state; (2) select an action via the current policy, collect the reward and next state; (3) update the policy using the observed return as the learning signal; (4) evaluate the updated policy on fresh rollouts. The concrete machinery—gymnasium for environment interaction, torch for parametric policy representation, and a policy‑gradient update rule—provides a working starting point for any sequential‑decision domain. The sharp edges listed above are the most common reasons RL projects appear to stall or produce degenerate policies after the initial prototype succeeds; recognizing them before they manifest in a production rollout saves significant remediation effort.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.