A rational agent is one that, for each percept history, selects the action that maximizes its expected performance measure given what it knows.
Four pieces
To define rationality precisely you need:
1. Performance measure: how outcomes are scored. 2. Prior knowledge: what the agent knows before acting. 3. Possible actions. 4. Percept history: what it has observed so far.
Rationality is not omniscience. A rational agent can take an action that turns out badly if the action was the best bet given what was known. Rationality is also not pure exploitation — when learning is possible, exploration can be the rational choice.
Limited rationality
Real agents don't have unlimited compute. Bounded rationality asks: what's the best the agent can do given its computational resources? Anytime algorithms, depth-limited search, and meta-reasoning (deciding how long to deliberate) all live in this space.
The objective specification problem
Specifying the right performance measure is often harder than building the agent. A few classical traps:
- Reward hacking: the agent finds an unintended way to score high (a cleaning robot dumps dirt back out, then "cleans" it again).
- Side effects: the objective rewards but doesn't penalize destroying ; the agent destroys to maximize .
- Misaligned proxies: optimizing what's measurable (clicks) rather than what's wanted (long-term satisfaction).
These aren't edge cases — they're systemic. They're the reason "do what I want" is currently an active research problem rather than a solved one.
Multi-objective utility
When there are multiple things to care about, combine into a single utility function. Linear combinations are common but require tradeoff coefficients that capture how much one objective is worth in terms of another. Pareto optimality gives a partial answer when you can't commit to weights.