Notes on Sutton and Barto's Reinforcement Learning: An Introduction (Chapter 1)

Essential Ideas

• Essential idea is goal directed learning
• True idea of learning through conditioning
  • Maximisation of a numerical reward signal systems actions
  • influence later outputs
  • Affect subsequent outputs
• Consequences of actions, reward signals, extended time periods - main three features of RL

Markov Decision Processes

• Markov decisions processes
  • Real problem an agent is facing while interacting with an environment to achieve a goal

Differences from Supervised and Unsupervised Learning

• How Supervised learning is different
  • Learning from a training set of labeled examples given by a knowledgeable external supervisor
  • Gives agent correct action tot take for that situation
  • objective is for system to extrapolate, generalise
  • not adequate for learning from interaction
  • because agent needs to learn from own tesdt and experience — not be supervised
• Also different from unsupervised learning
  • The agent tries to find structures in un-labeled data, unlike supervised where clear instructions are given
  • RL tries to maximise and amplify a successful reward signal

Exploration vs. Exploitation

• Trade off between exploring a new decision and exploiting a known decision
  • Grandmaster chess move is an example

Elements of Reinforcement Learning

• Elements of RL
  • Policy - learning agent’s behavior at a given time (stimulus - response rules)
  • Reward Signal - goal of the RL problem at hand (a single number - reward) (like pleasure or pain)
  • Value function - A reward signal but for the long-run
    • Total amount of reward agent can get in the future starting from that state.
  • Model (of the environment)
    • Inferences to be made about how an environment behaves (mimics it)
  • Temporal difference learning method
    • Rate of learning (step-size parameter) while keeping current value of earlier state close to value of later state