balancing a pendulum, the hard way

a 15 kg block on a 10 metre rail, a 2 metre arm, a 5 kg bob on the end, and a motor that pushes with up to 100 newtons. physics runs 240 times a second. the agent gets a say 30 times a second, and that number isn't a guess: an upright pendulum this long starts to fall in about half a second, and catching it takes ten-odd decisions in that time.

two modes. easy starts almost upright, so it's just balancing. normal starts hanging straight down, so it has to swing itself up first. that's where things got interesting (read: stalled).

the first reward paid the running total of time spent upright, every step. so step 100 paid a hundred times what step 1 did, and an episode's score grew with the square of its length. oops.

spinning averages half the reward for zero skill.

the fixed one scores one step at a time: mostly how upright the arm is, minus a little for drifting toward the walls, minus a little for spinning. that last one is there because a pendulum whirling round and round is above the rail half the time, and spinning averages half the reward for zero skill.

none of it gets checked by staring at numbers. it gets checked by ordering: upright and still has to beat hanging, the middle has to beat the wall, still has to beat spinning. the first version rewarded spinning (a sign error) and failed that test straight away.

first real learner: tabular Q-learning. chop every number the agent sees into bins, keep a table of how good each push is from each bin, nudge the table after every step.

with eight angle bins, the whole region where balancing is even possible fit inside two of them. so to the agent, nearly upright and about to fall looked identical. thirty-six bins fixed that, and it solved easy mode.

also, on a fresh all-zero table, picking the best push always picked the first one: full left. it pushed left 80% of the time until i broke ties at random.

normal mode was another story. on 100,000 episodes of the swing-up, the best it ever managed was 12.0 seconds above the rail, and by the end it was averaging 2.1. one table has to cover a big, coarse swing and fine control at the top, and every extra bin multiplies the table.

next up, Monte Carlo control. instead of updating after every step, it plays a whole episode, then walks back through it and credits every push with what actually happened afterwards. no guessing about the future, it just waits and finds out.

it solved easy mode inside 1,000 episodes: every one of its last 100 held for the full 10.0 seconds. the Q-learning run i'm comparing it to was still averaging 5.7 at the end of 10,000. (they don't use the same bins or the same number of pushes yet, so call it a promising heat, not a fair race.)

on the swing-up it gets above the rail and holds it for up to 10.3 seconds, averaging 2.8 by the end. better, not solved.

the original plan was model-based RL: learn the physics, then plan through it. that's still the next stop.

a 15 kg block on a 10 metre rail, a 2 metre arm, a 5 kg bob on the end, and a motor that pushes with up to 100 newtons. physics runs 240 times a second. the agent gets a say 30 times a second, and that number isn't a guess: an upright pendulum this long starts to fall in about half a second, and catching it takes ten-odd decisions in that time.

two modes. easy starts almost upright, so it's just balancing. normal starts hanging straight down, so it has to swing itself up first. that's where things got interesting (read: stalled).

the first reward paid the running total of time spent upright, every step. so step 100 paid a hundred times what step 1 did, and an episode's score grew with the square of its length. oops.

spinning averages half the reward for zero skill.

the fixed one scores one step at a time: mostly how upright the arm is, minus a little for drifting toward the walls, minus a little for spinning. that last one is there because a pendulum whirling round and round is above the rail half the time, and spinning averages half the reward for zero skill.

none of it gets checked by staring at numbers. it gets checked by ordering: upright and still has to beat hanging, the middle has to beat the wall, still has to beat spinning. the first version rewarded spinning (a sign error) and failed that test straight away.

first real learner: tabular Q-learning. chop every number the agent sees into bins, keep a table of how good each push is from each bin, nudge the table after every step.

with eight angle bins, the whole region where balancing is even possible fit inside two of them. so to the agent, nearly upright and about to fall looked identical. thirty-six bins fixed that, and it solved easy mode.

also, on a fresh all-zero table, picking the best push always picked the first one: full left. it pushed left 80% of the time until i broke ties at random.

normal mode was another story. on 100,000 episodes of the swing-up, the best it ever managed was 12.0 seconds above the rail, and by the end it was averaging 2.1. one table has to cover a big, coarse swing and fine control at the top, and every extra bin multiplies the table.

next up, Monte Carlo control. instead of updating after every step, it plays a whole episode, then walks back through it and credits every push with what actually happened afterwards. no guessing about the future, it just waits and finds out.

it solved easy mode inside 1,000 episodes: every one of its last 100 held for the full 10.0 seconds. the Q-learning run i'm comparing it to was still averaging 5.7 at the end of 10,000. (they don't use the same bins or the same number of pushes yet, so call it a promising heat, not a fair race.)

on the swing-up it gets above the rail and holds it for up to 10.3 seconds, averaging 2.8 by the end. better, not solved.

the original plan was model-based RL: learn the physics, then plan through it. that's still the next stop.

a 15 kg block on a 10 metre rail, a 2 metre arm, a 5 kg bob on the end, and a motor that pushes with up to 100 newtons. physics runs 240 times a second. the agent gets a say 30 times a second, and that number isn't a guess: an upright pendulum this long starts to fall in about half a second, and catching it takes ten-odd decisions in that time.

two modes. easy starts almost upright, so it's just balancing. normal starts hanging straight down, so it has to swing itself up first. that's where things got interesting (read: stalled).

the first reward paid the running total of time spent upright, every step. so step 100 paid a hundred times what step 1 did, and an episode's score grew with the square of its length. oops.

spinning averages half the reward for zero skill.

the fixed one scores one step at a time: mostly how upright the arm is, minus a little for drifting toward the walls, minus a little for spinning. that last one is there because a pendulum whirling round and round is above the rail half the time, and spinning averages half the reward for zero skill.

none of it gets checked by staring at numbers. it gets checked by ordering: upright and still has to beat hanging, the middle has to beat the wall, still has to beat spinning. the first version rewarded spinning (a sign error) and failed that test straight away.

first real learner: tabular Q-learning. chop every number the agent sees into bins, keep a table of how good each push is from each bin, nudge the table after every step.

with eight angle bins, the whole region where balancing is even possible fit inside two of them. so to the agent, nearly upright and about to fall looked identical. thirty-six bins fixed that, and it solved easy mode.

also, on a fresh all-zero table, picking the best push always picked the first one: full left. it pushed left 80% of the time until i broke ties at random.

normal mode was another story. on 100,000 episodes of the swing-up, the best it ever managed was 12.0 seconds above the rail, and by the end it was averaging 2.1. one table has to cover a big, coarse swing and fine control at the top, and every extra bin multiplies the table.

next up, Monte Carlo control. instead of updating after every step, it plays a whole episode, then walks back through it and credits every push with what actually happened afterwards. no guessing about the future, it just waits and finds out.

it solved easy mode inside 1,000 episodes: every one of its last 100 held for the full 10.0 seconds. the Q-learning run i'm comparing it to was still averaging 5.7 at the end of 10,000. (they don't use the same bins or the same number of pushes yet, so call it a promising heat, not a fair race.)

on the swing-up it gets above the rail and holds it for up to 10.3 seconds, averaging 2.8 by the end. better, not solved.

the original plan was model-based RL: learn the physics, then plan through it. that's still the next stop.