Reinventing Entropy | Compression & Intelligence Part 1

0:00 – 10:30

The Robot Instruction Problem: When Probability Meets Data

A robot on a distant moon receives movement commands: up, down, left, right. The naive approach uses two bits per instruction (00, 01, 10, 11), but there's a catch — these instructions aren't equally likely. Up comes 50% of the time, down 25%, and left and right each 12.5%. A clever encoding emerges: use a single bit (0) for up, two bits (10) for down, and three bits (110, 111) for left and right. The math is revealing. Half the time you send one bit, a quarter of the time two bits, the rest three bits. Multiply each by its frequency and you get 1.75 bits per instruction, beating the flat two-bit approach. But here's the magic — when you visualize all possible binary strings as a tree diagram, with each layer representing strings of increasing length, the clever encoding consumes exactly the same fraction of space as the probability of each instruction. Up's single bit claims half the diagram, down's two bits claim a quarter, and the three-bit codes each claim an eighth. The probabilities and the space consumption align perfectly, suggesting something fundamental is happening beneath the surface.