🎮LeMario Learns Super Mario Bros Dynamics From Pixels
Model learns game dynamics without knowing how to win
TL;DR
LeMario, a model trained on Super Mario Bros., uses JEPA to learn world dynamics from pixels and actions but struggles with actual gameplay progress. It excels in reward-free planning, beating baselines by over 45%. The architecture includes AdaLN-Zero for action injection.
Google just killed Kubernetes pricing as we know it. LeMario is a model trained on Super Mario Bros., using JEPA to learn world dynamics from pixels and actions but not how to actually make progress in the game. It excels at reward-free planning, beating persistence and shuffled action baselines by 45% and 47%, respectively. The architecture includes AdaLN-Zero for injecting actions into transformer blocks, starting with zero weights to gradually learn which gates to open during training. LeMario was trained on 737,134 frames from 280 episodes across 32 levels.
Key Points
LeMario uses JEPA to learn world dynamics from pixels and actions, trained on 737,134 frames
Model beats persistence baseline by 45% and shuffled action baseline by 47%
AdaLN-Zero injects actions into transformer blocks with zero weights initially
Cross-Entropy Method (CEM) used to search through imagination for optimal actions
Latent is a 192-number representation, probe recovers Mario's horizontal position
Why It Matters
If you're working on reward-free planning or reinforcement learning in complex environments like video games, LeMario shows promise. It excels at predicting game dynamics but falls short in actual gameplay progress, highlighting the gap between understanding and execution.
Frequently Asked Questions
Why does this matter?
If you're working on reward-free planning or reinforcement learning in complex environments like video games, LeMario shows promise. It excels at predicting game dynamics but falls short in actual gameplay progress, highlighting the gap between understanding and execution.
What happened?
LeMario, a model trained on Super Mario Bros., uses JEPA to learn world dynamics from pixels and actions but struggles with actual gameplay progress. It excels in reward-free planning, beating baselines by over 45%. The architecture includes AdaLN-Zero for action injection.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.