"Previous attempts to combine RL with neural networks had largely failed due to unstable learning. To address these instabilities, our Deep Q-Networks (DQN) algorithm stores all of the agent's experiences and then randomly samples and replays these experiences to provide diverse and decorrelated training data."
IIRC there is evidence rats replay their experiences, sped up, while they are asleep. Dreaming may be something else entirely though, because my dreams aren't anything like my memories of the day before.
Artificial neural networks can "dream" by predicting what frame it will see next. This is a really cool technique. They've shown slightly blurry videos of atari games being played that comes entirely from the network's dream. With no interaction with the game at all. You can even train the reinforcement learning on the dream sequences and improve it's performance.
But this also doesn't seem quite like what human dreams are. Human dreams are wild and unrealistic, while the NN dreams try to match the training data as closely as possible.
Quite a few of my dreams seem to be slightly scary situations (dangerous behavior of other drivers while I'm driving, for example).
This makes me think that we have a "scenario generation" engine as part of our brains, and so we're being presented with novel experiences (often "wild and unrealistic") to react to.
So this seems to be a pretty big step beyond the idea of presenting previously seen situations (which sounds like a really good first step, of course).
Reinforcement learning has a problem in that it gets very little labelled data. You may have a million frames, but the only label is the score. Which may only change a few times per game.
Training the net to predict the next frame is sort of unsupervised learning. It can learn the rules of the game without score information at all.
The second thing is that RL is different than prediction. Even if you can predict the next frame exactly, finding the optimal set of moves is still a hard problem. The algorithm needs to learn more than just predicting what will happen, but also what the optimal action is in every situation. That is something that can be practiced in simulations, or "dreams".
i have read somewhere the dreams are for training purposes and present the dreamer situations either from the past or present or combination. that would explain why dreams are mostly dangerous or "wild". this is how our mind tries to figure out how to deal with those situations in the future. good examples of this are same recurring dreams over and over. and also i think there is no sense of time in the brain while dreaming so it does not differentiate childhood "situations" from any current ones. It just keeps fucking trying to solve that damn math exam problem....
The comparison to dreaming reminds me of a comment in Information Theory, Inference, and Learning Algorithms:
"One way of viewing the two terms in the gradient (43.9) is as 'waking' and 'sleeping' rules. While the network is 'awake', it measures the correlation between x_i and x_j in the real world, and weights are increased in proportion. While the network is 'asleep', it 'dreams' about the world using the generative model (43.4), and measures the correlations between x_i and x_j in the model world; these correlations determine a proportional decrease in the weights. If the second-order correlations in the dream world match the correlations in the real world, then the two terms balance and the weights do not change."
... so, made the machines dream. Fancy!