More brain-inspired neural networks with Neuromodulation
While Artificial Neural Networks are inspired by the brain, there are many important mechanisms in the brain that most ANNs leave out. Neuroevolution gives us the possibility to explore the effect of some of these. An example is neuromodulation: Certain signals in the brain are known to modulate areas in various ways, for instance temporarily increasing or decreasing the learning capability in a brain region. Some work in neuroevolution has explored the possibilities of such neuromodulated neural networks, but there is much more work to be done.
Neuromodulated learning as illustrated here. Here, the neuromodulated neuron adjusts the learning rate of the indicated connections. Thereby, this neural network can adjust its learning rate to the current context, allowing it to learn when needed but at other times remain stable.
Learning at different timescales. We know that in the brain, learning happens across different timescales (some is fast, tracking the learning signal instantly, while other is slower, averaging out noise in the training signal). These two styles of learning can be implemented as so-called activity-gating (a signal that for a moment turns a neuron on or off) and plasticity-gating (a signal that temporarily changes learning rates) neuromodulation, respectively.
Some work has been done on implementing both of these styles of neuromodulation, but I am not familiar with any work that tries to combine them. Could a combination give the benefits of both quick behavior adaptation when needed and more long-lasting learned behavior change? A good starting point could be to implement both styles of neuromodulation and let an evolutionary algorithm optimize neural nets, deciding which style to use where. Would we see "fast" adaptation evolve for some tasks (e.g. tasks where the environment switches between different phases, requiring certain behaviors to be triggered at different times) and "slow" adaptation for others (e.g. tasks where the agent needs to adapt its strategy to observations it makes, such as the T-Maze below).
A T-Maze, famous from experiments in animal cognition. The agent needs to learn to seek out the highest reward, by observing its location and repeatedly seeking it out. After a few trials the rewards are swapped, and we can test if the agent is able to adjust its strategy. Plasticity-gating neuromodulation can solve this by having the agent learn to adjust its strategy depending on which reward it observes. Image from here.
Neuromodulated Learning and Deep Learning. There has recently been some work combining Deep Learning and Neuroevolution. The idea behind this is that DL is very good at extracting features from high-dimensional data such as images, but struggles with solving sparse-reward Reinforcement Learning tasks. Neuroevolution can help, by taking the features extracted by DL and learning how to act based on them (see figure below). Perhaps could this symbiosis be even more efficient by combining DL with neuromodulated neuroevolution? Neuromodulation is known to help evolving agents learn from rewards. Could the power of Deep Learning help bring neuromodulated learning to problems with high-dimensional inputs, for instance, a T-Maze with pixel inputs?
Combining Deep Learning and Neuroevolution. Image from here.
Publisert29. juni 2021 13:21
- Sist endret29. juni 2021 13:23
