There is a recent development of a more flexible and adaptive machine learning model for language processing, which employs a technique of periodic forgetting during training. This approach involves selectively erasing the embedding layer of a neural network, which contains tokens specific to a particular language, and then retraining the model with a new language.
This method allows the model to adapt more easily to new languages, especially when computational resources or training data are limited. It contrasts with traditional models that require extensive training data and computing power, and are less adaptable to changes or additions of languages. The article also draws parallels between this method and human memory processes, suggesting that adaptive forgetting could lead to more human-like and versatile AI language models.
In human cognition, memory is not just about retaining detailed information accurately; rather, it often involves remembering the general gist or essence of experiences. This process entails abstracting and extrapolating from specific details, which allows for flexibility in thinking and learning.
Applying a similar concept to AI, the method of adaptive forgetting in machine learning models mimics this aspect of human memory. By periodically resetting the embedding layer of a neural network during training, the model doesn’t just rigidly store specific information. Instead, it becomes more adept at abstracting and generalizing knowledge. This allows the AI to adapt more efficiently to new tasks or languages later on, just as humans can apply general principles or concepts learned from past experiences to new situations.
The idea is that, by incorporating mechanisms similar to human forgetting and relearning, AI models can achieve a more flexible and versatile understanding of language. This approach suggests that the depth of language comprehension in AI goes beyond just the meanings of individual words to encompass more abstract and conceptual understanding, much like how humans comprehend and use language.