Transformers learn through gradual rank increase

作者：Enric Boix-Adserà, Etai Littwin, Emmanuel Abbé, Samy Bengio, Joshua M. Susskind · 发表于：arXiv (Cornell University) · 年份：2023 · DOI：10.48550/arxiv.2306.07042 · 被引用次数：5 · 研究领域：Neural Networks and Applications、Blind Source Separation Techniques、Machine Learning and ELM

We identify incremental learning dynamics in transformers, where the difference between trained and initial weights progressively increases in rank. We rigorously prove this occurs under the simplifying assumptions of diagonal weight matrices and small initialization. Our experiments support the theory and also show that phenomenon can occur in practice without the simplifying assumptions.