"Fine-tuning" refers to the process of continuing to train language models (such as BERT or XLM), but this time with task-specific data, instead of just language modelling. While some studies have shown that this can be useful [1], there are many unanswered questions about what precisely changes within the language model when we fine-tune.
Does the language's notion of syntax or semantics change? What happens when we introduce other languages into the mix? Do different components in the language model contribute differently to fine-tuning? Merchant et al. run a variety of probes to try and quantify the differences between language models before and after fine-tuning [2]; this is a very general paper, and more specificity in one particular direction could be interesting.
Recommended reading:
- Peters, M. E., Ruder, S., & Smith, N. A. (2019). To tune or not to tune? Adapting Pretrained Representations to Diverse Tasks
- Merchant, A., Rahimtoroghi, E., Pavlick, E., & Tenney, I. (2020). What Happens To BERT Embeddings During Fine-tuning?
- Lauscher, A., Ravishankar, V., Vulić, I., & Glavaš, G. (2020). From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers.