Learning a better initialization for soft prompts via meta-learning

Download paper here

Prompt tuning (PT) is an effective approach to adapting pre-trained language models to downstream tasks. Without a good initialization, prompt tuning doesn't perform well under few-shot settings. So pre-trained prompt tuning (PPT) is proposed to initialize prompts by leveraging pre-training data. We propose MetaPT (Meta-learned Prompt Tuning) to further improve PPT's initialization by considering latent structure within the pre-training data. Specifically, we introduce the structure by first clustering pre-training data into different auxiliary tasks with unsupervised methods. Then we use these tasks to pre-train prompts with a meta-learning algorithm. Such a process can make prompts learn a better initialization by discovering commonalities among these auxiliary tasks. We evaluate our method on seven downstream tasks. Our MetaPT achieves better and more stable performance than the state-of-the-art method.

Recommended citation: @article{Huang2022LearningAB, title={Learning a Better Initialization for Soft Prompts via Meta-Learning}, author={Yukun Huang and Kun Qian and Zhou Yu}, journal={ArXiv}, year={2022}, volume={abs/2205.12471}, url={https://api.semanticscholar.org/CorpusID:249062905} }