My research spans the areas of natural language processing and machine learning. At the current stage, I am particularly interested advanced stimulation of language models. My research aims to develop theory, tools, and algorithms to effectively and efficiently drive language models (especially the large ones), and also establish a deeper understanding by observing the behaviors of models.
Selected Papers
Parameter-efficient Fine-tuning of Large-scale Pre-trained Language Models
Ning Ding,
Yujia Qin,
Guang Yang,
Fuchao Wei,
Zonghan Yang,
Yusheng Su,
Shengding Hu,
Yulin Chen,
Chi-Min Chan,
Weize Chen,
Jing Yi,
Weilin Zhao,
Zhiyuan Liu,
Hai-Tao Zheng,
Jianfei Chen,
Yang Liu,
Jie Tang,
Juanzi Li,
and Maosong Sun
Nature Machine Intelligence Cover Article of Nature Machine Intelligenceās March Issue World Artificial Intelligence Conference Youth Outstanding Paper Award
As pre-trained language models (PLMs) have become the fundamental infrastructure for various NLP tasks and researchers have readily enjoyed themselves in the pretrainingfinetuning paradigm, evidence from emerging research has continuously proven that larger models tend to yield better performance. However, despite the welcome outcome, the process of fine-tuning large-scale PLMs brings prohibitive adaptation costs. In fact, finetuning all the parameters of a colossal model and retaining separate instances for different tasks are practically infeasible. This necessitates a new branch of research focusing on the parameter-efficient adaptation of PLMs. In order to unleash the imagination of the possible advantages of such methods, not limited to parameter efficiency, we coined a new term delta tuning from a morphological point of view to refer to the original āparameter efficient tuningā. In contrast with the standard fine-tuning, delta tuning only fine-tunes a small portion of the model parameters while keeping the rest untouched, largely reducing both the computation and storage costs. Recent studies have demonstrated that a series of delta tuning methods with distinct tuned parameter selection could achieve performance on a par with full-parameter fine-tuning, suggesting a new promising way of stimulating large-scale PLMs. In this paper, we first formally describe the problem of delta tuning and then comprehensively review recent delta tuning approaches. We also propose a unified categorization criterion that divides existing delta tuning methods into three groups: addition-based, specification-based, and reparameterization-based methods. Though initially proposed as an efficient method to steer large models, we believe that some of the fascinating evidence discovered along with delta tuning could help further reveal the mechanisms of PLMs and even deep neural networks. To this end, we discuss the theoretical principles underlying the effectiveness of delta tuning and propose frameworks to interpret delta tuning from the perspective of optimization and optimal control, respectively. Furthermore, we provide a holistic empirical study of representative methods, where results on over 100 NLP tasks demonstrate a comprehensive performance comparison of different approaches. The experimental results also cover the analysis of combinatorial, scaling and transferable properties of delta tuning. To facilitate the research of delta tuning, we are also developing an open-source toolkit, OpenDelta , that enables practitioners to efficiently and flexibly implement delta tuning on PLMs. At last, we discuss a series of real-world applications of delta tuning.
@article{2023:delta,title={Parameter-efficient Fine-tuning of Large-scale Pre-trained Language Models},titleb={Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models},preview={NMI.jpg},bibtex_show={true},abbr={Nat. <br>Mach. <br>Intell.},code={https://github.com/thunlp/OpenDelta},author={Ding, Ning and Qin, Yujia and Yang, Guang and Wei, Fuchao and Yang, Zonghan and Su, Yusheng and Hu, Shengding and Chen, Yulin and Chan, Chi-Min and Chen, Weize and Yi, Jing and Zhao, Weilin and Liu, Zhiyuan and Zheng, Hai-Tao and Chen, Jianfei and Liu, Yang and Tang, Jie and Li, Juanzi and Sun, Maosong},journal={Nature Machine Intelligence},selected={true},arxiv={2203.06904},html={https://www.nature.com/articles/s42256-023-00626-4},pdf={https://www.nature.com/articles/s42256-023-00626-4},award={ <br> <strong> <font color="BB0A21"> Cover Article of Nature Machine Intelligence's March Issue </font></strong> <br> <strong> <font color="BB0A21"> World Artificial Intelligence Conference Youth Outstanding Paper Award </font></strong>},series={\newlin Nature Machine Intelligence},year={2023}}
Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
Ning Ding,
Yulin Chen,
Bokai Xu,
Yujia Qin,
Shengding Hu,
Zhiyuan Liu,
Maosong Sun,
and Bowen Zhou
@inproceedings{preprint:ultra,title={Enhancing Chat Language Models by Scaling High-quality Instructional Conversations},bibtex_show={true},author={Ding, Ning and Chen, Yulin and Xu, Bokai and Qin, Yujia and Hu, Shengding and Liu, Zhiyuan and Sun, Maosong and Zhou, Bowen},booktitle={EMNLP},selected={true},pdf={https://arxiv.org/abs/2305.14233},code={https://github.com/thunlp/UltraChat},series={\newlin EMNLP},year={2023}}
OpenPrompt: An Open-source Framework for Prompt-learning
Ning Ding,
Shengding Hu,
Weilin Zhao,
Yulin Chen,
Zhiyuan Liu,
Hai-Tao Zheng,
and Maosong Sun
ACL System Demonstration
2022
Best Demo Paper Award
Prompt-learning has become a new paradigm in modern natural language processing, which directly adapts pre-trained language models (PLMs) to cloze-style prediction, autoregressive modeling, or sequence to sequence generation, resulting in promising performances on various tasks. However, no standard implementation framework of prompt-learning is proposed yet, and most existing prompt-learning codebases, often unregulated, only provide limited implementations for specific scenarios. Since there are many details such as templating strategy, initializing strategy, and verbalizing strategy, etc. need to be considered in prompt-learning, practitioners face impediments to quickly adapting the desired prompt learning methods to their applications. In this paper, we present OpenPrompt, a unified easy-to-use toolkit to conduct prompt-learning over PLMs. OpenPrompt is a research-friendly framework that is equipped with efficiency, modularity, and extendibility, and its combinability allows the freedom to combine different PLMs, task formats, and prompting modules in a unified paradigm. Users could expediently deploy prompt-learning frameworks and evaluate the generalization of them on different NLP tasks without constraints.
@inproceedings{ding2021openprompt,preview={op.png},bibtex_show={true},title={ OpenPrompt: An Open-source Framework for Prompt-learning},author={Ding, Ning and Hu, Shengding and Zhao, Weilin and Chen, Yulin and Liu, Zhiyuan and Zheng, Hai-Tao and Sun, Maosong},year={2022},selected={true},booktitle={ACL System Demonstration},series={\newlin ACL System Demonstration},html={https://arxiv.org/abs/2111.01998},pdf={https://arxiv.org/pdf/2111.01998.pdf},code={https://github.com/thunlp/OpenPrompt},award={ <br> <img src="/assets/img/award.png" width="15px"> <strong> <font color="BB0A21"> Best Demo Paper Award </font></strong>}}
Few-NERD: A Few-shot Named Entity Recognition Dataset
Ning Ding,
Guangwei Xu,
Yulin Chen,
Xiaobin Wang,
Xu Han,
Pengjun Xie,
Hai-Tao Zheng,
and Zhiyuan Liu
Recently, considerable literature has grown up around the theme of few-shot named entity recognition (NER), but little published benchmark data specifically focused on the practical and challenging task. Current approaches collect existing supervised NER datasets and re-organize them to the few-shot setting for empirical study. These strategies conventionally aim to recognize coarse-grained entity types with few examples, while in practice, most unseen entity types are fine-grained. In this paper, we present Few-NERD, a large-scale human-annotated few-shot NER dataset with a hierarchy of 8 coarse-grained and 66 fine-grained entity types. Few-NERD consists of 188,238 sentences from Wikipedia, 4,601,160 words are included and each is annotated as context or a part of a two-level entity type. To the best of our knowledge, this is the first few-shot NER dataset and the largest human-crafted NER dataset. We construct benchmark tasks with different emphases to comprehensively assess the generalization capability of models. Extensive empirical results and analysis show thatFew-NERD is challenging and the problem requires further research. We make Few-NERD public at https://ningding97.github.io/fewnerd/
@inproceedings{ding2021few,preview={few-nerd-heat.png},title={Few-NERD: A Few-shot Named Entity Recognition Dataset},author={Ding, Ning and Xu, Guangwei and Chen, Yulin and Wang, Xiaobin and Han, Xu and Xie, Pengjun and Zheng, Hai-Tao and Liu, Zhiyuan},booktitle={ACL},series={\newlin ACL},bibtex_show={true},award={ <br> <strong> <font color="BB0A21"> Oral Presentation </font></strong>},selected={true},html={https://ningding97.github.io/fewnerd/},pdf={https://arxiv.org/pdf/2105.07464.pdf},code={https://github.com/thunlp/Few-NERD},year={2021}}
Prototypical Representation Learning for Relation Extraction
Ning Ding,
Xiaobin Wang,
Yao Fu,
Guangwei Xu,
Rui Wang,
Pengjun Xie,
Ying Shen,
Fei Huang,
Hai-Tao Zheng,
and Rui Zhang
International Conference on Learning Representations, ICLR
2021
Recognizing relations between entities is a pivotal task of relational learning. Learning relation representations from distantly-labeled datasets is difficult because of the abundant label noise and complicated expressions in human language. This paper aims to learn predictive, interpretable, and robust relation representations from distantly-labeled data that are effective in different settings, including supervised, distantly supervised, and few-shot learning. Instead of solely relying on the supervision from noisy labels, we propose to learn prototypes for each relation from contextual information to best explore the intrinsic semantics of relations. Prototypes are representations in the feature space abstracting the essential semantics of relations between entities in sentences. We learn prototypes based on objectives with clear geometric interpretation, where the prototypes are unit vectors uniformly dispersed in a unit ball, and statement embeddings are centered at the end of their corresponding prototype vectors on the surface of the ball. This approach allows us to learn meaningful, interpretable prototypes for the final classification. Results on several relation learning tasks show that our model significantly outperforms the previous state-of-the-art models. We further demonstrate the robustness of the encoder and the interpretability of prototypes with extensive experiments.
@inproceedings{ding2021prototypical,preview={proto.png},title={Prototypical Representation Learning for Relation Extraction},author={Ding, Ning and Wang, Xiaobin and Fu, Yao and Xu, Guangwei and Wang, Rui and Xie, Pengjun and Shen, Ying and Huang, Fei and Zheng, Hai-Tao and Zhang, Rui},booktitle={International Conference on Learning Representations, <br> ICLR},year={2021},bibtex_show={true},series={\newlin ICLR},selected={true},html={https://openreview.net/forum?id=aCgLmfhIy_f},pdf={https://openreview.net/forum?id=aCgLmfhIy_f},code={https://github.com/Alibaba-NLP/ProtoRE}}
Parameter-efficient Fine-tuning of Large-scale Pre-trained Language Models 
Best Demo Paper Award
