Ning Ding | papers

Preprint

Tool Learning with Foundation Models

Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, and et al.

Preprint

PDF CODE Bib
@article{preprint:tool_learning, title = {Tool Learning with Foundation Models}, bibtex_show = {true}, author = {Qin, Yujia and Hu, Shengding and Lin, Yankai and Chen, Weize and Ding, Ning and Cui, Ganqu and Zeng, Zheni and Huang, Yufei and Xiao, Chaojun and Han, Chi and Fung, Yi Ren and Su, Yusheng and Wang, Huadong and {et al.}}, journal = {Preprint}, selected = {false}, pdf = {https://arxiv.org/abs/2304.08354}, series = {\newlin arXiv}, code = {https://github.com/OpenBMB/BMTools/tree/main/bmtools}, year = {Preprint} }
Exploring Low-dimensional Intrinsic Task Subspace via Prompt Tuning

Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, and others

Preprint

ABSTRACT PDF CODE Bib

Why can pre-trained language models (PLMs) learn universal representations and effectively adapt to broad NLP tasks differing a lot superficially? In this work, we empirically find evidence indicating that the adaptations of PLMs to various few-shot tasks can be reparameterized as optimizing only a few free parameters in a unified low-dimensional intrinsic task subspace, which may help us understand why PLMs could easily adapt to various NLP tasks with small-scale data. To find such a subspace and examine its universality, we propose an analysis pipeline called intrinsic prompt tuning (IPT). Specifically, we resort to the recent success of prompt tuning and decompose the soft prompts of multiple NLP tasks into the same low-dimensional nonlinear subspace, then we learn to adapt the PLM to unseen data or tasks by only tuning parameters in this subspace. In the experiments, we study diverse few-shot NLP tasks and surprisingly find that in a 250-dimensional subspace found with 100 tasks, by only tuning 250 free parameters, we can recover 97% and 83% of the full prompt tuning performance for 100 seen tasks (using different training data) and 20 unseen tasks, respectively, showing great generalization ability of the found intrinsic task subspace. Besides being an analysis tool, IPT could further bring practical benefits, such as improving the prompt tuning stability.
@article{qin2021exploring, title = {Exploring Low-dimensional Intrinsic Task Subspace via Prompt Tuning}, author = {Qin, Yujia and Wang, Xiaozhi and Su, Yusheng and Lin, Yankai and Ding, Ning and Liu, Zhiyuan and Li, Juanzi and Hou, Lei and Li, Peng and Sun, Maosong and others}, year = {Preprint}, journal = {Preprint}, series = {\newlin arXiv}, bibtex_show = {true}, pdf = {https://arxiv.org/abs/2110.07867}, code = {https://github.com/thunlp/Intrinsic-Prompt-Tuning} }

2023

Parameter-efficient Fine-tuning of Large-scale Pre-trained Language Models

Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun

Nature Machine Intelligence
Cover Article of Nature Machine Intelligence’s March Issue
World Artificial Intelligence Conference Youth Outstanding Paper Award

ABSTRACT ArXiv Link PDF CODE Bib

As pre-trained language models (PLMs) have become the fundamental infrastructure for various NLP tasks and researchers have readily enjoyed themselves in the pretrainingfinetuning paradigm, evidence from emerging research has continuously proven that larger models tend to yield better performance. However, despite the welcome outcome, the process of fine-tuning large-scale PLMs brings prohibitive adaptation costs. In fact, finetuning all the parameters of a colossal model and retaining separate instances for different tasks are practically infeasible. This necessitates a new branch of research focusing on the parameter-efficient adaptation of PLMs. In order to unleash the imagination of the possible advantages of such methods, not limited to parameter efficiency, we coined a new term delta tuning from a morphological point of view to refer to the original “parameter efficient tuning”. In contrast with the standard fine-tuning, delta tuning only fine-tunes a small portion of the model parameters while keeping the rest untouched, largely reducing both the computation and storage costs. Recent studies have demonstrated that a series of delta tuning methods with distinct tuned parameter selection could achieve performance on a par with full-parameter fine-tuning, suggesting a new promising way of stimulating large-scale PLMs. In this paper, we first formally describe the problem of delta tuning and then comprehensively review recent delta tuning approaches. We also propose a unified categorization criterion that divides existing delta tuning methods into three groups: addition-based, specification-based, and reparameterization-based methods. Though initially proposed as an efficient method to steer large models, we believe that some of the fascinating evidence discovered along with delta tuning could help further reveal the mechanisms of PLMs and even deep neural networks. To this end, we discuss the theoretical principles underlying the effectiveness of delta tuning and propose frameworks to interpret delta tuning from the perspective of optimization and optimal control, respectively. Furthermore, we provide a holistic empirical study of representative methods, where results on over 100 NLP tasks demonstrate a comprehensive performance comparison of different approaches. The experimental results also cover the analysis of combinatorial, scaling and transferable properties of delta tuning. To facilitate the research of delta tuning, we are also developing an open-source toolkit, OpenDelta , that enables practitioners to efficiently and flexibly implement delta tuning on PLMs. At last, we discuss a series of real-world applications of delta tuning.
@article{2023:delta, title = {Parameter-efficient Fine-tuning of Large-scale Pre-trained Language Models}, titleb = {Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models}, preview = {NMI.jpg}, bibtex_show = {true}, abbr = {Nat. Mach. Intell.}, code = {https://github.com/thunlp/OpenDelta}, author = {Ding, Ning and Qin, Yujia and Yang, Guang and Wei, Fuchao and Yang, Zonghan and Su, Yusheng and Hu, Shengding and Chen, Yulin and Chan, Chi-Min and Chen, Weize and Yi, Jing and Zhao, Weilin and Liu, Zhiyuan and Zheng, Hai-Tao and Chen, Jianfei and Liu, Yang and Tang, Jie and Li, Juanzi and Sun, Maosong}, journal = {Nature Machine Intelligence}, selected = {true}, arxiv = {2203.06904}, html = {https://www.nature.com/articles/s42256-023-00626-4}, pdf = {https://www.nature.com/articles/s42256-023-00626-4}, award = { Cover Article of Nature Machine Intelligence's March Issue World Artificial Intelligence Conference Youth Outstanding Paper Award }, series = {\newlin Nature Machine Intelligence}, year = {2023} }
Enhancing Chat Language Models by Scaling High-quality Instructional Conversations

Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou

EMNLP 2023

PDF CODE Bib
@inproceedings{preprint:ultra, title = {Enhancing Chat Language Models by Scaling High-quality Instructional Conversations}, bibtex_show = {true}, author = {Ding, Ning and Chen, Yulin and Xu, Bokai and Qin, Yujia and Hu, Shengding and Liu, Zhiyuan and Sun, Maosong and Zhou, Bowen}, booktitle = {EMNLP}, selected = {true}, pdf = {https://arxiv.org/abs/2305.14233}, code = {https://github.com/thunlp/UltraChat}, series = {\newlin EMNLP}, year = {2023} }
Sparse Low-rank Adaptation of Pre-trained Language Models

Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou

EMNLP 2023

Bib
@inproceedings{preprint:sora, title = {Sparse Low-rank Adaptation of Pre-trained Language Models}, bibtex_show = {true}, author = {Ding, Ning and Chen, Yulin and Xu, Bokai and Qin, Yujia and Hu, Shengding and Liu, Zhiyuan and Sun, Maosong and Zhou, Bowen}, booktitle = {EMNLP}, selected = {false}, series = {\newlin EMNLP}, year = {2023} }
Exploring Lottery Prompts for Pre-trained Language Models

Yulin* Chen, Ning* Ding, Xiaobin Wang, Shengding Hu, Hai-Tao Zheng, Zhiyuan Liu, and Pengjun Xie

ACL 2023

PDF Bib
@inproceedings{2023:lottery, title = {Exploring Lottery Prompts for Pre-trained Language Models}, author = {Chen, Yulin* and Ding, Ning* and Wang, Xiaobin and Hu, Shengding and Zheng, Hai-Tao and Liu, Zhiyuan and Xie, Pengjun}, booktitle = {ACL}, selected = {false}, series = {\newlin ACL}, pdf = {https://arxiv.org/pdf/2305.19500.pdf}, bibtex_show = {true}, year = {2023} }
OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models

Shengding Hu, Ning Ding, Weilin Zhao, Xingtai Lv, Zhen Zhang, Zhiyuan Liu, and Maosong Sun

ACL System Demonstration 2023

PDF CODE Bib
@inproceedings{2023:opendelta, title = {OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models}, author = {Hu, Shengding and Ding, Ning and Zhao, Weilin and Lv, Xingtai and Zhang, Zhen and Liu, Zhiyuan and Sun, Maosong}, booktitle = {ACL System Demonstration}, selected = {false}, series = {\newlin ACL System Demonstration}, pdf = {https://arxiv.org/abs/2307.03084}, code = {https://github.com/thunlp/OpenDelta}, bibtex_show = {true}, year = {2023} }
Few-shot Classification with Hypersphere Modeling of Prototypes

Ning Ding, Yulin Chen, Ganqu Cui, Xiaobin Wang, Hai-Tao Zheng, Zhiyuan Liu, and Pengjun Xie

ACL Findings 2023

ABSTRACT PDF Bib

Metric-based meta-learning is one of the de facto standards in few-shot learning. It composes of representation learning and metrics calculation designs. Previous works construct class representations in different ways, varying from mean output embedding to covariance and distributions. However, using embeddings in space lacks expressivity and cannot capture class information robustly, while statistical complex modeling poses difﬁculty to metric designs. In this work, we use tensor ﬁelds (“areas”) to model classes from the geometrical perspective for few-shot learning. We present a simple and effective method, dubbed as hypersphere prototypes ( HyperProto ), where class information is represented by hyperspheres with dynamic sizes with two sets of learnable parameters: the hypersphere’s center and the radius. Extending from points to areas, hyperspheres are much more expressive than embeddings. Moreover, it is more convenient to perform metric-based classiﬁcation with hypersphere prototypes than statistical modeling, as we only need to calculate the distance from a data point to the surface of the hypersphere. Following this idea, we also develop two variants of prototypes under other measurements. Extensive experiments and analysis on few-shot learning tasks across NLP and CV and comparison with 20+ competitive baselines demonstrate the effectiveness of our approach.
@inproceedings{2023:hyper, title = {Few-shot Classification with Hypersphere Modeling of Prototypes}, bibtex_show = {true}, author = {Ding, Ning and Chen, Yulin and Cui, Ganqu and Wang, Xiaobin and Zheng, Hai-Tao and Liu, Zhiyuan and Xie, Pengjun}, booktitle = {ACL Findings}, selected = {false}, pdf = {https://arxiv.org/pdf/2211.05319.pdf}, series = {\newlin ACL Findings}, year = {2023} }
Parameter-efficient Weight Ensembling Facilitates Task-level Knowledge Transfer

Xingtai Lv*, Ning Ding*, Yujia Qin, Zhiyuan Liu, and Maosong Sun

ACL Short 2023

PDF Bib
@inproceedings{2023:deltasearch, title = {Parameter-efficient Weight Ensembling Facilitates Task-level Knowledge Transfer}, author = {Lv*, Xingtai and Ding*, Ning and Qin, Yujia and Liu, Zhiyuan and Sun, Maosong}, booktitle = {ACL Short}, selected = {false}, series = {\newlin ACL Short}, pdf = {https://aclanthology.org/2023.acl-short.24.pdf}, bibtex_show = {true}, year = {2023} }

2022

OpenPrompt: An Open-source Framework for Prompt-learning

Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Hai-Tao Zheng, and Maosong Sun

ACL System Demonstration 2022
Best Demo Paper Award

ABSTRACT Link PDF CODE Bib

Prompt-learning has become a new paradigm in modern natural language processing, which directly adapts pre-trained language models (PLMs) to cloze-style prediction, autoregressive modeling, or sequence to sequence generation, resulting in promising performances on various tasks. However, no standard implementation framework of prompt-learning is proposed yet, and most existing prompt-learning codebases, often unregulated, only provide limited implementations for specific scenarios. Since there are many details such as templating strategy, initializing strategy, and verbalizing strategy, etc. need to be considered in prompt-learning, practitioners face impediments to quickly adapting the desired prompt learning methods to their applications. In this paper, we present OpenPrompt, a unified easy-to-use toolkit to conduct prompt-learning over PLMs. OpenPrompt is a research-friendly framework that is equipped with efficiency, modularity, and extendibility, and its combinability allows the freedom to combine different PLMs, task formats, and prompting modules in a unified paradigm. Users could expediently deploy prompt-learning frameworks and evaluate the generalization of them on different NLP tasks without constraints.
@inproceedings{ding2021openprompt, preview = {op.png}, bibtex_show = {true}, title = { OpenPrompt: An Open-source Framework for Prompt-learning}, author = {Ding, Ning and Hu, Shengding and Zhao, Weilin and Chen, Yulin and Liu, Zhiyuan and Zheng, Hai-Tao and Sun, Maosong}, year = {2022}, selected = {true}, booktitle = {ACL System Demonstration}, series = {\newlin ACL System Demonstration}, html = {https://arxiv.org/abs/2111.01998}, pdf = {https://arxiv.org/pdf/2111.01998.pdf}, code = {https://github.com/thunlp/OpenPrompt}, award = { <img src="/assets/img/award.png" width="15px"> Best Demo Paper Award } }
Prompt-Learning for Fine-Grained Entity Typing

Ning Ding, Yulin Chen, Xu Han, Guangwei Xu, Pengjun Xie, Hai-Tao Zheng, Zhiyuan Liu, Juanzi Li, and Kim Hong-Gee

EMNLP Findings 2022

ABSTRACT Link PDF CODE Bib

As an effective approach to tune pre-trained language models (PLMs) for specific tasks, prompt-learning has recently attracted much attention from researchers. By using cloze-style language prompts to stimulate the versatile knowledge of PLMs, prompt-learning can achieve promising results on a series of NLP tasks, such as natural language inference, sentiment classification, and knowledge probing. In this work, we investigate the application of prompt-learning on fine-grained entity typing in fully supervised, few-shot and zero-shot scenarios. We first develop a simple and effective prompt-learning pipeline by constructing entity-oriented verbalizers and templates and conducting masked language modeling. Further, to tackle the zero-shot regime, we propose a self-supervised strategy that carries out distribution-level optimization in prompt-learning to automatically summarize the information of entity types. Extensive experiments on three fine-grained entity typing benchmarks (with up to 86 classes) under fully supervised, few-shot and zero-shot settings show that prompt-learning methods significantly outperform fine-tuning baselines, especially when the training data is insufficient.
@inproceedings{ding2021prompt, title = {Prompt-Learning for Fine-Grained Entity Typing}, author = {Ding, Ning and Chen, Yulin and Han, Xu and Xu, Guangwei and Xie, Pengjun and Zheng, Hai-Tao and Liu, Zhiyuan and Li, Juanzi and Hong-Gee, Kim}, year = {2022}, booktitle = {EMNLP Findings}, bibtex_show = {true}, series = {\newlin EMNLP Findings}, code = {https://github.com/thunlp/OpenPrompt}, html = {https://arxiv.org/pdf/2108.10604v1.pdf}, pdf = {https://arxiv.org/pdf/2108.10604v1.pdf} }
PTR: Prompt Tuning with Rules for Text Classification

Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun

AI Open 2022

ABSTRACT Link PDF CODE Bib

Fine-tuned pre-trained language models (PLMs) have achieved awesome performance on almost all NLP tasks. By using additional prompts to fine-tune PLMs, we can further stimulate the rich knowledge distributed in PLMs to better serve downstream task. Prompt tuning has achieved promising results on some few-class classification tasks such as sentiment classification and natural language inference. However, manually designing lots of language prompts is cumbersome and fallible. For those auto-generated prompts, it is also expensive and time-consuming to verify their effectiveness in non-few-shot scenarios. Hence, it is challenging for prompt tuning to address many-class classification tasks. To this end, we propose prompt tuning with rules (PTR) for many-class text classi- fication, and apply logic rules to construct prompts with several sub-prompts. In this way, PTR is able to encode prior knowledge of each class into prompt tuning. We conduct experiments on relation classification, a typical many-class classification task, and the results on benchmarks show that PTR can significantly and consistently outperform existing state-of-the-art baselines. This indicates that PTR is a promising approach to take advantage of PLMs for those complicated classification tasks.
@inproceedings{han2021ptr, title = {PTR: Prompt Tuning with Rules for Text Classification}, author = {Han, Xu and Zhao, Weilin and Ding, Ning and Liu, Zhiyuan and Sun, Maosong}, booktitle = {AI Open}, code = {https://github.com/thunlp/PTR}, series = {\newlin AI Open}, bibtex_show = {true}, html = {https://arxiv.org/abs/2105.11259}, pdf = {https://arxiv.org/pdf/2105.11259.pdf}, year = {2022} }
Sparse Structure Search for Parameter-Efficient Tuning

Shengding Hu, Zhen Zhang, Ning Ding, Yadao Wang, Yasheng Wang, Zhiyuan Liu, and Maosong Sun

NeurIPS 2022

ABSTRACT Link PDF Bib

Adapting large pre-trained models (PTMs) through fine-tuning imposes prohibitive computational and storage burdens. Recent studies of parameter-efficient tuning (PET) find that only optimizing a small portion of parameters conditioned on PTMs could yield on-par performance compared to conventional fine-tuning. Generally, PET methods exquisitely design parameter-efficient modules (PET modules) which could be applied to arbitrary fine-grained positions inside PTMs. However, the effectiveness of these fine-grained positions largely relies on sophisticated manual designation, thereby usually producing sub-optimal results. In contrast to the manual designation, we explore constructing PET modules in an automatic manner. We automatically \textbfSearch for the \textbfSparse \textbfStructure of \textbfParameter-\textbfEfficient \textbfTuning (S3PET). Based on a unified framework of various PET methods, S3PET conducts the differentiable PET structure search through bi-level optimization and proposes shifted global sigmoid method to explicitly control the number of trainable parameters. Extensive experiments show that S3PET surpasses manual and random structures with less trainable parameters. The searched structures preserve more than 99% fine-tuning performance with 0.01% trainable parameters. Moreover, the advantage of S3PET is amplified with extremely low trainable parameters budgets (0.0009%∼0.01%). The searched structures are transferable and explainable, providing suggestions and guidance for the future design of PET methods.
@inproceedings{hu2022sparse, title = {Sparse Structure Search for Parameter-Efficient Tuning}, author = {Hu, Shengding and Zhang, Zhen and Ding, Ning and Wang, Yadao and Wang, Yasheng and Liu, Zhiyuan and Sun, Maosong}, booktitle = {NeurIPS}, bibtex_show = {true}, series = {\newlin NeurIPS}, html = {https://arxiv.org/abs/2206.07382}, pdf = {https://arxiv.org/abs/2206.07382}, year = {2022} }
Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification

Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Juanzi Li, and Maosong Sun

ACL 2022
Oral Presentation

ABSTRACT Link PDF CODE Bib

Tuning pre-trained language models (PLMs) with task-specific prompts has been a promising approach for text classification. Particularly, previous studies suggest that prompt-tuning has remarkable superiority in the low-data scenario over the generic fine-tuning methods with extra classifiers. The core idea of prompt-tuning is to insert text pieces, i.e., template, to the input and transform a classification problem into a masked language modeling problem, where a crucial step is to construct a projection, i.e., verbalizer, between a label space and a label word space. A verbalizer is usually handcrafted or searched by gradient descent, which may lack coverage and bring considerable bias and high variances to the results. In this work, we focus on incorporating external knowledge into the verbalizer, forming a knowledgeable prompt-tuning (KPT), to improve and stabilize prompt-tuning. Specifically, we expand the label word space of the verbalizer using external knowledge bases (KBs) and refine the expanded label word space with the PLM itself before predicting with the expanded label word space. Extensive experiments on zero and few-shot text classification tasks demonstrate the effectiveness of knowledgeable prompt-tuning.
@inproceedings{hu2021knowledgeable, title = {Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification}, author = {Hu, Shengding and Ding, Ning and Wang, Huadong and Liu, Zhiyuan and Li, Juanzi and Sun, Maosong}, year = {2022}, booktitle = {ACL}, series = {\newlin ACL}, bibtex_show = {true}, award = { Oral Presentation }, code = {https://github.com/thunlp/KnowledgeablePromptTuning}, html = {https://arxiv.org/abs/2108.02035}, pdf = {https://arxiv.org/pdf/2108.02035.pdf} }
Prototypical Verbalizer for Prompt-based Few-shot Tuning

Ganqu Cui, Shengding Hu, Ning Ding, Longtao Huang, and Zhiyuan Liu

ACL 2022

ABSTRACT PDF CODE Bib

Prompt-based tuning for pre-trained language models (PLMs) has shown its effectiveness in few-shot learning. Typically, prompt-based tuning wraps the input text into a cloze question. To make predictions, the model maps the output words to labels via a verbalizer, which is either manually designed or automatically built. However, manual verbalizers heavily depend on domain-specific prior knowledge and human efforts, while finding appropriate label words automatically still remains challenging.In this work, we propose the prototypical verbalizer (ProtoVerb) which is built directly from training data. Specifically, ProtoVerb learns prototype vectors as verbalizers by contrastive learning. In this way, the prototypes summarize training instances and are able to enclose rich class-level semantics. We conduct experiments on both topic classification and entity typing tasks, and the results demonstrate that ProtoVerb significantly outperforms current automatic verbalizers, especially when training data is extremely scarce. More surprisingly, ProtoVerb consistently boosts prompt-based tuning even on untuned PLMs, indicating an elegant non-tuning way to utilize PLMs. Our codes are avaliable at https://github.com/thunlp/OpenPrompt.
@inproceedings{cui-etal-2022-prototypical, title = {Prototypical Verbalizer for Prompt-based Few-shot Tuning}, author = {Cui, Ganqu and Hu, Shengding and Ding, Ning and Huang, Longtao and Liu, Zhiyuan}, booktitle = { ACL}, code = {https://github.com/thunlp/OpenPrompt}, month = may, series = {\newlin ACL}, bibtex_show = {true}, year = {2022}, address = {Dublin, Ireland}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.acl-long.483}, pdf = {https://aclanthology.org/2022.acl-long.483}, doi = {10.18653/v1/2022.acl-long.483}, pages = {7014--7024} }
ProQA: Structural Prompt-based Pre-training for Unified Question Answering

Wanjun Zhong, Yifan Gao, Ning Ding, Yujia Qin, Zhiyuan Liu, Ming Zhou, Jiahai Wang, Jian Yin, and Nan Duan

NAACL 2022

ABSTRACT PDF CODE Bib

Question Answering (QA) is a longstanding challenge in natural language processing. Existing QA works mostly focus on specific question types, knowledge domains, or reasoning skills. The specialty in QA research hinders systems from modeling commonalities between tasks and generalization for wider applications. To address this issue, we present ProQA, a unified QA paradigm that solves various tasks through a single model. ProQA takes a unified structural prompt as the bridge and improves the QA-centric ability by structural prompt-based pre-training. Through a structurally designed prompt-based input schema, ProQA concurrently models the knowledge generalization for all QA tasks while keeping the knowledge customization for every specific QA task. Furthermore, ProQA is pre-trained with structural prompt-formatted large-scale synthesized corpus, which empowers the model with the commonly-required QA ability. Experimental results on 11 QA benchmarks demonstrate that ProQA consistently boosts performance on both full data fine-tuning, few-shot learning, and zero-shot testing scenarios. Furthermore, ProQA exhibits strong ability in both continual learning and transfer learning by taking the advantages of the structural prompt.
@inproceedings{zhong2022proqa, title = {ProQA: Structural Prompt-based Pre-training for Unified Question Answering}, author = {Zhong, Wanjun and Gao, Yifan and Ding, Ning and Qin, Yujia and Liu, Zhiyuan and Zhou, Ming and Wang, Jiahai and Yin, Jian and Duan, Nan}, booktitle = {NAACL}, year = {2022}, series = {\newlin NAACL}, bibtex_show = {true}, code = {https://github.com/zhongwanjun/ProQA}, pdf = {https://arxiv.org/abs/2205.04040} }

2021

Few-NERD: A Few-shot Named Entity Recognition Dataset

Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Hai-Tao Zheng, and Zhiyuan Liu

ACL 2021
Oral Presentation

ABSTRACT Link PDF CODE Bib

Recently, considerable literature has grown up around the theme of few-shot named entity recognition (NER), but little published benchmark data specifically focused on the practical and challenging task. Current approaches collect existing supervised NER datasets and re-organize them to the few-shot setting for empirical study. These strategies conventionally aim to recognize coarse-grained entity types with few examples, while in practice, most unseen entity types are fine-grained. In this paper, we present Few-NERD, a large-scale human-annotated few-shot NER dataset with a hierarchy of 8 coarse-grained and 66 fine-grained entity types. Few-NERD consists of 188,238 sentences from Wikipedia, 4,601,160 words are included and each is annotated as context or a part of a two-level entity type. To the best of our knowledge, this is the first few-shot NER dataset and the largest human-crafted NER dataset. We construct benchmark tasks with different emphases to comprehensively assess the generalization capability of models. Extensive empirical results and analysis show thatFew-NERD is challenging and the problem requires further research. We make Few-NERD public at https://ningding97.github.io/fewnerd/
@inproceedings{ding2021few, preview = {few-nerd-heat.png}, title = {Few-NERD: A Few-shot Named Entity Recognition Dataset}, author = {Ding, Ning and Xu, Guangwei and Chen, Yulin and Wang, Xiaobin and Han, Xu and Xie, Pengjun and Zheng, Hai-Tao and Liu, Zhiyuan}, booktitle = {ACL}, series = {\newlin ACL}, bibtex_show = {true}, award = { Oral Presentation }, selected = {true}, html = {https://ningding97.github.io/fewnerd/}, pdf = {https://arxiv.org/pdf/2105.07464.pdf}, code = {https://github.com/thunlp/Few-NERD}, year = {2021} }

CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding

Dong Wang, Ning* Ding, Piji Li, and Hai-Tao Zheng

ACL 2021
Oral Presentation

Link PDF CODE Bib

@inproceedings{wang2021cline,
  title = {CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding},
  bibtex_show = {true},
  author = {Wang, Dong and Ding, Ning* and Li, Piji and Zheng, Hai-Tao},
  booktitle = { ACL},
  series = {\newlin ACL},
  award = { <br>  <strong> <font color="BB0A21">  Oral Presentation  </font></strong>},
  html = {https://arxiv.org/abs/2107.00440},
  pdf = {https://arxiv.org/pdf/2107.00440.pdf},
  code = {https://github.com/kandorm/CLINE},
  year = {2021}
}

Pre-Trained Models: Past, Present and Future

Xu* Han, Zhengyan* Zhang, Ning* Ding, Yuxian* Gu, Xiao* Liu, Yuqi* Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji-Rong Wen, Jinhui Yuan, Wayne Xin Zhao, and Zhu Jun

AI Open 2021

ABSTRACT Link PDF Bib

Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI). Owing to sophisticated pre-training objectives and huge model parameters, large-scale PTMs can ef- fectively capture knowledge from massive la- beled and unlabeled data. By storing knowl- edge into huge parameters and fine-tuning on specific tasks, the rich knowledge implicitly encoded in huge parameters can benefit a vari- ety of downstream tasks, which has been exten- sively demonstrated via experimental verifica- tion and empirical analysis. It is now the con- sensus of the AI community to adopt PTMs as backbone for downstream tasks rather than learning models from scratch. In this paper, we take a deep look into the history of pre- training, especially its special relation with transfer learning and self-supervised learning, to reveal the crucial position of PTMs in the AI development spectrum. Further, we compre- hensively review the latest breakthroughs of PTMs. These breakthroughs are driven by the surge of computational power and the increas- ing availability of data, towards four impor- tant directions: designing effective architec- tures, utilizing rich contexts, improving com- putational efficiency, and conducting interpre- tation and theoretical analysis. Finally, we dis- cuss a series of open problems and research directions of PTMs, and hope our view can in- spire and advance the future study of PTMs.
@inproceedings{han2021pre, title = {Pre-Trained Models: Past, Present and Future}, author = {Han, Xu* and Zhang, Zhengyan* and Ding, Ning* and Gu, Yuxian* and Liu, Xiao* and Huo, Yuqi* and Qiu, Jiezhong and Zhang, Liang and Han, Wentao and Huang, Minlie and Jin, Qin and Lan, Yanyan and Liu, Yang and Liu, Zhiyuan and Lu, Zhiwu and Qiu, Xipeng and Song, Ruihua and Tang, Jie and Wen, Ji-Rong and Yuan, Jinhui and Zhao, Wayne Xin and Jun, Zhu}, booktitle = {AI Open}, bibtex_show = {true}, series = {\newlin AI Open}, html = {https://arxiv.org/abs/2106.07139}, pdf = {https://arxiv.org/pdf/2106.07139.pdf}, year = {2021} }
Prototypical Representation Learning for Relation Extraction

Ning Ding, Xiaobin Wang, Yao Fu, Guangwei Xu, Rui Wang, Pengjun Xie, Ying Shen, Fei Huang, Hai-Tao Zheng, and Rui Zhang

International Conference on Learning Representations,
ICLR 2021

ABSTRACT Link PDF CODE Bib

Recognizing relations between entities is a pivotal task of relational learning. Learning relation representations from distantly-labeled datasets is difficult because of the abundant label noise and complicated expressions in human language. This paper aims to learn predictive, interpretable, and robust relation representations from distantly-labeled data that are effective in different settings, including supervised, distantly supervised, and few-shot learning. Instead of solely relying on the supervision from noisy labels, we propose to learn prototypes for each relation from contextual information to best explore the intrinsic semantics of relations. Prototypes are representations in the feature space abstracting the essential semantics of relations between entities in sentences. We learn prototypes based on objectives with clear geometric interpretation, where the prototypes are unit vectors uniformly dispersed in a unit ball, and statement embeddings are centered at the end of their corresponding prototype vectors on the surface of the ball. This approach allows us to learn meaningful, interpretable prototypes for the final classification. Results on several relation learning tasks show that our model significantly outperforms the previous state-of-the-art models. We further demonstrate the robustness of the encoder and the interpretability of prototypes with extensive experiments.
@inproceedings{ding2021prototypical, preview = {proto.png}, title = {Prototypical Representation Learning for Relation Extraction}, author = {Ding, Ning and Wang, Xiaobin and Fu, Yao and Xu, Guangwei and Wang, Rui and Xie, Pengjun and Shen, Ying and Huang, Fei and Zheng, Hai-Tao and Zhang, Rui}, booktitle = {International Conference on Learning Representations, ICLR}, year = {2021}, bibtex_show = {true}, series = {\newlin ICLR}, selected = {true}, html = {https://openreview.net/forum?id=aCgLmfhIy_f}, pdf = {https://openreview.net/forum?id=aCgLmfhIy_f}, code = {https://github.com/Alibaba-NLP/ProtoRE} }

2020

Coupling Distant Annotation and Adversarial Training for Cross-Domain Chinese Word Segmentation

Ning Ding, Dingkun Long, Guangwei Xu, Muhua Zhu, Pengjun Xie, Xiaobin Wang, and Hai-Tao Zheng

ACL 2020
Oral Presentation

ABSTRACT Link PDF CODE Bib

Fully supervised neural approaches have achieved significant progress in the task of Chinese word segmentation (CWS). Nevertheless, the performance of supervised models always drops gravely if the domain shifts due to the distribution gap across domains and the out of vocabulary (OOV) problem. In order to simultaneously alleviate the issues, this paper intuitively couples distant annotation and adversarial training for cross-domain CWS. 1) We rethink the essence of “Chinese words” and design an automatic distant annotation mechanism, which does not need any supervision or pre-defined dictionaries on the target domain. The method could effectively explore domain-specific words and distantly annotate the raw texts for the target domain. 2) We further develop a sentence-level adversarial training procedure to perform noise reduction and maximum utilization of the source domain information. Experiments on multiple real-world datasets across various domains show the superiority and robustness of our model, significantly outperforming previous state-of-the-arts cross-domain CWS methods.
@inproceedings{ding2020coupling, title = {Coupling Distant Annotation and Adversarial Training for Cross-Domain Chinese Word Segmentation}, author = {Ding, Ning and Long, Dingkun and Xu, Guangwei and Zhu, Muhua and Xie, Pengjun and Wang, Xiaobin and Zheng, Hai-Tao}, booktitle = { ACL}, pages = {6662--6671}, series = {\newlin ACL}, year = {2020}, bibtex_show = {true}, award = { Oral Presentation }, html = {https://www.aclweb.org/anthology/2020.acl-main.595/}, code = {https://github.com/Alibaba-NLP/DAAT-CWS}, pdf = {https://www.aclweb.org/anthology/2020.acl-main.595/} }
Hierarchy-Aware Global Model for Hierarchical Text Classification

Jie Zhou, Chunping Ma, Dingkun Long, Guangwei Xu, Ning Ding, Haoyu Zhang, Pengjun Xie, and Gongshen Liu

ACL 2020

ABSTRACT Link PDF CODE Bib

Hierarchical text classification is an essential yet challenging subtask of multi-label text classification with a taxonomic hierarchy. Existing methods have difficulties in modeling the hierarchical label structure in a global view. Furthermore, they cannot make full use of the mutual interactions between the text feature space and the label space. In this paper, we formulate the hierarchy as a directed graph and introduce hierarchy-aware structure encoders for modeling label dependencies. Based on the hierarchy encoder, we propose a novel end-to-end hierarchy-aware global model (HiAGM) with two variants. A multi-label attention variant (HiAGM-LA) learns hierarchy-aware label embeddings through the hierarchy encoder and conducts inductive fusion of label-aware text features. A text feature propagation model (HiAGM-TP) is proposed as the deductive variant that directly feeds text features into hierarchy encoders. Compared with previous works, both HiAGM-LA and HiAGM-TP achieve significant and consistent improvements on three benchmark datasets.
@inproceedings{zhou-etal-2020-hierarchy, title = {Hierarchy-Aware Global Model for Hierarchical Text Classification}, author = {Zhou, Jie and Ma, Chunping and Long, Dingkun and Xu, Guangwei and Ding, Ning and Zhang, Haoyu and Xie, Pengjun and Liu, Gongshen}, booktitle = {ACL}, month = jul, year = {2020}, bibtex_show = {true}, series = {\newlin ACL}, address = {Online}, publisher = {Association for Computational Linguistics}, html = {https://www.aclweb.org/anthology/2020.acl-main.104}, doi = {10.18653/v1/2020.acl-main.104}, pages = {1106--1117}, code = {https://github.com/Alibaba-NLP/HiAGM}, pdf = {https://www.aclweb.org/anthology/2020.acl-main.104} }
Modeling Relation Paths for Knowledge Graph Completion

Ying Shen, Ning Ding, Hai-Tao Zheng, Yaliang Li, and Min Yang

IEEE Transactions on Knowledge and Data Engineering (TKDE),

ABSTRACT Link PDF Bib

Knowledge graphs (KG) often encounter knowledge incompleteness. The path reasoning that predicts the unknown path relation between pairwise entities based on existing facts is one of the most promising approaches to the knowledge graph completion. However, most conventional path reasoning methods exclusively consider the entity description included in fact triples, ignoring both the type information of entities and the interaction between different semantic representations. In this study, we propose a novel method, Type-aware Attentive Path Reasoning (TAPR), to complete the knowledge graph by simultaneously considering KG structural information, textual information, and type information. More specifically, we first leverage types to enrich the representational learning of entities and relationships. Next, we describe a type-level attention to select the most relevant type of given entity in a specific triple without any predefined rules or patterns to reduce the impact of noisy types. After learning the distributed representation of all paths, path-level attention assigns different weights to paths, from which relations among entity pairs are calculated. We conduct a series of experiments on a real-world dataset to demonstrate the effectiveness of TAPR. Experimental results show that our method significantly outperforms all baselines on link prediction and entity prediction tasks.
@article{shen2020modeling, author = {Shen, Ying and Ding, Ning and Zheng, Hai-Tao and Li, Yaliang and Yang, Min}, journal = {IEEE Transactions on Knowledge and Data Engineering (TKDE),}, publisher = {IEEE}, title = {Modeling Relation Paths for Knowledge Graph Completion}, year = {2020}, series = {\newlin TKDE}, bibtex_show = {true}, html = {https://ieeexplore.ieee.org/abstract/document/8974254}, pdf = {https://ieeexplore.ieee.org/abstract/document/8974254} }
Integrating Linguistic Knowledge to Sentence Paraphrase Generation

Zibo Lin, Ziran Li, Ning Ding, Hai-Tao Zheng, Ying Shen, Xiaobin Zhao, and Cong-Zhi Zheng

AAAI 2020

ABSTRACT Link PDF CODE Bib

Paraphrase generation aims to rewrite a text with different words while keeping the same meaning. Previous work performs the task based solely on the given dataset while ignoring the availability of external linguistic knowledge. However, it is intuitive that a model can generate more expressive and diverse paraphrase with the help of such knowledge. To fill this gap, we propose Knowledge-Enhanced Paraphrase Network (KEPN), a transformer-based framework that can leverage external linguistic knowledge to facilitate paraphrase generation. (1) The model integrates synonym information from the external linguistic knowledge into the paraphrase generator, which is used to guide the decision on whether to generate a new word or replace it with a synonym. (2) To locate the synonym pairs more accurately, we adopt an incremental encoding scheme to incorporate position information of each synonym. Besides, a multi-task architecture is designed to help the framework jointly learn the selection of synonym pairs and the generation of expressive paraphrase. Experimental results on both English and Chinese datasets show that our method significantly outperforms the state-of-the-art approaches in terms of both automatic and human evaluation.
@inproceedings{2020-paraphrase, title = {Integrating Linguistic Knowledge to Sentence Paraphrase Generation}, author = {Lin, Zibo and Li, Ziran and Ding, Ning and Zheng, Hai-Tao and Shen, Ying and Zhao, Xiaobin and Zheng, Cong-Zhi}, booktitle = {AAAI}, year = {2020}, series = {\newlin AAAI}, bibtex_show = {true}, pdf = {AAAI2020-para.pdf}, code = {https://github.com/LINMouMouZiBo/KEPN}, html = {https://ojs.aaai.org/index.php/AAAI/article/view/6354} }
Infobox-to-text Generation with Tree-like Planning based Attention Network

Yang Bai, Ziran Li, Ning Ding, Ying Shen, and Hai-Tao Zheng

IJCAI 2020

ABSTRACT Link PDF Bib

We study the problem of infobox-to-text generation that aims to generate a textual description from a key-value table. Representing the input infobox as a sequence, previous neural methods using end-to-end models without order-planning suffer from the problems of incoherence and inadaptability to disordered input. Recent planning-based models only implement static order-planning to guide the generation, which may cause error propagation between planning and generation. To address these issues, we propose a Tree-like PLanning based Attention Network (Tree-PLAN) which leverages both static order-planning and dynamic tuning to guide the generation. A novel tree-like tuning encoder is designed to dynamically tune the static order-plan for better planning by merging the most relevant attributes together layer by layer. Experiments conducted on two datasets show that our model outperforms previous methods on both automatic and human evaluation, and demonstrate that our model has better adaptability to disordered input.
@inproceedings{2020-infobox, title = {Infobox-to-text Generation with Tree-like Planning based Attention Network}, author = {Bai, Yang and Li, Ziran and Ding, Ning and Shen, Ying and Zheng, Hai-Tao}, booktitle = {IJCAI}, series = {\newlin IJCAI}, publisher = {International Joint Conferences on Artificial Intelligence Organization}, editor = {Bessiere, Christian}, year = {2020}, bibtex_show = {true}, html = {https://doi.org/10.24963/ijcai.2020/522}, pdf = {IJCAI2020-infobox.pdf} }
Triple-to-Text Generation with an Anchor-to-Prototype Framework

Ziran Li, Zibo Lin, Ning Ding, Hai-Tao Zheng, and Ying Shen

IJCAI 2020

ABSTRACT Link PDF Bib

Generating a textual description from a set of RDF triplets is a challenging task in natural language generation. Recent neural methods have become the mainstream for this task, which often generate sentences from scratch. However, due to the huge gap between the structured input and the unstructured output, the input triples alone are insufficient to decide an expressive and specific description. In this paper, we propose a novel anchor-to-prototype framework to bridge the gap between structured RDF triples and natural text. The model retrieves a set of prototype descriptions from the training data and extracts writing patterns from them to guide the generation process. Furthermore, to make a more precise use of the retrieved prototypes, we employ a triple anchor that aligns the input triples into groups so as to better match the prototypes. Experimental results on both English and Chinese datasets show that our method significantly outperforms the state-of-the-art baselines in terms of both automatic and manual evaluation, demonstrating the benefit of learning guidance from retrieved prototypes to facilitate triple-to-text generation.
@inproceedings{2020:anchor-to-frame, title = {Triple-to-Text Generation with an Anchor-to-Prototype Framework}, author = {Li, Ziran and Lin, Zibo and Ding, Ning and Zheng, Hai-Tao and Shen, Ying}, booktitle = {IJCAI}, series = {\newlin IJCAI}, publisher = {International Joint Conferences on Artificial Intelligence Organization}, year = {2020}, bibtex_show = {true}, doi = {10.24963/ijcai.2020/523}, html = {https://doi.org/10.24963/ijcai.2020/523}, pdf = {IJCAI2020-triple.pdf} }

2019

Event Detection with Trigger-Aware Lattice Neural Network

Ning Ding, Ziran Li, Zhiyuan Liu, and Hai-Tao Zheng

EMNLP 2019

ABSTRACT Link PDF CODE Bib

Event detection (ED) aims to locate trigger words in raw text and then classify them into correct event types. In this task, neural net- work based models became mainstream in re- cent years. However, two problems arise when it comes to languages without natural delim- iters, such as Chinese. First, word-based mod- els severely suffer from the problem of word- trigger mismatch, limiting the performance of the methods. In addition, even if trigger words could be accurately located, the ambi- guity of polysemy of triggers could still af- fect the trigger classification stage. To ad- dress the two issues simultaneously, we pro- pose the Trigger-aware Lattice Neural Net- work (TLNN). (1) The framework dynami- cally incorporates word and character informa- tion so that the trigger-word mismatch issue can be avoided. (2) Moreover, for polysemous characters and words, we model all senses of them with the help of an external linguistic knowledge base, so as to alleviate the prob- lem of ambiguous triggers. Experiments on two benchmark datasets show that our model could effectively tackle the two issues and outperforms previous state-of-the-art methods significantly, giving the best results. The source code of this paper can be obtained from https://github.com/thunlp/TLNN.
@inproceedings{2019:tlnn, address = {HongKong, China}, author = {Ding, Ning and Li, Ziran and Liu, Zhiyuan and Zheng, Hai-Tao}, booktitle = {EMNLP}, month = oct, series = {\newlin EMNLP}, publisher = {Association for Computational Linguistics}, title = {Event Detection with Trigger-Aware Lattice Neural Network}, year = {2019}, bibtex_show = {true}, pdf = {https://www.aclweb.org/anthology/D19-1033/}, html = {https://www.aclweb.org/anthology/D19-1033/}, code = {https://github.com/thunlp/TLNN} }
Chinese Relation Extraction with Multi-Grained Information and External Linguistic Knowledge

Ziran* Li, Ning* Ding, Zhiyuan Liu, Hai-Tao Zheng, and Ying Shen

ACL 2019
Oral Presentation

ABSTRACT Link PDF CODE DATA Bib

Chinese relation extraction is conducted using neural networks with either character-based or word-based inputs, and most existing methods typically suffer from segmentation errors and ambiguity of polysemy. To address the issues, we propose a multi-grained lattice framework (MG lattice) for Chinese relation extraction to take advantage of multi-grained language information and external linguistic knowledge. In this framework, (1) we incorporate word-level information into character sequence inputs so that segmentation errors can be avoided. (2) We also model multiple senses of polysemous words with the help of external linguistic knowledge, so as to alleviate polysemy ambiguity. Experiments on three real-world datasets in distinct domains show consistent and significant superiority and robustness of our model, as compared with other baselines. We will release the source code of this paper in the future.
@inproceedings{2019:multigrained, title = {{C}hinese Relation Extraction with Multi-Grained Information and External Linguistic Knowledge}, author = {Li, Ziran* and Ding, Ning* and Liu, Zhiyuan and Zheng, Hai-Tao and Shen, Ying}, booktitle = {ACL}, month = jul, year = {2019}, series = {\newlin ACL}, publisher = {Association for Computational Linguistics}, html = {https://www.aclweb.org/anthology/P19-1430}, doi = {10.18653/v1/P19-1430}, pages = {4377--4386}, bibtex_show = {true}, data = {https://github.com/thunlp/Chinese_NRE/tree/master/data/FinRE}, pdf = {https://www.aclweb.org/anthology/P19-1430}, code = {https://github.com/thunlp/Chinese_NRE}, award = { Oral Presentation } }