We released KnowCoder, a powerful Large Language Model for Universal Information Extraction that injects thousands of structured knowledge through code.

KnowCoder completed various evaluations on 33 widely used information extraction benchmarks:

- After code pretraining on around 1.5B automatically constructed data, KnowCoder already attains remarkable generalization ability and achieves NER improvements compared to LLaMA2 by 49.8% relative F1 under the few-shot setting.

- After instruction tuning, KnowCoder further exhibits strong generalization ability on unseen schemas and achieves up to 12.5% and 21.9% under the zero-shot setting and the low resource setting, respectively.

- Additionally, based on our unified schema representations, various human-annotated datasets can simultaneously be utilized to refine KnowCoder, which achieves significant improvements up to 7.5% under the supervised setting.

The code-style schema representation method comprises three basic classes, namely, "Entity", "Relation", and "Event". Based on the three basic classes, we represent all the concepts in the schemas by the corresponding classes. Thus, the instances of each concept can be represented by the objects of the corresponding class. A schema consists of class name, class inheritance, class comments, type hint, and class method. The detailed explanation of each component can be found in our paper.

We construct the code-style schema library under this schema representation method based on Wikidata (Note that we use the Wikidata dump up to 20220704). We select the concepts included in the existing IE datasets created from Wikidata, i.e., KELM, UniversalNER, InstructIE, and LSEE, and derive the constraints among concepts according to their co-occurrences. To construct the taxonomies, we extract the "subclass of" relations among these concepts from Wikidata. To obtain the description of a concept, we use its definition from Wikidata directly or generate its descriptions using GPT-4 if its definition in Wikidata is missing. Finally, the constructed schema library encompasses over 29,177 entity types, 876 relation types, and 519 event types. The detailed statistics of the schema are shown in Table \ref{schema_stat}. Here, "#Type" denotes the total number of types, "#Type w/ desc." indicates the count of types with descriptions, and "#Type w/o desc." signifies the count of types without descriptions.

The schema library data can be found in 🤗Schema-Library.

The datasets consist of three parts: schema understanding data, schema following data, and specific domain IE data.

The datasets are released in 🤗Huggingface-KnowCoder.

The schema understanding data includes schema definition codes and schema instance codes.

The schema understanding data can be found in 🤗Schema-Understanding.

The schema definition codes are built based on the schema library, with statistical results shown in Schema Library Construction.

The schema instance cods are constructed based on KELM. The statistical results are as follows.

The cases of schema instance codes in schema understanding data shown in this file.

The schema following data can be found in 🤗Schema-Following.

The schema following data is constructed on UniversalNER, InstructIE, and LSEE. The statistics of schema following data are presented in Schema Instance Codes.

The cases of schema following data are shown here.

Note: Because some data sets have copyright requirements and need licenses, we cannot directly release this part of the data now. If you have a license for restricted datasets, you can contact us via emails to obtain data.

Additionally, for specific domain Information Extraction (IE), we conduct experiments utilizing 33 datasets, comprising 23 datasets for the NER task, 8 datasets for the RE task, and 2 datasets for the ED and EAE tasks. Specifically, under the supervised setting, we employ 18 datasets for the NER task, including ACE04, ACE05, AnatEM, Broad Twitter, bc2gm, bc5cdr, CoNLL03, DIANN, FabNER, FindVehicle, GENIA, MIT-Movie, MIT-Restaurant, MultiNERD, ncbi-disease, Ontonotes5, WikiANN, and WNUT17. For the RE task, we utilize 8 datasets under the supervised setting, including ACE05, ADE corpus, CoNLL04, GIDS, kbp37, NYT, SciERC, and semeval RE. For the ED and EAE tasks, ACE05 and CASIE are employed.

Under the zero-shot setting, we take 7 datasets for the NER task, comprising 5 CrossNER subsets (AI, literature, music, politics, science), MIT-Movie, and MIT-Restaurant. For the RE task, we adopt GIDS under the zero-shot setting. For the ED and EAE tasks, CASIE is adopted under the zero-shot setting.

The detailed statistic of each dataset is shown as follows. Here, "#Type" indicates the number of types, while "#Train", "#Dev", and "#Test" denote the number of sentences in the training, development, and testing datasets, respectively. Here is the overview of the datasets on specific domain IE by task and size. Note that the statistics for each dataset in the figure encompass the total number of train, dev, and test sets.

After Schema Understanding, we can obtain the KnowCoder (SU. only).

To verify the generalization ability of KnowCoder (SU. only), we conduct few-shot experiments on 7 datasets across NER tasks.

After Schema Understanding and Schema Following on LLaMA2, we can obtain the KnowCoder.

To verify the generalization ability of KnowCoder, we conduct zero-shot experiments on 9 datasets across NER, RE, and ED tasks.

To further investigate the generalization ability of KnowCoder for IE tasks, we conduct experiments by refine KnowCoder with three different partitions of the original training sets (1/5/10% ratio) across four tasks to further evaluate its performance in low-resource scenarios.

Based on our unified schema representations, various human-annotated datasets can simultaneously be utilized to refine KnowCoder.

To further investigate the IE ability of KnowCoder, we conduct supervised experiments on four IE tasks, including NER, RE, ED, and EAE. Under the supervised evaluation, KnowCoder is further refined with 28 IE datasets.