codex_kg — Wikidata knowledge graph benchmarks (CoDEx, 2020)

Description

Knowledge graphs extracted from Wikidata by Safavi and Koutra as benchmarks for knowledge graph completion (link prediction and triple classification), the CoDEx datasets. Each node is an entity of Wikidata (e.g. a person, a country, an organization, a language or a disease), and a directed edge from entity A to entity B is a triple (A, relation, B), e.g. that a person has the country of citizenship B. Pairs of entities may be connected by several edges with different relations.

The authors first defined a seed set of entity types and relations of 13 domains (e.g. business, geography, medicine, music, politics, science, sports), and queried Wikidata for the statements whose subject has a seed type and whose relation is a seed relation, which gives the raw dump (raw: 380,038 entities, 75 relations, 1,156,222 triples). From it, they extracted three benchmarks of increasing size, as k-cores of the raw graph, from which they removed the relations that largely overlap with the inverse of another one:

codex_s (k = 15): 2,034 entities and 42 relations;
codex_m (k = 10): 17,050 entities and 51 relations;
codex_l (k = 5): 77,951 entities and 69 relations.

The triples of the benchmarks are divided into training, validation and test sets; all of them are included, and the set is given as an edge property. The "hard negatives" of codex_s and codex_m, triples verified to be false that are used for triple classification, are left out, as are the Wikipedia page extracts of the entities and the labels in other languages than English.

The labels, descriptions, URLs and types are given by the data only for the entities of codex_l, so they are empty for the other entities of raw (302,087 of them).

The networks have the following properties:

wikidata_id: the Wikidata identifier of the entity, e.g. Q7604.
label: the English label of the entity (empty if absent).
description: the English description of the entity in Wikidata (empty if absent).
wikipedia_url: the URL of the English Wikipedia article of the entity (empty if absent).
types: the English labels of the types of the entity in Wikidata (e.g. human).
Edge relation_id: the Wikidata property of the relation, e.g. P27.
Edge relation: the English label of the relation, e.g. country of citizenship (empty for the relations of raw that are not described in the data).
Edge split (not for raw): the set of the benchmark the triple belongs to: train, valid or test.

Tags

Informational Relatedness Unweighted Multigraph Metadata

Citation

T. Safavi, D. Koutra, "CoDEx: A Comprehensive Knowledge Graph Completion Benchmark." Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 8328-8350 (2020) 10.18653/v1/2020.emnlp-main.669 Sci-Hub

Upstream URL OK

https://github.com/tsafavi/codex

Upstream license

MIT, https://github.com/tsafavi/codex/blob/3132e426c2a6b643b70bad679905a3a6270be440/LICENSE

Networks

Tip: click on a column header to sort the list, or hover over it for a description.

Column legend

$\left<k\right>$: Mean degree
$\sigma_k$: Standard deviation of the degrees
$\lambda_h$: Largest eigenvalue of the Hashimoto (non-backtracking) matrix
$\tau$: Random walk mixing time
$r$: Degree assortativity
$c$: Global clustering coefficient
$\oslash$: (Pseudo-) diameter
$S$: Relative size of the largest component
NPs: Node properties
EPs: Edge properties

Name	Nodes	Edges	$\left<k\right>$	$\sigma_k$	$\lambda_h$	$\tau$	$r$	$c$	$\oslash$	$S$	Kind	Mode	NPs	EPs	gt	GraphML	GML	csv
codex_s	2,034	36,543	17.97	60.60	79.34	12.96	-0.24	0.11	5	1.00	Directed	Unipartite	wikidata_id label description wikipedia_url types	relation_id relation split	206 KiB	361 KiB	349 KiB	286 KiB
codex_m	17,050	206,205	12.09	116.71	129.20	26.53	-0.19	0.01	5	1.00	Directed	Unipartite	wikidata_id label description wikipedia_url types	relation_id relation split	1.3 MiB	2.3 MiB	2.2 MiB	2.0 MiB
codex_l	77,951	612,437	7.86	159.13	171.22	15.89	-0.17	0.00	8	1.00	Directed	Unipartite	wikidata_id label description wikipedia_url types	relation_id relation split	5.0 MiB	7.9 MiB	7.8 MiB	7.4 MiB
raw	380,038	1,156,222	3.04	121.40	175.37	216.82	-0.09	0.00	17	0.99	Directed	Unipartite	wikidata_id label description wikipedia_url types	relation_id relation	11.5 MiB	17.8 MiB	17.8 MiB	15.4 MiB

Ridiculograms^*

^* These are automatically generated force-directed visualizations, and can be quite meaningless for networks both big and small. They should not be taken seriously as sources of scientific insight. See here for a discussion.