logo Netzschleuder network catalog, repository, and centrifuge

History of everycure_kg

The changes to the source code of this entry in the repository, newest first, each with the differences it made to the entry's files. They are also listed on GitLab.

[entries] Remove superfluous details of the upstream distribution from the descriptions by Tiago de Paula Peixoto dc07aa0
The descriptions mentioned incidental details of how the upstream data
are made available: the file formats (CSV, JSON, XML, GEXF, SBML, HIF,
Pajek, UCINET, ...), file names, where the files were published or
obtained from (Zenodo, figshare, Kaggle, the Pajek datasets page, the
Internet Archive), and that some networks are only available in the gt
format. These are left out; caveats about the data themselves are kept,
referring to "the data" instead of "the file". The description of
sp_baboons now lists its properties instead of the upstream columns,
and says that events without a recipient are left out.
diff --git a/entries/everycure_kg/__init__.py b/entries/everycure_kg/__init__.py
index e8463d6..1b4e732 100644
--- a/entries/everycure_kg/__init__.py
+++ b/entries/everycure_kg/__init__.py
@@ -27,7 +27,7 @@ from .. import *
 title = "Every Cure biomedical knowledge graph (2025-2026)"
 description = r"""A large biomedical knowledge graph, the integrated knowledge graph of Every Cure (MATRIX project, for drug repurposing), which merges, harmonizes and filters the knowledge graphs ROBOKOP and RTX-KG2. These integrate the knowledge of many primary sources (ontologies, and databases such as UMLS, SemMedDB (relations mined from the literature), ChEMBL, DrugBank, Reactome or CTD). The nodes are entities of the Biolink model (e.g. `biolink:Gene`, `biolink:SmallMolecule`, `biolink:Disease`, `biolink:PhenotypicFeature`, `biolink:Protein`, `biolink:Publication`), identified by compact URIs (e.g. `NCBIGene:7157`, `CHEBI:15365`, `MONDO:0005148`), and each directed edge is a statement (subject, predicate, object), with a predicate of the Biolink model (e.g. `biolink:treats`, `biolink:interacts_with`, `biolink:subclass_of`) and its provenance. Two entities can be joined by several edges (with different predicates or sources), so that the networks are multigraphs.
 
-There is one network for each release published by Every Cure on Hugging Face, named by its date: `2025-12-05`, `2025-12-18` and `2026-05-28` (the release of February 2026).
+There is one network for each release of Every Cure, named by its date: `2025-12-05`, `2025-12-18` and `2026-05-28` (the release of February 2026).
 
 The networks have the following properties (those given in each release):
 
[entries] Add the date when each entry was added, and remove icon_hash by Tiago de Paula Peixoto beff7c9
The dates are those of the commits that first added the entries' files
(2020-08-11 for the entries of the initial commit); renamed entries keep
the date of their first name.

Changed: none
diff --git a/entries/everycure_kg/__init__.py b/entries/everycure_kg/__init__.py
index bfe0d77..e8463d6 100644
--- a/entries/everycure_kg/__init__.py
+++ b/entries/everycure_kg/__init__.py
@@ -57,7 +57,7 @@ citation = [('Every Cure, "Every Cure Integrated Knowledge Graph." Hugging Face
              'https://doi.org/10.1021/acs.jcim.9b00683'),
             ('E. C. Wood, A. K. Glen, L. G. Kvarfordt, F. Womack, L. Acevedo, T. S. Yoon, C. Ma, V. Flores, M. Sinha, Y. Chodpathumwan, A. Termehchy, J. C. Roach, L. Mendoza, A. S. Hoffman, E. W. Deutsch, D. Koslicki, S. A. Ramsey, "RTX-KG2: a system for building a semantically standardized knowledge graph for translational biomedicine." BMC Bioinformatics 23, 400 (2022)',
              'https://doi.org/10.1186/s12859-022-04932-3')]
-icon_hash = None
+date_added = '2026-10-31'
 # Parsing needs an estimated 60-75 GB for the largest release (measured on
 # samples: ~550 bytes per edge, plus the nodes and coerce_props)
 # CC BY 4.0 for the integration by Every Cure; the licenses of the sources
[entries] Prefix the properties in the descriptions with (Node), (Edge) or (Graph) by Tiago de Paula Peixoto bc3dbbd
The lists of properties at the end of the descriptions marked edge
properties with "Edge" but left node properties unmarked, or grouped
them in "Nodes: `a`, ...; `b`, ..." items, so the kind of a property
was often unclear. Every item now starts with (Node), (Edge) or (Graph),
followed by the property names, a colon and the explanation, one item
per property or group of properties; qualifications (kinds of nodes,
networks) follow the names in parentheses. Properties that were only
named got a short explanation, and a few items that named the wrong
kind of property were corrected.
diff --git a/entries/everycure_kg/__init__.py b/entries/everycure_kg/__init__.py
index 4050327..bfe0d77 100644
--- a/entries/everycure_kg/__init__.py
+++ b/entries/everycure_kg/__init__.py
@@ -31,22 +31,22 @@ There is one network for each release published by Every Cure on Hugging Face, n
 
 The networks have the following properties (those given in each release):
 
-- `name`: the identifier of the entity (e.g. `NCBIGene:7157`).
-- `label`: the name of the entity (e.g. `TP53`).
-- `category`: the most specific Biolink category of the entity, and `all_categories` all of its categories.
-- `description`: the description of the entity.
-- `equivalent_identifiers`: the identifiers of the entity in other databases.
-- `publications`: the publications of the entity (for nodes) or supporting the statement (for edges, separated by `|`), e.g. PubMed identifiers.
-- `labels`: other labels of the entity.
-- `iri`: the international resource identifier (IRI) of the entity.
-- `upstream_data_source`: the knowledge graphs from which the entity or the statement comes (`robokop`, `rtxkg2`; separated by `|` for edges).
-- `in_nodes`: whether the entity is in the list of nodes of the release (the few entities found only in the edges have no other attributes).
-- Edge `predicate`: the Biolink predicate of the statement.
-- Edge `primary_knowledge_source`: the primary source of the statement (e.g. `infores:drugbank`), and `primary_knowledge_sources` all of them; `aggregator_knowledge_source`: the sources through which it was aggregated (lists separated by `|`).
-- Edge `knowledge_level` (e.g. `knowledge_assertion`, `prediction`), `agent_type` (e.g. `manual_agent`, `text_mining_agent`): how the statement was obtained.
-- Edge `num_references`, `num_sentences`: the number of supporting references and sentences (-1 or NaN if not given).
-- Edge `has_confidence_score`, `extraction_confidence_score`, `affinity`: the confidence of the statement, the confidence of its extraction from the text, and the binding affinity (NaN if not given).
-- Edge `other_attributes`: the other attributes of the statement that are given (in JSON), mostly qualifiers of the Biolink model (e.g. `qualified_predicate`, `object_aspect_qualifier`, `object_direction_qualifier`, `species_context_qualifier`), `affinity_parameter` and `supporting_study_method_type`."""
+- (Node) `name`: the identifier of the entity (e.g. `NCBIGene:7157`).
+- (Node) `label`: the name of the entity (e.g. `TP53`).
+- (Node) `category`: the most specific Biolink category of the entity, and `all_categories` all of its categories.
+- (Node) `description`: the description of the entity.
+- (Node) `equivalent_identifiers`: the identifiers of the entity in other databases.
+- (Node) `publications`: the publications of the entity (for nodes) or supporting the statement (for edges, separated by `|`), e.g. PubMed identifiers.
+- (Node) `labels`: other labels of the entity.
+- (Node) `iri`: the international resource identifier (IRI) of the entity.
+- (Node) `upstream_data_source`: the knowledge graphs from which the entity or the statement comes (`robokop`, `rtxkg2`; separated by `|` for edges).
+- (Node) `in_nodes`: whether the entity is in the list of nodes of the release (the few entities found only in the edges have no other attributes).
+- (Edge) `predicate`: the Biolink predicate of the statement.
+- (Edge) `primary_knowledge_source`: the primary source of the statement (e.g. `infores:drugbank`), and `primary_knowledge_sources` all of them; `aggregator_knowledge_source`: the sources through which it was aggregated (lists separated by `|`).
+- (Edge) `knowledge_level` (e.g. `knowledge_assertion`, `prediction`), `agent_type` (e.g. `manual_agent`, `text_mining_agent`): how the statement was obtained.
+- (Edge) `num_references`, `num_sentences`: the number of supporting references and sentences (-1 or NaN if not given).
+- (Edge) `has_confidence_score`, `extraction_confidence_score`, `affinity`: the confidence of the statement, the confidence of its extraction from the text, and the binding affinity (NaN if not given).
+- (Edge) `other_attributes`: the other attributes of the statement that are given (in JSON), mostly qualifiers of the Biolink model (e.g. `qualified_predicate`, `object_aspect_qualifier`, `object_direction_qualifier`, `species_context_qualifier`), `affinity_parameter` and `supporting_study_method_type`."""
 tags = ['Biological', 'Informational', 'Unweighted', 'Multigraph', 'Metadata']
 url = ['https://huggingface.co/datasets/everycure/kg-edges',
        'https://huggingface.co/datasets/everycure/kg-nodes',
[entries] New entry: everycure_kg by Tiago de Paula Peixoto 4ac079e
diff --git a/entries/everycure_kg/__init__.py b/entries/everycure_kg/__init__.py
new file mode 100644
index 0000000..4050327
--- /dev/null
+++ b/entries/everycure_kg/__init__.py
@@ -0,0 +1,272 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2026 Tiago de Paula Peixoto <tiago@skewed.de>
+#
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option) any
+# later version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import json
+
+import pyarrow
+import pyarrow.compute as pc
+import pyarrow.parquet as pq
+
+from .. import *
+
+title = "Every Cure biomedical knowledge graph (2025-2026)"
+description = r"""A large biomedical knowledge graph, the integrated knowledge graph of Every Cure (MATRIX project, for drug repurposing), which merges, harmonizes and filters the knowledge graphs ROBOKOP and RTX-KG2. These integrate the knowledge of many primary sources (ontologies, and databases such as UMLS, SemMedDB (relations mined from the literature), ChEMBL, DrugBank, Reactome or CTD). The nodes are entities of the Biolink model (e.g. `biolink:Gene`, `biolink:SmallMolecule`, `biolink:Disease`, `biolink:PhenotypicFeature`, `biolink:Protein`, `biolink:Publication`), identified by compact URIs (e.g. `NCBIGene:7157`, `CHEBI:15365`, `MONDO:0005148`), and each directed edge is a statement (subject, predicate, object), with a predicate of the Biolink model (e.g. `biolink:treats`, `biolink:interacts_with`, `biolink:subclass_of`) and its provenance. Two entities can be joined by several edges (with different predicates or sources), so that the networks are multigraphs.
+
+There is one network for each release published by Every Cure on Hugging Face, named by its date: `2025-12-05`, `2025-12-18` and `2026-05-28` (the release of February 2026).
+
+The networks have the following properties (those given in each release):
+
+- `name`: the identifier of the entity (e.g. `NCBIGene:7157`).
+- `label`: the name of the entity (e.g. `TP53`).
+- `category`: the most specific Biolink category of the entity, and `all_categories` all of its categories.
+- `description`: the description of the entity.
+- `equivalent_identifiers`: the identifiers of the entity in other databases.
+- `publications`: the publications of the entity (for nodes) or supporting the statement (for edges, separated by `|`), e.g. PubMed identifiers.
+- `labels`: other labels of the entity.
+- `iri`: the international resource identifier (IRI) of the entity.
+- `upstream_data_source`: the knowledge graphs from which the entity or the statement comes (`robokop`, `rtxkg2`; separated by `|` for edges).
+- `in_nodes`: whether the entity is in the list of nodes of the release (the few entities found only in the edges have no other attributes).
+- Edge `predicate`: the Biolink predicate of the statement.
+- Edge `primary_knowledge_source`: the primary source of the statement (e.g. `infores:drugbank`), and `primary_knowledge_sources` all of them; `aggregator_knowledge_source`: the sources through which it was aggregated (lists separated by `|`).
+- Edge `knowledge_level` (e.g. `knowledge_assertion`, `prediction`), `agent_type` (e.g. `manual_agent`, `text_mining_agent`): how the statement was obtained.
+- Edge `num_references`, `num_sentences`: the number of supporting references and sentences (-1 or NaN if not given).
+- Edge `has_confidence_score`, `extraction_confidence_score`, `affinity`: the confidence of the statement, the confidence of its extraction from the text, and the binding affinity (NaN if not given).
+- Edge `other_attributes`: the other attributes of the statement that are given (in JSON), mostly qualifiers of the Biolink model (e.g. `qualified_predicate`, `object_aspect_qualifier`, `object_direction_qualifier`, `species_context_qualifier`), `affinity_parameter` and `supporting_study_method_type`."""
+tags = ['Biological', 'Informational', 'Unweighted', 'Multigraph', 'Metadata']
+url = ['https://huggingface.co/datasets/everycure/kg-edges',
+       'https://huggingface.co/datasets/everycure/kg-nodes',
+       'https://docs.dev.everycure.org/releases/public_data_releases/']
+citation = [('Every Cure, "Every Cure Integrated Knowledge Graph." Hugging Face datasets everycure/kg-nodes and everycure/kg-edges (2026)',
+             'https://huggingface.co/datasets/everycure/kg-edges'),
+            ('C. Bizon, S. Cox, J. Balhoff, Y. Kebede, P. Wang, K. Morton, K. Fecho, A. Tropsha, "ROBOKOP KG and KGB: Integrated Knowledge Graphs from Federated Sources." Journal of Chemical Information and Modeling 59, 4968-4973 (2019)',
+             'https://doi.org/10.1021/acs.jcim.9b00683'),
+            ('E. C. Wood, A. K. Glen, L. G. Kvarfordt, F. Womack, L. Acevedo, T. S. Yoon, C. Ma, V. Flores, M. Sinha, Y. Chodpathumwan, A. Termehchy, J. C. Roach, L. Mendoza, A. S. Hoffman, E. W. Deutsch, D. Koslicki, S. A. Ramsey, "RTX-KG2: a system for building a semantically standardized knowledge graph for translational biomedicine." BMC Bioinformatics 23, 400 (2022)',
+             'https://doi.org/10.1186/s12859-022-04932-3')]
+icon_hash = None
+# Parsing needs an estimated 60-75 GB for the largest release (measured on
+# samples: ~550 bytes per edge, plus the nodes and coerce_props)
+# CC BY 4.0 for the integration by Every Cure; the licenses of the sources
+# (some non-commercial or share-alike) remain in force for their content
+upstream_license = ('Creative Commons Attribution 4.0 International (CC BY 4.0) for the integration; the licenses of the sources apply to their content',
+                    'https://huggingface.co/datasets/everycure/kg-edges/blob/main/LICENSE.md')
+huge = True
+upstream_prefix = 'https://huggingface.co/datasets/everycure'
+# Releases: (commit of kg-nodes, number of node files, commit of kg-edges,
+# number of edge files)
+releases = {"2025-12-05": ("165e66dbee72d44a9e367456ba5df553dc67b945", 11,
+                           "19c97e74720f87619a8082793fbe4b69baf951cd", 29),
+            "2025-12-18": ("627570d47d7df03e1cc3de7cfb175555f4bc881f", 11,
+                           "d89b3988fe08b475335b6e0bcadbff3d213a3836", 36),
+            "2026-05-28": ("c9c60b2c55e72cedb757b0422f4ea160a9bffc5c", 8,
+                           "fcd0585957ab5137771b256dad3714dcaf44b7db", 49)}
+
+def release_files(alt):
+    nrev, nn, erev, ne = releases[alt]
+    return ([f"kg-nodes/resolve/{nrev}/data/nodes/train-{i:05d}-of-{nn:05d}.parquet"
+             for i in range(nn)],
+            [f"kg-edges/resolve/{erev}/data/edges/train-{i:05d}-of-{ne:05d}.parquet"
+             for i in range(ne)])
+
+files = [(tuple(sum(release_files(alt), [])), alt, None) for alt in releases]
+
+def fetch_upstream(force=False):
+    return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
+                                force)
+
+# For checks only: if set, only (about) this many edges are read
+_check_rows = None
+
+# columns of the edges stored as they are (the other string columns go into
+# "other_attributes")
+ECOLS = {"predicate", "primary_knowledge_source", "primary_knowledge_sources",
+         "aggregator_knowledge_source", "upstream_data_source",
+         "knowledge_level", "agent_type", "publications", "num_references",
+         "num_sentences", "has_confidence_score", "extraction_confidence_score",
+         "affinity"}
+# names of the node properties for the node columns
+VCOLS = {"id": "name", "name": "label", "category": "category",
+         "all_categories": "all_categories", "description": "description",
+         "equivalent_identifiers": "equivalent_identifiers",
+         "publications": "publications", "labels": "labels",
+         "international_resource_identifier": "iri",
+         "upstream_data_source": "upstream_data_source"}
+
+def ptype(t):
+    if pyarrow.types.is_list(t) or pyarrow.types.is_large_list(t):
+        return "vector<string>"
+    if pyarrow.types.is_integer(t):
+        return "int64_t"
+    if pyarrow.types.is_floating(t):
+        return "double"
+    if pyarrow.types.is_boolean(t):
+        return "bool"
+    return "string"
+
+def pyvals(col, t):
+    "The values of the arrow column as Python values for a property of type t."
+    if t == "int64_t":
+        return col.fill_null(-1).to_numpy(zero_copy_only=False)
+    if t == "double":
+        return col.cast(pyarrow.float64()).fill_null(numpy.nan).to_numpy(zero_copy_only=False)
+    if t == "bool":
+        return col.fill_null(False).to_numpy(zero_copy_only=False)
+    if t == "string":
+        return [x if x is not None else "" for x in col.to_pylist()]
+    return [x if x is not None else [] for x in col.to_pylist()]
+
+def joined(col):
+    "The column as strings, with the lists joined by '|', and nulls empty."
+    if pyarrow.types.is_list(col.type) or pyarrow.types.is_large_list(col.type):
+        col = pc.binary_join(col, "|")
+    return col.cast(pyarrow.string()).fill_null("")
+
+def global_codes(col, index):
+    """The codes of the values of the string column `col` in the dictionary
+    `index` (of values to codes), which is extended with the new values."""
+    d = col.dictionary_encode()
+    m = numpy.array([index.setdefault(x, len(index))
+                     for x in d.dictionary.to_pylist()], dtype="int32")
+    return m[d.indices.to_numpy(zero_copy_only=False)]
+
+def open_parquet(name, fname):
+    "The parquet file (stored compressed), as a seekable file."
+    return open_upstream_file(name, fname, "rb", seekable=True)
+
+@cache_network()
+@coerce_props()
+@annotate()
+def parse(alts=None):
+    name = __name__.split(".")[-1]
+    for alt in releases:
+        if alts is not None and alt not in alts:
+            continue
+        nfiles, efiles = release_files(alt)
+        # nodes, file by file (the node tables are large: descriptions,
+        # identifiers)
+        if _check_rows is not None:
+            nfiles = nfiles[:1]
+        g = Graph(directed=True)
+        ids = []
+        vprops = {}
+        for fname in nfiles:
+            with open_parquet(name, fname) as f:
+                t = pq.read_table(f)
+            N = g.num_vertices()
+            g.add_vertex(t.num_rows)
+            ids.append(t.column("id").combine_chunks())
+            for c in t.column_names:
+                if c not in VCOLS:
+                    continue
+                pt = ptype(t.schema.field(c).type)
+                if c not in vprops:
+                    vprops[c] = g.new_vp(pt)
+                p = vprops[c]
+                if pt in ("int64_t", "double", "bool"):
+                    p.a[N:] = pyvals(t.column(c), pt)
+                else:
+                    for i, x in enumerate(pyvals(t.column(c), pt)):
+                        p[N + i] = x
+            del t
+        ids = pyarrow.concat_arrays(ids)
+        assert pc.count_distinct(ids).as_py() == len(ids)
+        for c, p in vprops.items():
+            g.vp[VCOLS[c]] = p
+        extra = {}
+        # edges: the numeric columns and the strings with few distinct
+        # values (as codes into dictionaries) are collected as arrays, and
+        # stored at the end; the publications and the other attributes are
+        # stored as the edges are added
+        n = 0
+        ecols = None
+        nums, codes, index = {}, {}, {}
+        pubs = g.new_ep("string")
+        oprop = g.new_ep("string")
+        for fname in efiles:
+            if _check_rows is not None and n >= _check_rows:
+                break
+            with open_parquet(name, fname) as f:
+                pf = pq.ParquetFile(f)
+                schema = pf.schema_arrow
+                if ecols is None:
+                    etypes = {c: ptype(schema.field(c).type) for c in schema.names}
+                    ecols = [c for c in schema.names
+                             if (c in ECOLS or etypes[c] in ("int64_t", "double", "bool"))
+                             and c not in ("subject", "object", "publications")]
+                    other = [c for c in schema.names if c not in ecols and
+                             c not in ("subject", "object", "publications")]
+                    for c in ecols:
+                        if etypes[c] in ("int64_t", "double", "bool"):
+                            nums[c] = []
+                        else:
+                            codes[c], index[c] = [], {}
+                for b in pf.iter_batches(batch_size=1 << 18):
+                    n += b.num_rows
+                    ends = []
+                    for c in ("subject", "object"):
+                        col = b.column(c)
+                        idx = pc.index_in(col, value_set=ids)
+                        miss = pc.is_null(idx).to_numpy(zero_copy_only=False)
+                        if miss.any():
+                            # entities absent from the list of nodes
+                            for x in pc.unique(col.filter(miss)).to_pylist():
+                                if x not in extra:
+                                    extra[x] = len(ids) + len(extra)
+                            v = idx.fill_null(-1).to_numpy(zero_copy_only=False).copy()
+                            v[miss] = [extra[x] for x in col.filter(miss).to_pylist()]
+                        else:
+                            v = idx.to_numpy(zero_copy_only=False)
+                        ends.append(v)
+                    if extra and len(ids) + len(extra) > g.num_vertices():
+                        g.add_vertex(len(ids) + len(extra) - g.num_vertices())
+                    for c in nums:
+                        nums[c].append(pyvals(b.column(c), etypes[c]))
+                    for c in codes:
+                        codes[c].append(global_codes(joined(b.column(c)), index[c]))
+                    if "publications" in schema.names:
+                        pv = joined(b.column("publications")).to_pylist()
+                    else:
+                        pv = [""] * b.num_rows
+                    ocols = [b.column(c).to_pylist() for c in other]
+                    ovals = [json.dumps({c: x for c, x in zip(other, xs)
+                                         if x is not None})
+                             if any(x is not None for x in xs) else ""
+                             for xs in zip(*ocols)] if other else [""] * b.num_rows
+                    g.add_edge_list(zip(ends[0], ends[1], pv, ovals),
+                                    eprops=[pubs, oprop])
+                    del pv, ocols, ovals, ends
+        for c in ecols:
+            if c in nums:
+                g.ep[c] = g.new_ep(etypes[c], vals=numpy.concatenate(nums[c]))
+                nums[c] = None
+            else:
+                cp = g.new_ep("int32_t", vals=numpy.concatenate(codes[c]))
+                codes[c] = None
+                labels = list(index[c].keys())
+                p = g.new_ep("string")
+                map_property_values(cp, p, lambda i: labels[i])
+                g.ep[c] = p
+                del cp
+        g.ep.publications = pubs
+        g.ep.other_attributes = oprop
+        in_nodes = g.new_vp("bool", val=True)
+        if extra:
+            in_nodes.a[len(ids):] = False
+            for x, v in extra.items():
+                g.vp.name[v] = x
+        g.vp.in_nodes = in_nodes
+        yield alt, g