History of ctd_associations
The changes to the source code of this entry in the repository, newest first, each with the differences it made to the entry's files. They are also listed on GitLab.
[entries] Add the date when each entry was added, and remove icon_hash beff7c9
The dates are those of the commits that first added the entries' files (2020-08-11 for the entries of the initial commit); renamed entries keep the date of their first name. Changed: none
diff --git a/entries/ctd_associations/__init__.py b/entries/ctd_associations/__init__.py
index 8868f28..92478e5 100644
--- a/entries/ctd_associations/__init__.py
+++ b/entries/ctd_associations/__init__.py
@@ -55,7 +55,7 @@ tags = ['Biological', 'Chemical', 'Public Health', 'Unweighted', 'Multigraph', '
url = ['https://ctdbase.org/', 'https://ctdbase.org/downloads/']
citation = [('A. P. Davis, T. C. Wiegers, D. Sciaky, F. Barkalow, M. Strong, B. Wyatt, J. Wiegers, R. McMorran, S. Abrar, C. J. Mattingly, "Comparative Toxicogenomics Database\'s 20th anniversary: update 2025." Nucleic Acids Research 53, D1328-D1334 (2025)',
'https://doi.org/10.1093/nar/gkae883')]
-icon_hash = None
+date_added = '2026-10-28'
# The legal notices of CTD (ctdbase.org/about/legal.jsp, now behind a
# browser verification; read from the Internet Archive snapshot of
# 2025-04-05): use only for research and educational purposes, reproduction
[entries] Prefix the properties in the descriptions with (Node), (Edge) or (Graph) bc3dbbd
The lists of properties at the end of the descriptions marked edge properties with "Edge" but left node properties unmarked, or grouped them in "Nodes: `a`, ...; `b`, ..." items, so the kind of a property was often unclear. Every item now starts with (Node), (Edge) or (Graph), followed by the property names, a colon and the explanation, one item per property or group of properties; qualifications (kinds of nodes, networks) follow the names in parentheses. Properties that were only named got a short explanation, and a few items that named the wrong kind of property were corrected.
diff --git a/entries/ctd_associations/__init__.py b/entries/ctd_associations/__init__.py
index 0710cb0..8868f28 100644
--- a/entries/ctd_associations/__init__.py
+++ b/entries/ctd_associations/__init__.py
@@ -35,22 +35,22 @@ The diseases are terms of the "merged disease vocabulary" (MEDIC) of CTD, which
The networks have the following properties:
-- `ctd_id`: the identifier of the node: the MeSH identifier of the chemical (e.g. `D001241`, aspirin), the NCBI Gene identifier of the gene (e.g. `5743`), or the MeSH or OMIM identifier of the disease (e.g. `MESH:D003920`, `OMIM:125853`).
-- `name`: the name of the chemical, the symbol of the gene, or the name of the disease.
-- `cas_rn`: the CAS Registry Number of the chemical (empty if none, and for other nodes).
-- `is_chemical` (`chemical_disease`, `chemical_gene`): whether the node is a chemical.
-- `is_gene` (`gene_disease`, `gene_disease_inferred`): whether the node is a gene (otherwise it is a disease).
-- Edge `direct_evidence` (`chemical_disease`, `gene_disease`): the kind of curated evidence, `marker/mechanism` or `therapeutic` (empty for inferred associations).
-- Edge `inference_gene` (`chemical_disease`): the symbol of the gene through which the association is inferred (empty for curated associations).
-- Edge `inference_score` (`chemical_disease`, `gene_disease_inferred`): the inference score of the association (NaN for curated associations).
-- Edge `inference_chemicals` (`gene_disease_inferred`): the names of the chemicals through which the association is inferred, separated by `|`; `num_chemicals`, their number.
-- Edge `omim_ids` (`chemical_disease`, `gene_disease`): the OMIM identifiers of the evidence, separated by `|` (empty if none).
-- Edge `pubmed_ids`: the PubMed identifiers of the articles that support the association or interaction, separated by `|` (in `gene_disease_inferred`, those of all its inference chemicals).
-- Edge `organism`, `organism_id` (`chemical_gene`): the name and NCBI taxonomy identifier of the organism of the interaction (empty and -1 if not given).
-- Edge `gene_forms` (`chemical_gene`): the forms of the gene product involved, e.g. `protein`, `mRNA`, separated by `|`.
-- Edge `interaction` (`chemical_gene`): the description of the interaction.
-- Edge `interaction_actions` (`chemical_gene`): the actions of the interaction, e.g. `increases^expression`, separated by `|`.
-- `inferred_associations` (graph, `gene_disease`): the number of inferred gene-disease associations, which are in `gene_disease_inferred`."""
+- (Node) `ctd_id`: the identifier of the node: the MeSH identifier of the chemical (e.g. `D001241`, aspirin), the NCBI Gene identifier of the gene (e.g. `5743`), or the MeSH or OMIM identifier of the disease (e.g. `MESH:D003920`, `OMIM:125853`).
+- (Node) `name`: the name of the chemical, the symbol of the gene, or the name of the disease.
+- (Node) `cas_rn`: the CAS Registry Number of the chemical (empty if none, and for other nodes).
+- (Node) `is_chemical` (`chemical_disease`, `chemical_gene`): whether the node is a chemical.
+- (Node) `is_gene` (`gene_disease`, `gene_disease_inferred`): whether the node is a gene (otherwise it is a disease).
+- (Edge) `direct_evidence` (`chemical_disease`, `gene_disease`): the kind of curated evidence, `marker/mechanism` or `therapeutic` (empty for inferred associations).
+- (Edge) `inference_gene` (`chemical_disease`): the symbol of the gene through which the association is inferred (empty for curated associations).
+- (Edge) `inference_score` (`chemical_disease`, `gene_disease_inferred`): the inference score of the association (NaN for curated associations).
+- (Edge) `inference_chemicals` (`gene_disease_inferred`): the names of the chemicals through which the association is inferred, separated by `|`; `num_chemicals`, their number.
+- (Edge) `omim_ids` (`chemical_disease`, `gene_disease`): the OMIM identifiers of the evidence, separated by `|` (empty if none).
+- (Edge) `pubmed_ids`: the PubMed identifiers of the articles that support the association or interaction, separated by `|` (in `gene_disease_inferred`, those of all its inference chemicals).
+- (Edge) `organism`, `organism_id` (`chemical_gene`): the name and NCBI taxonomy identifier of the organism of the interaction (empty and -1 if not given).
+- (Edge) `gene_forms` (`chemical_gene`): the forms of the gene product involved, e.g. `protein`, `mRNA`, separated by `|`.
+- (Edge) `interaction` (`chemical_gene`): the description of the interaction.
+- (Edge) `interaction_actions` (`chemical_gene`): the actions of the interaction, e.g. `increases^expression`, separated by `|`.
+- (Node) `inferred_associations` (graph, `gene_disease`): the number of inferred gene-disease associations, which are in `gene_disease_inferred`."""
tags = ['Biological', 'Chemical', 'Public Health', 'Unweighted', 'Multigraph', 'Metadata']
url = ['https://ctdbase.org/', 'https://ctdbase.org/downloads/']
citation = [('A. P. Davis, T. C. Wiegers, D. Sciaky, F. Barkalow, M. Strong, B. Wyatt, J. Wiegers, R. McMorran, S. Abrar, C. J. Mattingly, "Comparative Toxicogenomics Database\'s 20th anniversary: update 2025." Nucleic Acids Research 53, D1328-D1334 (2025)',
[entries] ctd_associations: add the gene-disease associations inferred through chemicals ff687fa
The inferred gene-disease associations of CTD were left out, as more than 100 million rows. They are now the network gene_disease_inferred, with one edge per gene-disease pair (about 36 million): CTD lists the association once per inference chemical, and these rows (consecutive in the file) are merged into one edge, with the names of the chemicals, their number, the inference score of the pair and the union of the PubMed identifiers. The network is built from the file already used for gene_disease; the other networks are unchanged.
diff --git a/entries/ctd_associations/__init__.py b/entries/ctd_associations/__init__.py
index 5bd6528..0710cb0 100644
--- a/entries/ctd_associations/__init__.py
+++ b/entries/ctd_associations/__init__.py
@@ -16,6 +16,7 @@
# You should have received a copy of the GNU Affero General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
+import array
import pyarrow
import pyarrow.csv
import pyarrow.compute as pc
@@ -27,7 +28,8 @@ description = r"""Bipartite networks of chemicals, genes and diseases from the C
- `chemical_disease`: chemicals and diseases. An edge is either a curated association from the literature, with its direct evidence (`marker/mechanism`: the chemical correlates with the disease or may play a role in its etiology; `therapeutic`: the chemical has a known or potential therapeutic role in the disease), or an association inferred by CTD through a gene that has curated interactions with the chemical and a curated association with the disease, with the inference gene and an inference score (higher for better connected networks of chemicals, genes and diseases). A chemical and a disease are connected by one edge per inference gene, and by an edge per kind of direct evidence.
- `chemical_gene`: chemicals and genes. An edge is a curated interaction between a chemical and a gene product (or the gene itself) in an organism, e.g. "Aspirin results in decreased expression of PTGS2 mRNA", with the interaction actions (e.g. `decreases^expression`); the genes are identified by their NCBI Gene identifier, irrespective of the organism of the interaction (CTD maps the genes of all organisms to the identifiers of the genes it lists, mostly of vertebrates).
-- `gene_disease`: genes and diseases, with an edge for each curated association (marker/mechanism or therapeutic) of a gene with a disease. The gene-disease associations inferred by CTD through chemicals (more than 100 million, which can be derived from the other two networks) are left out.
+- `gene_disease`: genes and diseases, with an edge for each curated association (marker/mechanism or therapeutic) of a gene with a disease.
+- `gene_disease_inferred`: genes and diseases, with an edge for each association inferred by CTD through chemicals that have curated interactions with the gene and curated associations with the disease, with the inference chemicals and the inference score of the gene-disease pair. CTD lists the association once per inference chemical (130 million rows in the release of September 2026); these are merged into one edge per gene-disease pair (about 36 million).
The diseases are terms of the "merged disease vocabulary" (MEDIC) of CTD, which combines the MeSH disease terms and OMIM; the chemicals are MeSH terms. The chemical-pathway, gene-pathway, disease-pathway, chemical-phenotype and exposure data of CTD are not included.
@@ -37,17 +39,18 @@ The networks have the following properties:
- `name`: the name of the chemical, the symbol of the gene, or the name of the disease.
- `cas_rn`: the CAS Registry Number of the chemical (empty if none, and for other nodes).
- `is_chemical` (`chemical_disease`, `chemical_gene`): whether the node is a chemical.
-- `is_gene` (`gene_disease`): whether the node is a gene (otherwise it is a disease).
+- `is_gene` (`gene_disease`, `gene_disease_inferred`): whether the node is a gene (otherwise it is a disease).
- Edge `direct_evidence` (`chemical_disease`, `gene_disease`): the kind of curated evidence, `marker/mechanism` or `therapeutic` (empty for inferred associations).
- Edge `inference_gene` (`chemical_disease`): the symbol of the gene through which the association is inferred (empty for curated associations).
-- Edge `inference_score` (`chemical_disease`): the inference score of the association (NaN for curated associations).
+- Edge `inference_score` (`chemical_disease`, `gene_disease_inferred`): the inference score of the association (NaN for curated associations).
+- Edge `inference_chemicals` (`gene_disease_inferred`): the names of the chemicals through which the association is inferred, separated by `|`; `num_chemicals`, their number.
- Edge `omim_ids` (`chemical_disease`, `gene_disease`): the OMIM identifiers of the evidence, separated by `|` (empty if none).
-- Edge `pubmed_ids`: the PubMed identifiers of the articles that support the association or interaction, separated by `|`.
+- Edge `pubmed_ids`: the PubMed identifiers of the articles that support the association or interaction, separated by `|` (in `gene_disease_inferred`, those of all its inference chemicals).
- Edge `organism`, `organism_id` (`chemical_gene`): the name and NCBI taxonomy identifier of the organism of the interaction (empty and -1 if not given).
- Edge `gene_forms` (`chemical_gene`): the forms of the gene product involved, e.g. `protein`, `mRNA`, separated by `|`.
- Edge `interaction` (`chemical_gene`): the description of the interaction.
- Edge `interaction_actions` (`chemical_gene`): the actions of the interaction, e.g. `increases^expression`, separated by `|`.
-- `inferred_associations` (graph, `gene_disease`): the number of inferred gene-disease associations that are left out."""
+- `inferred_associations` (graph, `gene_disease`): the number of inferred gene-disease associations, which are in `gene_disease_inferred`."""
tags = ['Biological', 'Chemical', 'Public Health', 'Unweighted', 'Multigraph', 'Metadata']
url = ['https://ctdbase.org/', 'https://ctdbase.org/downloads/']
citation = [('A. P. Davis, T. C. Wiegers, D. Sciaky, F. Barkalow, M. Strong, B. Wyatt, J. Wiegers, R. McMorran, S. Abrar, C. J. Mattingly, "Comparative Toxicogenomics Database\'s 20th anniversary: update 2025." Nucleic Acids Research 53, D1328-D1334 (2025)',
@@ -67,7 +70,8 @@ upstream_license = ("CTD legal notices: research and educational use only; no co
upstream_prefix = 'https://ctdbase.org/reports'
files = [("CTD_chemicals_diseases.tsv.gz", "chemical_disease", None),
("CTD_chem_gene_ixns.tsv.gz", "chemical_gene", None),
- ("CTD_genes_diseases.tsv.gz", "gene_disease", None)]
+ ("CTD_genes_diseases.tsv.gz", "gene_disease", None),
+ ("CTD_genes_diseases.tsv.gz", "gene_disease_inferred", None)]
COLUMNS = {
"chemical_disease": ["ChemicalName", "ChemicalID", "CasRN", "DiseaseName",
@@ -79,6 +83,11 @@ COLUMNS = {
"gene_disease": ["GeneSymbol", "GeneID", "DiseaseName", "DiseaseID",
"DirectEvidence", "InferenceChemicalName",
"InferenceScore", "OmimIDs", "PubMedIDs"]}
+COLUMNS["gene_disease_inferred"] = COLUMNS["gene_disease"]
+
+# For checks only: if set, only (about) this many rows of the inferred
+# gene-disease associations are read
+_check_rows = None
def fetch_upstream(force=False):
return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
@@ -135,6 +144,122 @@ def first_values(t, idcol, cols, ids):
first = pc.index_in(ids, value_set=t.column(idcol))
return [t.column(c).take(first).to_pylist() for c in cols]
+def inferred_gene_disease(name, fname):
+ """The network of the inferred gene-disease associations, with one edge
+ per gene-disease pair. The rows (more than 100 million, one per pair and
+ inference chemical) are read block by block; the rows of a pair are
+ consecutive in the file, and are merged into one edge, with the names of
+ the chemicals and the union of the PubMed identifiers (a pair split
+ between two blocks is carried over to the next one)."""
+ cols = COLUMNS["gene_disease_inferred"]
+ skip = comment_lines(name, fname)
+ vidx = {}
+ vnames = {}
+ # numbers in compact arrays, and the strings in Arrow arrays (one per
+ # block), to save memory
+ src, tgt = array.array("q"), array.array("q")
+ score, nchem = array.array("d"), array.array("i")
+ chems, pmids = [], []
+ carry = None
+ n = 0
+
+ def runs(gs, ds, sc, cn, pm, final):
+ "Merge the consecutive rows of each pair (except an unfinished last one)."
+ if len(gs) == 0:
+ return 0
+ start = numpy.flatnonzero(numpy.r_[True, (gs[1:] != gs[:-1]) |
+ (ds[1:] != ds[:-1])])
+ end = numpy.r_[start[1:], len(gs)]
+ if final:
+ k = len(gs)
+ else:
+ k = start[-1]
+ start, end = start[:-1], end[:-1]
+ src.extend(gs[start])
+ tgt.extend(ds[start])
+ score.extend(sc[start])
+ nchem.extend((end - start).astype("int32"))
+ cs, ps = [], []
+ for a, b in zip(start.tolist(), end.tolist()):
+ cs.append("|".join(cn[a:b]))
+ ids = set()
+ for x in pm[a:b]:
+ if x:
+ ids.update(x.split("|"))
+ ps.append("|".join(sorted(ids, key=int)))
+ chems.append(pyarrow.array(cs, type=pyarrow.large_string()))
+ pmids.append(pyarrow.array(ps, type=pyarrow.large_string()))
+ return k
+
+ with open_upstream_file(name, fname, "rb") as f:
+ reader = pyarrow.csv.open_csv(
+ f,
+ read_options=pyarrow.csv.ReadOptions(column_names=cols,
+ skip_rows=skip,
+ block_size=1 << 26),
+ parse_options=pyarrow.csv.ParseOptions(delimiter="\t",
+ quote_char=False),
+ convert_options=pyarrow.csv.ConvertOptions(
+ column_types=dict({c: pyarrow.string() for c in cols},
+ InferenceScore=pyarrow.float64()),
+ strings_can_be_null=False))
+ for b in reader:
+ if _check_rows is not None and n >= _check_rows:
+ break
+ b = b.filter(pc.equal(b.column("DirectEvidence"), ""))
+ n += b.num_rows
+ assert pc.all(pc.equal(b.column("OmimIDs"), "")).as_py()
+ codes = []
+ for idc, namec, is_gene in (("GeneID", "GeneSymbol", True),
+ ("DiseaseID", "DiseaseName", False)):
+ col = b.column(idc)
+ u = pc.unique(col)
+ first = pc.index_in(u, value_set=col)
+ keys = [(is_gene, x) for x in u.to_pylist()]
+ for k, nm in zip(keys, b.column(namec).take(first).to_pylist()):
+ if k not in vidx:
+ vidx[k] = len(vidx)
+ vnames[k] = nm
+ m = numpy.array([vidx[k] for k in keys], dtype="int64")
+ codes.append(m[pc.index_in(col, value_set=u).to_numpy()])
+ block = [codes[0], codes[1],
+ b.column("InferenceScore").to_numpy(),
+ b.column("InferenceChemicalName").to_pylist(),
+ b.column("PubMedIDs").to_pylist()]
+ del b, codes
+ if carry is not None:
+ block = [numpy.r_[carry[0], block[0]], numpy.r_[carry[1], block[1]],
+ numpy.r_[carry[2], block[2]], carry[3] + block[3],
+ carry[4] + block[4]]
+ k = runs(*block, final=False)
+ carry = [x[k:] for x in block]
+ del block
+ if carry is not None:
+ runs(*carry, final=True)
+ del carry
+ g = Graph(directed=False)
+ g.add_vertex(len(vidx))
+ g.add_edge_list(numpy.column_stack([numpy.frombuffer(src, dtype="int64"),
+ numpy.frombuffer(tgt, dtype="int64")]))
+ del src, tgt
+ keys = list(vidx.keys())
+ g.vp.ctd_id = g.new_vp("string", vals=[k[1] for k in keys])
+ g.vp.name = g.new_vp("string", vals=[vnames[k] for k in keys])
+ g.vp.is_gene = g.new_vp("bool", vals=[k[0] for k in keys])
+ del vidx, vnames, keys
+ g.ep.inference_score = g.new_ep("double", vals=numpy.frombuffer(score))
+ g.ep.num_chemicals = g.new_ep("int32_t",
+ vals=numpy.frombuffer(nchem, dtype="int32"))
+ del score, nchem
+ def drain(chunks):
+ "The strings of the chunks, each chunk freed once it is read."
+ chunks.reverse()
+ while chunks:
+ yield from chunks.pop().to_pylist()
+ for pname, chunks in (("inference_chemicals", chems), ("pubmed_ids", pmids)):
+ g.ep[pname] = g.new_ep("string", vals=drain(chunks))
+ return g
+
@cache_network()
@coerce_props()
@annotate()
@@ -143,6 +268,9 @@ def parse(alts=None):
for fname, alt, fmt in files:
if alts is not None and alt not in alts:
continue
+ if alt == "gene_disease_inferred":
+ yield alt, inferred_gene_disease(name, fname)
+ continue
cols = COLUMNS[alt]
t, nrows = read_rows(name, fname, cols,
keep="DirectEvidence" if alt == "gene_disease" else None)
[entries] New entry: ctd_associations ffdcf78
diff --git a/entries/ctd_associations/__init__.py b/entries/ctd_associations/__init__.py
new file mode 100644
index 0000000..5bd6528
--- /dev/null
+++ b/entries/ctd_associations/__init__.py
@@ -0,0 +1,192 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2026 Tiago de Paula Peixoto <tiago@skewed.de>
+#
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option) any
+# later version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along
+# with this program. If not, see <http://www.gnu.org/licenses/>.
+
+import pyarrow
+import pyarrow.csv
+import pyarrow.compute as pc
+
+from .. import *
+
+title = "Comparative Toxicogenomics Database associations (CTD, 2026)"
+description = r"""Bipartite networks of chemicals, genes and diseases from the Comparative Toxicogenomics Database (CTD), which curates from the scientific literature how environmental chemicals and drugs interact with genes and proteins, and how chemicals and genes are related to diseases, in many species. The data are those of the release of September 2026.
+
+- `chemical_disease`: chemicals and diseases. An edge is either a curated association from the literature, with its direct evidence (`marker/mechanism`: the chemical correlates with the disease or may play a role in its etiology; `therapeutic`: the chemical has a known or potential therapeutic role in the disease), or an association inferred by CTD through a gene that has curated interactions with the chemical and a curated association with the disease, with the inference gene and an inference score (higher for better connected networks of chemicals, genes and diseases). A chemical and a disease are connected by one edge per inference gene, and by an edge per kind of direct evidence.
+- `chemical_gene`: chemicals and genes. An edge is a curated interaction between a chemical and a gene product (or the gene itself) in an organism, e.g. "Aspirin results in decreased expression of PTGS2 mRNA", with the interaction actions (e.g. `decreases^expression`); the genes are identified by their NCBI Gene identifier, irrespective of the organism of the interaction (CTD maps the genes of all organisms to the identifiers of the genes it lists, mostly of vertebrates).
+- `gene_disease`: genes and diseases, with an edge for each curated association (marker/mechanism or therapeutic) of a gene with a disease. The gene-disease associations inferred by CTD through chemicals (more than 100 million, which can be derived from the other two networks) are left out.
+
+The diseases are terms of the "merged disease vocabulary" (MEDIC) of CTD, which combines the MeSH disease terms and OMIM; the chemicals are MeSH terms. The chemical-pathway, gene-pathway, disease-pathway, chemical-phenotype and exposure data of CTD are not included.
+
+The networks have the following properties:
+
+- `ctd_id`: the identifier of the node: the MeSH identifier of the chemical (e.g. `D001241`, aspirin), the NCBI Gene identifier of the gene (e.g. `5743`), or the MeSH or OMIM identifier of the disease (e.g. `MESH:D003920`, `OMIM:125853`).
+- `name`: the name of the chemical, the symbol of the gene, or the name of the disease.
+- `cas_rn`: the CAS Registry Number of the chemical (empty if none, and for other nodes).
+- `is_chemical` (`chemical_disease`, `chemical_gene`): whether the node is a chemical.
+- `is_gene` (`gene_disease`): whether the node is a gene (otherwise it is a disease).
+- Edge `direct_evidence` (`chemical_disease`, `gene_disease`): the kind of curated evidence, `marker/mechanism` or `therapeutic` (empty for inferred associations).
+- Edge `inference_gene` (`chemical_disease`): the symbol of the gene through which the association is inferred (empty for curated associations).
+- Edge `inference_score` (`chemical_disease`): the inference score of the association (NaN for curated associations).
+- Edge `omim_ids` (`chemical_disease`, `gene_disease`): the OMIM identifiers of the evidence, separated by `|` (empty if none).
+- Edge `pubmed_ids`: the PubMed identifiers of the articles that support the association or interaction, separated by `|`.
+- Edge `organism`, `organism_id` (`chemical_gene`): the name and NCBI taxonomy identifier of the organism of the interaction (empty and -1 if not given).
+- Edge `gene_forms` (`chemical_gene`): the forms of the gene product involved, e.g. `protein`, `mRNA`, separated by `|`.
+- Edge `interaction` (`chemical_gene`): the description of the interaction.
+- Edge `interaction_actions` (`chemical_gene`): the actions of the interaction, e.g. `increases^expression`, separated by `|`.
+- `inferred_associations` (graph, `gene_disease`): the number of inferred gene-disease associations that are left out."""
+tags = ['Biological', 'Chemical', 'Public Health', 'Unweighted', 'Multigraph', 'Metadata']
+url = ['https://ctdbase.org/', 'https://ctdbase.org/downloads/']
+citation = [('A. P. Davis, T. C. Wiegers, D. Sciaky, F. Barkalow, M. Strong, B. Wyatt, J. Wiegers, R. McMorran, S. Abrar, C. J. Mattingly, "Comparative Toxicogenomics Database\'s 20th anniversary: update 2025." Nucleic Acids Research 53, D1328-D1334 (2025)',
+ 'https://doi.org/10.1093/nar/gkae883')]
+icon_hash = None
+# The legal notices of CTD (ctdbase.org/about/legal.jsp, now behind a
+# browser verification; read from the Internet Archive snapshot of
+# 2025-04-05): use only for research and educational purposes, reproduction
+# or use for commercial purposes prohibited without written permission;
+# publications that use CTD data must cite it, link to its pages, and notify
+# CTD. Redistribution is not forbidden.
+upstream_license = ("CTD legal notices: research and educational use only; no commercial use without permission; cite and link CTD",
+ "https://ctdbase.org/about/legal.jsp")
+# The files are replaced in place every month (no versioned releases): the
+# cached copies (of the release of September 2026, at the time of writing)
+# are the ones used
+upstream_prefix = 'https://ctdbase.org/reports'
+files = [("CTD_chemicals_diseases.tsv.gz", "chemical_disease", None),
+ ("CTD_chem_gene_ixns.tsv.gz", "chemical_gene", None),
+ ("CTD_genes_diseases.tsv.gz", "gene_disease", None)]
+
+COLUMNS = {
+ "chemical_disease": ["ChemicalName", "ChemicalID", "CasRN", "DiseaseName",
+ "DiseaseID", "DirectEvidence", "InferenceGeneSymbol",
+ "InferenceScore", "OmimIDs", "PubMedIDs"],
+ "chemical_gene": ["ChemicalName", "ChemicalID", "CasRN", "GeneSymbol",
+ "GeneID", "GeneForms", "Organism", "OrganismID",
+ "Interaction", "InteractionActions", "PubMedIDs"],
+ "gene_disease": ["GeneSymbol", "GeneID", "DiseaseName", "DiseaseID",
+ "DirectEvidence", "InferenceChemicalName",
+ "InferenceScore", "OmimIDs", "PubMedIDs"]}
+
+def fetch_upstream(force=False):
+ return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
+ force)
+
+def comment_lines(name, fname):
+ "The number of comment lines (starting with '#') at the top of the file."
+ n = 0
+ with open_upstream_file(name, fname, "rb") as f:
+ for line in f:
+ if not line.startswith(b"#"):
+ break
+ n += 1
+ return n
+
+def read_rows(name, fname, cols, keep=None):
+ """The table of the file, read in blocks; if `keep` is given, only the
+ rows with a non-empty value of that column are kept. Returns the table
+ and the number of rows read."""
+ skip = comment_lines(name, fname)
+ batches, n = [], 0
+ with open_upstream_file(name, fname, "rb") as f:
+ reader = pyarrow.csv.open_csv(
+ f,
+ read_options=pyarrow.csv.ReadOptions(column_names=cols,
+ skip_rows=skip,
+ block_size=1 << 26),
+ parse_options=pyarrow.csv.ParseOptions(delimiter="\t",
+ quote_char=False),
+ convert_options=pyarrow.csv.ConvertOptions(
+ column_types=dict({c: pyarrow.string() for c in cols},
+ InferenceScore=pyarrow.float64(),
+ OrganismID=pyarrow.int64()),
+ strings_can_be_null=False))
+ for b in reader:
+ n += b.num_rows
+ if keep is not None:
+ b = b.filter(pc.not_equal(b.column(keep), ""))
+ batches.append(b)
+ return pyarrow.Table.from_batches(batches), n
+
+def string_ep(g, values, name):
+ """Store the string column `values` as the string edge property `name`,
+ converted from integer codes (one Python string per distinct value)."""
+ d = values.combine_chunks().dictionary_encode()
+ codes = g.new_ep("int", vals=d.indices.to_numpy())
+ labels = d.dictionary.to_pylist()
+ p = g.new_ep("string")
+ map_property_values(codes, p, lambda i: labels[i])
+ g.ep[name] = p
+
+def first_values(t, idcol, cols, ids):
+ "The values of `cols` of the first row of each of `ids` in column `idcol`."
+ first = pc.index_in(ids, value_set=t.column(idcol))
+ return [t.column(c).take(first).to_pylist() for c in cols]
+
+@cache_network()
+@coerce_props()
+@annotate()
+def parse(alts=None):
+ name = __name__.split(".")[-1]
+ for fname, alt, fmt in files:
+ if alts is not None and alt not in alts:
+ continue
+ cols = COLUMNS[alt]
+ t, nrows = read_rows(name, fname, cols,
+ keep="DirectEvidence" if alt == "gene_disease" else None)
+ if alt == "chemical_disease":
+ a, b = ("ChemicalID", ["ChemicalName", "CasRN"]), ("DiseaseID", ["DiseaseName"])
+ elif alt == "chemical_gene":
+ a, b = ("ChemicalID", ["ChemicalName", "CasRN"]), ("GeneID", ["GeneSymbol"])
+ else:
+ a, b = ("GeneID", ["GeneSymbol"]), ("DiseaseID", ["DiseaseName"])
+ ia = pc.unique(t.column(a[0]))
+ ib = pc.unique(t.column(b[0]))
+ Na = len(ia)
+ g = Graph(directed=False)
+ g.add_vertex(Na + len(ib))
+ g.add_edge_list(numpy.column_stack(
+ [pc.index_in(t.column(a[0]), value_set=ia).to_numpy(),
+ Na + pc.index_in(t.column(b[0]), value_set=ib).to_numpy()]))
+ va = first_values(t, a[0], a[1], ia)
+ vb = first_values(t, b[0], b[1], ib)
+ g.vp.ctd_id = g.new_vp("string", vals=ia.to_pylist() + ib.to_pylist())
+ g.vp.name = g.new_vp("string", vals=va[0] + vb[0])
+ side = "is_gene" if alt == "gene_disease" else "is_chemical"
+ g.vp[side] = g.new_vp("bool")
+ g.vp[side].a[:Na] = 1
+ if alt != "gene_disease":
+ g.vp.cas_rn = g.new_vp("string", vals=va[1] + [""] * len(ib))
+ del va, vb, ia, ib
+ if alt == "chemical_disease":
+ string_ep(g, t.column("DirectEvidence"), "direct_evidence")
+ string_ep(g, t.column("InferenceGeneSymbol"), "inference_gene")
+ g.ep.inference_score = g.new_ep(
+ "double", vals=t.column("InferenceScore").to_numpy(zero_copy_only=False))
+ string_ep(g, t.column("OmimIDs"), "omim_ids")
+ elif alt == "chemical_gene":
+ string_ep(g, t.column("Organism"), "organism")
+ g.ep.organism_id = g.new_ep(
+ "int64_t", vals=t.column("OrganismID").fill_null(-1).to_numpy())
+ string_ep(g, t.column("GeneForms"), "gene_forms")
+ string_ep(g, t.column("Interaction"), "interaction")
+ string_ep(g, t.column("InteractionActions"), "interaction_actions")
+ else:
+ string_ep(g, t.column("DirectEvidence"), "direct_evidence")
+ string_ep(g, t.column("OmimIDs"), "omim_ids")
+ g.gp.inferred_associations = g.new_gp("int64_t", val=nrows - len(t))
+ string_ep(g, t.column("PubMedIDs"), "pubmed_ids")
+ del t
+ yield alt, g