History of regulondb
The changes to the source code of this entry in the repository, newest first, each with the differences it made to the entry's files. They are also listed on GitLab.
[entries] New entry: regulondb c272ac9
diff --git a/entries/regulondb/__init__.py b/entries/regulondb/__init__.py
new file mode 100644
index 0000000..e4048d2
--- /dev/null
+++ b/entries/regulondb/__init__.py
@@ -0,0 +1,320 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2020-2026 Tiago de Paula Peixoto <tiago@skewed.de>
+#
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option) any
+# later version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along
+# with this program. If not, see <http://www.gnu.org/licenses/>.
+
+
+from .. import *
+import process_entry
+import re
+from collections import defaultdict, Counter
+
+title = "RegulonDB E. coli transcriptional regulatory networks (2026)"
+description = """Networks of transcriptional regulation in the bacterium *Escherichia coli* K-12, from RegulonDB, the database of the Computational Genomics Program of the Center for Genomic Sciences (CCG) of the National Autonomous University of Mexico (UNAM), release 14.5 (2026). RegulonDB is curated mainly from the literature (and from high-throughput experiments), and records which regulators (transcription factors, but also small RNAs and compounds such as ppGpp) activate or repress the transcription of which genes and transcription units, and which sigma factors of the RNA polymerase recognize their promoters. Each regulatory interaction has a function (activation `+`, repression `-`, dual `-+`, or unknown) and a confidence level based on its evidence: confirmed (`C`), strong (`S`) or weak (`W`) (or unknown, `?`).
+
+The entry contains the network files of the release:
+
+- `regulator_gene`: directed edges from each regulator to the genes whose transcription it regulates (`NetworkRegulatorGene`, with the additional columns of `NetworkRegulatorGene_internal`).
+- `regulator_conformation_gene`: the same, with an edge for each active conformation of the regulator (e.g. a transcription factor bound to its effector) that regulates the gene (`NetworkRegulatorConfGene`); the regulator and the gene may be joined by several edges.
+- `regulator_tu`: directed edges from each regulator to the transcription units (sets of genes transcribed from a promoter) that it regulates (`NetworkRegulatorTU`).
+- `regulator_regulator`: directed edges between regulators, from each regulator to the regulators whose genes it regulates (`NetworkRegulatorRegulator`).
+- `sigma_gene` and `sigma_tu`: directed edges from each sigma factor to the genes or transcription units transcribed from promoters that it recognizes (`NetworkSigmaGene` and `NetworkSigmaTU`).
+
+The entry also contains the network of transcription factors and the genes they regulate of earlier releases (`network_tf_gene.txt`, the precursor of `NetworkRegulatorGene`), from the copies of the file kept by the Internet Archive, one network for each of its seven distinct versions:
+
+- `tf_gene_v9.2`: release 9.2 (2016-08-09), 4,424 interactions;
+- `tf_gene_v9.4`: release 9.4 (2017-08-05), 4,490 interactions;
+- `tf_gene_v10.0`: release 10.0 (2018-06-18), 4,564 interactions;
+- `tf_gene_v10.6.3`: release 10.6.3 (2019-02-12), 4,627 interactions;
+- `tf_gene_v10.7`: release 10.7 (2020-04-05), 4,670 interactions;
+- `tf_gene_v10.9`: release 10.9 (2021-06-29), 4,727 interactions;
+- `tf_gene_v11.0`: release 11.0 (2022-08-22), 4,775 interactions.
+
+These contain only transcription factors (not small RNAs or compounds). Each line of the file is an edge, so that a transcription factor and a gene can be joined by several edges with different functions (e.g. one for activation and one for repression, in about 200 cases). The files of releases 9.2 to 10.7 identify transcription factors and genes by their names only. The interactions of release 10.9 mostly have no confidence level (`null` upstream).
+
+Except for `regulator_regulator`, the networks are bipartite: regulators (or sigma factors) and their targets are distinct nodes, even when a regulator regulates its own gene. The releases of RegulonDB before 9.2 (e.g. version 8, of 2013) are no longer available for download; data derived from earlier releases are part of the entries `ecoli_transcription`, `gerstein_regulatory` and `signed_biological`.
+
+The files of this entry are available only with an access token, since the RegulonDB terms of use forbid integrating RegulonDB, in part or as a whole, into other databank systems without the prior written consent of CCG-UNAM (RegulonDB is free for academic and noncommercial use).
+
+The networks have the following properties:
+
+- `name`: the name of the regulator, sigma factor, gene or transcription unit (transcription units are named by their genes followed by their promoter in brackets, e.g. `fkpA[fkpAp1]`); `id`: the RegulonDB identifier (empty in `sigma_gene` and `sigma_tu`, and in the `tf_gene_<release>` networks before release 10.9, whose files give only names).
+- In `regulator_gene`, `regulator_conformation_gene`, `regulator_tu` and the `tf_gene_<release>` networks, `is_regulator`: whether the node is a regulator (otherwise a gene or a transcription unit).
+- In `regulator_gene`, `regulator_conformation_gene` and `regulator_tu`, `regulator_genes`: the gene(s) that code for the regulator (empty for regulators that are not proteins and for targets).
+- In `regulator_gene`, `regulator_type`: the kind of regulator (`TF`, `sRNA`, `compound` or `Regulator`; empty for genes); `synonyms`: the synonyms of the regulator or the gene.
+- In `sigma_gene` and `sigma_tu`, `is_sigma`: whether the node is a sigma factor.
+- Edge `function`: `+` (activation), `-` (repression), `-+` (dual; `+-` in the `tf_gene_<release>` networks), `?` or empty (unknown); `confidence`: the confidence level (`C`, `S`, `W` or `?`; in the `tf_gene_<release>` networks, the upstream levels `Confirmed`, `Strong` and `Weak` are written in the same way, and the level is empty where it is `null` upstream; the files of releases 9.2 and 10.6.3 do not list the column of the levels in their headers, but contain it). In release 10.7, whose file gives the function as `activator`, `repressor` or `unknown`, these are written `+`, `-` and `?`.
+- In the `tf_gene_<release>` networks, edge `evidence`: the evidence codes of the interaction (e.g. `GEA`, `BPP`).
+- In `regulator_gene`, edge `confidence_internal`: the confidence level given in the "internal" version of the file (`NetworkRegulatorGene_internal`), which differs from `confidence` for 207 of the 7,351 interactions; `evidence_category`: the categories of the evidence of the interaction (e.g. `Classical expression`), from the internal file.
+- In `regulator_conformation_gene`, edge `conformation`: the name of the active conformation of the regulator.
+- In `sigma_gene` and `sigma_tu`, edge `promoter_evidence`: the evidence codes for the promoter and its recognition by the sigma factor."""
+tags = ['Biological', 'Gene regulation', 'Unweighted', 'Metadata']
+url = ['https://regulondb.ccg.unam.mx/datasets',
+ 'https://web.archive.org/web/20221016225322/http://regulondb.ccg.unam.mx/menu/download/datasets/files/network_tf_gene.txt']
+citation = [('H. Salgado, S. Gama-Castro, P. Lara, C. Mejia-Almonte, G. Alarcón-Carranza, A. G. López-Almazo, et al., "RegulonDB v12.0: a comprehensive resource of transcriptional regulation in E. coli K-12." Nucleic Acids Research 52, D255-D264 (2024)',
+ 'https://doi.org/10.1093/nar/gkad1072')]
+icon_hash = None
+upstream_license = ("RegulonDB terms and conditions (free for academic/noncommercial use; no integration into other databank systems without written consent)",
+ "https://regulondb.ccg.unam.mx/manual/aboutUs/terms-conditions")
+restricted = True
+
+# The files are served by the GraphQL web service of RegulonDB
+upstream_prefix = 'https://regulondb.ccg.unam.mx/graphql'
+release = '14.5'
+
+# The server does not send the intermediate certificate of its chain; it is
+# fetched from the issuer's address (Authority Information Access), and the
+# chain is verified as usual.
+intermediate_cert = 'http://secure.globalsign.com/cacert/gsgccr46ovtlsca2025.crt'
+
+nets = [('regulator_gene', ('NetworkRegulatorGene', 'NetworkRegulatorGene_internal')),
+ ('regulator_conformation_gene', ('NetworkRegulatorConfGene',)),
+ ('regulator_tu', ('NetworkRegulatorTU',)),
+ ('regulator_regulator', ('NetworkRegulatorRegulator',)),
+ ('sigma_gene', ('NetworkSigmaGene',)),
+ ('sigma_tu', ('NetworkSigmaTU',))]
+files = [(tuple(f + '.tsv' for f in fs), alt, None) for alt, fs in nets]
+
+# Earlier releases of the TF-gene network (network_tf_gene.txt), from pinned
+# Internet Archive snapshots of its download page, one for each distinct
+# version of the file: release, Wayback timestamp
+old_releases = [('9.2', '20161231110502'), ('9.4', '20170914131743'),
+ ('10.0', '20180802125033'), ('10.6.3', '20200208170021'),
+ ('10.7', '20200623104158'), ('10.9', '20220307174256'),
+ ('11.0', '20221016225322')]
+old_tf_gene_url = 'http://regulondb.ccg.unam.mx/menu/download/datasets/files/network_tf_gene.txt'
+archived_files = {f'network_tf_gene_v{r}.txt': f'https://web.archive.org/web/{t}id_/{old_tf_gene_url}'
+ for r, t in old_releases}
+files += [(f'network_tf_gene_v{r}.txt', f'tf_gene_v{r}', None) for r, t in old_releases]
+
+def _ssl_context():
+ ctx = ssl.create_default_context()
+ with open_url(intermediate_cert, timeout=60) as f:
+ ctx.load_verify_locations(cadata=f.read())
+ return ctx
+
+def _query(ctx, query, variables={}):
+ data = json.dumps({"query": query, "variables": variables}).encode()
+ with open_url(upstream_prefix, headers={"Content-Type": "application/json"},
+ method="POST", data=data, context=ctx, timeout=600) as f:
+ ret = json.loads(f.read())
+ if ret.get("errors"):
+ raise IOError(f"RegulonDB query error: {ret['errors']}")
+ return ret["data"]
+
+def fetch_upstream(force=False):
+ name = __name__.split(".")[-1]
+ fetch_archived_files(name, archived_files, force, tries=3)
+ fnames = [f for fs, alt, fmt in files if not alt.startswith("tf_gene_")
+ for f in fs]
+ if not force and not process_entry._check_health and \
+ all(upstream_file_exists(name, f) for f in fnames):
+ return
+ ctx = _ssl_context()
+ meta = _query(ctx, "{ listAllDownloadableFiles { fileName rdbVersion } }")
+ versions = {x["fileName"]: x["rdbVersion"]
+ for x in meta["listAllDownloadableFiles"]}
+ for fname in fnames:
+ base = fname[:-len(".tsv")]
+ if versions.get(base) != release:
+ raise IOError(f"RegulonDB file {base}: release {versions.get(base)}, expected {release}")
+ if process_entry._check_health:
+ return
+ for fname in fnames:
+ if not force and upstream_file_exists(name, fname):
+ continue
+ base = fname[:-len(".tsv")]
+ d = _query(ctx, "query($f: String) { getDataOfFile(fileName: $f) { content } }",
+ {"f": base})
+ content = d["getDataOfFile"]["content"]
+ assert content.startswith("1)"), content[:100]
+ with open_upstream_file(name, fname, "wb") as output:
+ output.write(content.encode())
+
+def _read(name, fname, ncols):
+ with open_upstream_file(name, fname, "r") as f:
+ rows = [l.rstrip("\r\n").split("\t") for l in f if l.strip()]
+ rows = [[x.strip() for x in r] for r in rows]
+ assert rows[0][0].startswith("1)") and len(rows[0]) == ncols, rows[0]
+ rows = rows[1:]
+ assert all(len(r) == ncols for r in rows), [r for r in rows if len(r) != ncols][:3]
+ return rows
+
+def _graph(edges, vinfo, eprops):
+ """Build a network from the edges (source key, target key, edge property
+ values), the vertex information (key -> dict of properties) and the
+ names and types of the edge properties."""
+ g = Graph(directed=True)
+ vidx = {}
+ for k in vinfo:
+ vidx[k] = len(vidx)
+ g.add_vertex(len(vidx))
+ g.add_edge_list([(vidx[s], vidx[t]) for s, t, vals in edges])
+ vprops = {}
+ for k, info in vinfo.items():
+ for p, (t, x) in info.items():
+ if p not in vprops:
+ vprops[p] = g.new_vp(t)
+ vprops[p][g.vertex(vidx[k])] = x
+ for p, vp in vprops.items():
+ g.vp[p] = vp
+ for i, (p, t) in enumerate(eprops):
+ g.ep[p] = g.new_ep(t, vals=[vals[i] for s, t_, vals in edges])
+ return g
+
+def _parse_old_tf_gene(name, fname, release):
+ """A TF-gene network of an earlier release (network_tf_gene.txt)."""
+ with open_upstream_file(name, fname, "rb") as f:
+ data = f.read()
+ try:
+ text = data.decode("utf-8")
+ except UnicodeDecodeError:
+ text = data.decode("latin-1") # release 11.0 (in its comments)
+ m = re.search(r"^# Release: (\S+)", text, re.M)
+ assert m and m.group(1) == release, (fname, m and m.group(1))
+ rows = [l.split("\t") for l in text.splitlines()
+ if l.strip() and not l.startswith("#")]
+ ncols = {len(r) for r in rows}
+ assert len(ncols) == 1, ncols
+ ncols = ncols.pop()
+ funcs = {"activator": "+", "repressor": "-", "unknown": "?"}
+ confs = {"confirmed": "C", "strong": "S", "weak": "W", "null": ""}
+ vinfo = {}
+ edges = []
+ for r in rows:
+ if ncols == 8: # with identifiers (10.9, 11.0)
+ sid, sname, tid, tname, func, ev, cl = r[:7]
+ assert r[7] == ""
+ else:
+ sname, tname, func, ev, cl = r[:5]
+ assert ncols == 5 or r[5] == ""
+ sid = tid = ""
+ s = ("tf", sid or sname)
+ t = ("gene", tid or tname)
+ for k, n, i, reg in [(s, sname, sid, True), (t, tname, tid, False)]:
+ if k not in vinfo:
+ vinfo[k] = dict(name=("string", n), id=("string", i),
+ is_regulator=("bool", reg))
+ else:
+ assert vinfo[k]["name"][1] == n, (k, n)
+ assert ev.startswith("[") and ev.endswith("]"), ev
+ edges.append((s, t, (funcs.get(func, func), confs[cl.lower()],
+ [y.strip() for y in ev[1:-1].split(",") if y.strip()])))
+ eprops = [('function', 'string'), ('confidence', 'string'),
+ ('evidence', 'vector<string>')]
+ return _graph(edges, vinfo, eprops)
+
+@cache_network()
+@coerce_props(vforce={'name': 'string', 'id': 'string'},
+ eforce={'function': 'string', 'confidence': 'string'})
+@annotate()
+def parse(alts=None):
+ name = __name__.split(".")[-1]
+
+ def split_list(x):
+ # split at the commas outside parentheses
+ return [y.strip() for y in re.split(r",(?![^()]*\))", x) if y.strip()]
+
+ for r, t in old_releases:
+ alt = f'tf_gene_v{r}'
+ if alts is not None and alt not in alts:
+ continue
+ yield alt, _parse_old_tf_gene(name, f'network_tf_gene_v{r}.txt', r)
+
+ for alt, fs in nets:
+ if alts is not None and alt not in alts:
+ continue
+ vinfo = {}
+ edges = []
+
+ names = defaultdict(Counter)
+
+ def add_vertex(key, info):
+ names[key][info['name'][1]] += 1
+ if key not in vinfo:
+ vinfo[key] = info
+ else:
+ for p, v in info.items():
+ old = vinfo[key][p]
+ if old[1] in ("", []) and v[1] not in ("", []):
+ vinfo[key][p] = v
+ return key
+
+ if alt == 'regulator_gene':
+ rows = _read(name, fs[0] + '.tsv', 7)
+ irows = _read(name, fs[1] + '.tsv', 10)
+ assert len(rows) == len(irows)
+ for r, ir in zip(rows, irows):
+ # the names differ in one row (`H-+NS` for `H-NS`); they
+ # are chosen by majority below
+ assert (r[0], r[3], r[4], r[5]) == \
+ (ir[0], ir[3], ir[4], ir[7]), (r, ir)
+ s = add_vertex(r[0], dict(name=("string", r[1]), id=("string", r[0]),
+ is_regulator=("bool", True),
+ regulator_genes=("vector<string>", split_list(r[2])),
+ regulator_type=("string", ir[2]),
+ synonyms=("vector<string>", split_list(ir[6]))))
+ t = add_vertex(r[3], dict(name=("string", r[4]), id=("string", r[3]),
+ is_regulator=("bool", False),
+ regulator_genes=("vector<string>", []),
+ regulator_type=("string", ""),
+ synonyms=("vector<string>", split_list(ir[5]))))
+ edges.append((s, t, (r[5], r[6], ir[8],
+ [y for y in ir[9].split("|") if y])))
+ eprops = [('function', 'string'), ('confidence', 'string'),
+ ('confidence_internal', 'string'),
+ ('evidence_category', 'vector<string>')]
+ elif alt in ('regulator_conformation_gene', 'regulator_tu',
+ 'regulator_regulator'):
+ conf = alt == 'regulator_conformation_gene'
+ rows = _read(name, fs[0] + '.tsv', 8 if conf else 7)
+ for r in rows:
+ if conf:
+ rid, rname, rgenes, cname, tid, tname, func, cl = r
+ else:
+ rid, rname, rgenes, tid, tname, func, cl = r
+ info = dict(name=("string", rname), id=("string", rid),
+ regulator_genes=("vector<string>", split_list(rgenes)))
+ if alt != 'regulator_regulator':
+ info['is_regulator'] = ("bool", True)
+ s = add_vertex(rid, info)
+ info = dict(name=("string", tname), id=("string", tid),
+ regulator_genes=("vector<string>", []))
+ if alt != 'regulator_regulator':
+ info['is_regulator'] = ("bool", False)
+ t = add_vertex(tid, info)
+ edges.append((s, t, (func, cl, cname) if conf else (func, cl)))
+ eprops = [('function', 'string'), ('confidence', 'string')]
+ if conf:
+ eprops.append(('conformation', 'string'))
+ else:
+ rows = _read(name, fs[0] + '.tsv', 5)
+ for sname, tname, func, ev, cl in rows:
+ s = add_vertex(("sigma", sname), dict(name=("string", sname),
+ id=("string", ""),
+ is_sigma=("bool", True)))
+ t = add_vertex(("target", tname), dict(name=("string", tname),
+ id=("string", ""),
+ is_sigma=("bool", False)))
+ assert ev.startswith("[") and ev.endswith("]"), ev
+ edges.append((s, t, (func, cl, split_list(ev[1:-1]))))
+ eprops = [('function', 'string'), ('confidence', 'string'),
+ ('promoter_evidence', 'vector<string>')]
+
+ for key, c in names.items():
+ vinfo[key]['name'] = ("string", c.most_common(1)[0][0])
+ g = _graph(edges, vinfo, eprops)
+ yield alt, g