logo Netzschleuder network catalog, repository, and centrifuge

History of microsynteny_networks

The changes to the source code of this entry in the repository, newest first, each with the differences it made to the entry's files. They are also listed on GitLab.

[entries] Remove superfluous details of the upstream distribution from the descriptions by Tiago de Paula Peixoto dc07aa0
The descriptions mentioned incidental details of how the upstream data
are made available: the file formats (CSV, JSON, XML, GEXF, SBML, HIF,
Pajek, UCINET, ...), file names, where the files were published or
obtained from (Zenodo, figshare, Kaggle, the Pajek datasets page, the
Internet Archive), and that some networks are only available in the gt
format. These are left out; caveats about the data themselves are kept,
referring to "the data" instead of "the file". The description of
sp_baboons now lists its properties instead of the upstream columns,
and says that events without a recipient are left out.
diff --git a/entries/microsynteny_networks/__init__.py b/entries/microsynteny_networks/__init__.py
index 53fa378..b68b3bc 100644
--- a/entries/microsynteny_networks/__init__.py
+++ b/entries/microsynteny_networks/__init__.py
@@ -28,7 +28,7 @@ import pyarrow.csv as pacsv
 from .. import *
 
 title = "Microsynteny networks of mammal and plant genomes (2019, 2021)"
-description = r"""Networks of microsynteny (conserved gene order) between the genes of many genomes, built by T. Zhao, M. E. Schranz and collaborators to study the evolution of genome structure in mammals and flowering plants. The protein-coding genes of the pairs of genomes were compared (with the best hits of each gene in the other genomes), and collinear blocks of homologous genes (syntenic blocks) were detected with MCScanX (for the networks of 2019, with the 5 best hits per gene, at least 5 anchor gene pairs per block, and at most 15 (mammals) or 25 (plants) genes between anchors: the parameters `b5`, `s5`, `m15` and `m25` in the names of the files); the anchor pairs of all the blocks form the network.
+description = r"""Networks of microsynteny (conserved gene order) between the genes of many genomes, built by T. Zhao, M. E. Schranz and collaborators to study the evolution of genome structure in mammals and flowering plants. The protein-coding genes of the pairs of genomes were compared (with the best hits of each gene in the other genomes), and collinear blocks of homologous genes (syntenic blocks) were detected with MCScanX (for the networks of 2019, with the 5 best hits per gene, at least 5 anchor gene pairs per block, and at most 15 (mammals) or 25 (plants) genes between anchors); the anchor pairs of all the blocks form the network.
 
 Nodes are genes, and an edge connects two genes that are anchors of the same syntenic block (one edge per block, so that a pair of genes in several blocks is connected by several edges).
 
[entries] Add the date when each entry was added, and remove icon_hash by Tiago de Paula Peixoto beff7c9
The dates are those of the commits that first added the entries' files
(2020-08-11 for the entries of the initial commit); renamed entries keep
the date of their first name.

Changed: none
diff --git a/entries/microsynteny_networks/__init__.py b/entries/microsynteny_networks/__init__.py
index 6708cf4..53fa378 100644
--- a/entries/microsynteny_networks/__init__.py
+++ b/entries/microsynteny_networks/__init__.py
@@ -54,7 +54,7 @@ citation = [('T. Zhao, M. E. Schranz, "Network-based microsynteny analysis ident
              'https://doi.org/10.1073/pnas.1801757116'),
             ('T. Zhao, A. Zwaenepoel, J.-Y. Xue, S.-M. Kao, Z. Li, M. E. Schranz, Y. Van de Peer, "Whole-genome microsynteny-based phylogeny of angiosperms." Nature Communications 12, 3498 (2021)',
              'https://doi.org/10.1038/s41467-021-23665-0')]
-icon_hash = None
+date_added = '2026-10-31'
 upstream_license = ('Creative Commons CC0 1.0 Universal (CC0 1.0)',
                     'https://creativecommons.org/publicdomain/zero/1.0/')
 # Harvard Dataverse file ids (doi:10.7910/DVN/BDMA7A version 7, and
[entries] Prefix the properties in the descriptions with (Node), (Edge) or (Graph) by Tiago de Paula Peixoto bc3dbbd
The lists of properties at the end of the descriptions marked edge
properties with "Edge" but left node properties unmarked, or grouped
them in "Nodes: `a`, ...; `b`, ..." items, so the kind of a property
was often unclear. Every item now starts with (Node), (Edge) or (Graph),
followed by the property names, a colon and the explanation, one item
per property or group of properties; qualifications (kinds of nodes,
networks) follow the names in parentheses. Properties that were only
named got a short explanation, and a few items that named the wrong
kind of property were corrected.
diff --git a/entries/microsynteny_networks/__init__.py b/entries/microsynteny_networks/__init__.py
index 4ae2cb8..6708cf4 100644
--- a/entries/microsynteny_networks/__init__.py
+++ b/entries/microsynteny_networks/__init__.py
@@ -40,14 +40,14 @@ Nodes are genes, and an edge connects two genes that are anchors of the same syn
 
 The networks have the following properties:
 
-- `name`: the identifier of the gene, prefixed by the code of its genome (e.g. `Hsap...` for mammals, `ath_...` for plants).
-- `genome`: the code of the genome of the gene.
-- `species`: the species of the genome (from the tables of the genomes of the studies; empty if the code is not in the tables).
-- `cluster`: the cluster of the gene in the clustering of the network with Infomap by the authors (`-1` if the gene is not in the clustering).
-- `chromosome` (`plants_123`): the chromosome (or scaffold) of the gene.
-- Edge `block`, `anchor` (except the BUSCO networks): the syntenic block of the anchor pair, identified in the data as `<genome 1>_<genome 2><n>-<k>b`, where `n` (`block`) numbers the blocks of the pair of genomes (those of the two genes of the edge) and `k` (`anchor`) the anchor pairs of the block.
-- Edge `block_score` (except the BUSCO networks): the score of the block computed by MCScanX.
-- Edge `evalue`, `block_evalue`, `block_size` (`plants_123`): the E-value of the anchor pair, and the E-value and the number of anchor pairs of the block."""
+- (Node) `name`: the identifier of the gene, prefixed by the code of its genome (e.g. `Hsap...` for mammals, `ath_...` for plants).
+- (Node) `genome`: the code of the genome of the gene.
+- (Node) `species`: the species of the genome (from the tables of the genomes of the studies; empty if the code is not in the tables).
+- (Node) `cluster`: the cluster of the gene in the clustering of the network with Infomap by the authors (`-1` if the gene is not in the clustering).
+- (Node) `chromosome` (`plants_123`): the chromosome (or scaffold) of the gene.
+- (Edge) `block`, `anchor` (except the BUSCO networks): the syntenic block of the anchor pair, identified in the data as `<genome 1>_<genome 2><n>-<k>b`, where `n` (`block`) numbers the blocks of the pair of genomes (those of the two genes of the edge) and `k` (`anchor`) the anchor pairs of the block.
+- (Edge) `block_score` (except the BUSCO networks): the score of the block computed by MCScanX.
+- (Edge) `evalue`, `block_evalue`, `block_size` (`plants_123`): the E-value of the anchor pair, and the E-value and the number of anchor pairs of the block."""
 tags = ['Biological', 'Genetic', 'Unweighted', 'Multigraph', 'Metadata']
 url = ['https://doi.org/10.7910/DVN/BDMA7A', 'https://doi.org/10.7910/DVN/7ZZWIH']
 citation = [('T. Zhao, M. E. Schranz, "Network-based microsynteny analysis identifies major differences and genomic outliers in mammalian and angiosperm genomes." Proceedings of the National Academy of Sciences 116, 2165-2174 (2019)',
[entries] New entry: microsynteny_networks by Tiago de Paula Peixoto 04a9c59
diff --git a/entries/microsynteny_networks/__init__.py b/entries/microsynteny_networks/__init__.py
new file mode 100644
index 0000000..4ae2cb8
--- /dev/null
+++ b/entries/microsynteny_networks/__init__.py
@@ -0,0 +1,206 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2026 Tiago de Paula Peixoto <tiago@skewed.de>
+#
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option) any
+# later version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import csv
+import gzip
+import io
+from contextlib import contextmanager
+
+import pyarrow as pa
+import pyarrow.compute as pc
+import pyarrow.csv as pacsv
+
+from .. import *
+
+title = "Microsynteny networks of mammal and plant genomes (2019, 2021)"
+description = r"""Networks of microsynteny (conserved gene order) between the genes of many genomes, built by T. Zhao, M. E. Schranz and collaborators to study the evolution of genome structure in mammals and flowering plants. The protein-coding genes of the pairs of genomes were compared (with the best hits of each gene in the other genomes), and collinear blocks of homologous genes (syntenic blocks) were detected with MCScanX (for the networks of 2019, with the 5 best hits per gene, at least 5 anchor gene pairs per block, and at most 15 (mammals) or 25 (plants) genes between anchors: the parameters `b5`, `s5`, `m15` and `m25` in the names of the files); the anchor pairs of all the blocks form the network.
+
+Nodes are genes, and an edge connects two genes that are anchors of the same syntenic block (one edge per block, so that a pair of genes in several blocks is connected by several edges).
+
+- `mammals`: 87 mammalian genomes (Zhao and Schranz, 2019).
+- `mammals_busco`: the subnetwork of the BUSCO genes (near-universal single-copy orthologs) of the 87 mammals, built separately with other gene identifiers (NCBI), as given by the authors (two columns, without blocks).
+- `plants_107`: 107 genomes of flowering plants and other plants (Zhao and Schranz, 2019).
+- `plants_107_busco`: the subnetwork of the BUSCO genes of the 107 plants (without blocks).
+- `plants_123`: 123 plant genomes, mostly flowering plants (Zhao et al., 2021).
+
+The networks have the following properties:
+
+- `name`: the identifier of the gene, prefixed by the code of its genome (e.g. `Hsap...` for mammals, `ath_...` for plants).
+- `genome`: the code of the genome of the gene.
+- `species`: the species of the genome (from the tables of the genomes of the studies; empty if the code is not in the tables).
+- `cluster`: the cluster of the gene in the clustering of the network with Infomap by the authors (`-1` if the gene is not in the clustering).
+- `chromosome` (`plants_123`): the chromosome (or scaffold) of the gene.
+- Edge `block`, `anchor` (except the BUSCO networks): the syntenic block of the anchor pair, identified in the data as `<genome 1>_<genome 2><n>-<k>b`, where `n` (`block`) numbers the blocks of the pair of genomes (those of the two genes of the edge) and `k` (`anchor`) the anchor pairs of the block.
+- Edge `block_score` (except the BUSCO networks): the score of the block computed by MCScanX.
+- Edge `evalue`, `block_evalue`, `block_size` (`plants_123`): the E-value of the anchor pair, and the E-value and the number of anchor pairs of the block."""
+tags = ['Biological', 'Genetic', 'Unweighted', 'Multigraph', 'Metadata']
+url = ['https://doi.org/10.7910/DVN/BDMA7A', 'https://doi.org/10.7910/DVN/7ZZWIH']
+citation = [('T. Zhao, M. E. Schranz, "Network-based microsynteny analysis identifies major differences and genomic outliers in mammalian and angiosperm genomes." Proceedings of the National Academy of Sciences 116, 2165-2174 (2019)',
+             'https://doi.org/10.1073/pnas.1801757116'),
+            ('T. Zhao, A. Zwaenepoel, J.-Y. Xue, S.-M. Kao, Z. Li, M. E. Schranz, Y. Van de Peer, "Whole-genome microsynteny-based phylogeny of angiosperms." Nature Communications 12, 3498 (2021)',
+             'https://doi.org/10.1038/s41467-021-23665-0')]
+icon_hash = None
+upstream_license = ('Creative Commons CC0 1.0 Universal (CC0 1.0)',
+                    'https://creativecommons.org/publicdomain/zero/1.0/')
+# Harvard Dataverse file ids (doi:10.7910/DVN/BDMA7A version 7, and
+# doi:10.7910/DVN/7ZZWIH version 3):
+# 3317124 87Mammal_SynNet-b5s5m15.gz, 3317127 its Infomap clusters,
+# 3317136 87Mammal_BUSCO_cleaned-network_b5s5m15.gz, 3317135 its clusters,
+# 3317112 107Plant-SynNet-b5s5m25.gz, 3317114 its clusters,
+# 3317138 107Plant-BUSCO-cleaned-network.gz, 3317137 its clusters,
+# 3317134 Table-S1-Genomes-Used.tab (mammals),
+# 3657697 123Genomes_synteny_network_entire_edgelist.tar.gz,
+# 3657696 its Infomap clusters (tar.gz), 3657415 Table_S1.tab (123 genomes)
+upstream_prefix = 'https://dataverse.harvard.edu/api/access/datafile'
+NET123 = "3657697:tar.gz:125genome_blocks_fulledges_final_filtered"
+CLU123 = "3657696:tar.gz:Fixed-125genome_blocks_fulledges_final_filtered.clu_2cols"
+files = [(("3317124", "3317127", "3317134"), "mammals", "blocks"),
+         (("3317136", "3317135", "3317134"), "mammals_busco", "pairs"),
+         (("3317112", "3317114", "3657415"), "plants_107", "blocks"),
+         (("3317138", "3317137", "3657415"), "plants_107_busco", "pairs"),
+         ((NET123, CLU123, "3657415"), "plants_123", "blocks8")]
+
+def fetch_upstream(force=False):
+    return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
+                                force)
+
+_check_rows = None
+
+def read_table(f, ncols, numeric=()):
+    "Read a tab-separated table without header, as a pyarrow table."
+    reader = pacsv.open_csv(
+        f, read_options=pacsv.ReadOptions(column_names=[f"c{i}" for i in range(ncols)],
+                                          block_size=1 << 26),
+        parse_options=pacsv.ParseOptions(delimiter="\t"),
+        convert_options=pacsv.ConvertOptions(
+            column_types={f"c{i}": (pa.float64() if i in numeric else pa.string())
+                          for i in range(ncols)}))
+    batches = []
+    n = 0
+    for b in reader:
+        batches.append(b)
+        n += b.num_rows
+        if _check_rows is not None and n >= _check_rows:
+            break
+    return pa.Table.from_batches(batches)
+
+@contextmanager
+def open_data(name, fname):
+    "Open an upstream file in binary mode, decompressing it if it is gzipped."
+    with open_upstream_file(name, fname, "rb") as f:
+        f = io.BufferedReader(f) if not hasattr(f, "peek") else f
+        if f.peek(2)[:2] == b"\x1f\x8b":
+            with gzip.open(f) as fz:
+                yield fz
+        else:
+            yield f
+
+def species_table(name, fname):
+    "Genome codes to species names."
+    sp = {}
+    with open_upstream_file(name, fname, "r", encoding="latin1") as f:
+        r = csv.reader(f, delimiter="\t")
+        h = next(r)
+        ia = h.index("Abbreviation")
+        isp = h.index("Species") if "Species" in h else h.index("SpeciesName")
+        for row in r:
+            if len(row) > max(ia, isp):
+                sp[row[ia].strip('"')] = row[isp].strip('"')
+    return sp
+
+def genome_code(g):
+    return g.split("_", 1)[0] if "_" in g[:5] else g[:4]
+
+@cache_network()
+@coerce_props()
+@annotate()
+def parse(alts=None):
+    name = __name__.split(".")[-1]
+    for (fnet, fclu, fsp), alt, fmt in files:
+        if alts is not None and alt not in alts:
+            continue
+        if fmt == "blocks":
+            # block, block score, gene, gene
+            with open_data(name, fnet) as f:
+                t = read_table(f, 4, (1,))
+            a, b = t["c2"], t["c3"]
+        elif fmt == "blocks8":
+            # block, gene, gene, E-value of the pair, block score, E-value of
+            # the block, number of anchors of the block, chromosomes
+            with open_data(name, fnet) as f:
+                t = read_table(f, 8, (3, 4, 5, 6))
+            a, b = t["c1"], t["c2"]
+        else:
+            # (the BUSCO networks are separated by spaces)
+            with open_data(name, fnet) as f:
+                text = f.read().replace(b" ", b"\t")
+            t = read_table(io.BytesIO(text), 2)
+            del text
+            a, b = t["c0"], t["c1"]
+        genes = pc.unique(pa.chunked_array(a.chunks + b.chunks))
+        if isinstance(genes, pa.ChunkedArray):
+            genes = genes.combine_chunks()
+        ia = pc.index_in(a, value_set=genes).to_numpy()
+        ib = pc.index_in(b, value_set=genes).to_numpy()
+        g = Graph(directed=False)
+        g.add_vertex(len(genes))
+        g.add_edge_list(numpy.stack([ia, ib], axis=1).astype("int64"))
+        names = genes.to_pylist()
+        del genes
+        g.vp.name = g.new_vp("string", vals=names)
+        codes = [genome_code(x) for x in names]
+        g.vp.genome = g.new_vp("string", vals=codes)
+        sp = species_table(name, fsp)
+        g.vp.species = g.new_vp("string", vals=[sp.get(c, "") for c in codes])
+        clu = {}
+        with open_data(name, fclu) as fb:
+            for l in io.TextIOWrapper(fb, encoding="utf8"):
+                x = l.split()
+                if len(x) == 2 and x[1].lstrip("-").isdigit():
+                    clu[x[0]] = int(x[1])
+        g.vp.cluster = g.new_vp("int64_t", vals=[clu.get(x, -1) for x in names])
+        del names, codes
+        if fmt in ("blocks", "blocks8"):
+            # the block identifiers <genome 1>_<genome 2><n>-<k>b are stored
+            # as the numbers n and k (the genomes are those of the genes)
+            m = pc.extract_regex(t["c0"], r"(?P<n>[0-9]+)-(?P<k>[0-9]+)b$")
+            assert m.null_count == 0
+            g.ep.block = g.new_ep("int32_t", vals=pc.cast(pc.struct_field(m, "n"), pa.int32()).to_numpy())
+            g.ep.anchor = g.new_ep("int32_t", vals=pc.cast(pc.struct_field(m, "k"), pa.int32()).to_numpy())
+            del m
+        if fmt == "blocks":
+            g.ep.block_score = g.new_ep("double", vals=t["c1"].to_numpy())
+        elif fmt == "blocks8":
+            g.ep.block_score = g.new_ep("double", vals=t["c4"].to_numpy())
+            g.ep.evalue = g.new_ep("double", vals=t["c3"].to_numpy())
+            g.ep.block_evalue = g.new_ep("double", vals=t["c5"].to_numpy())
+            g.ep.block_size = g.new_ep("int32_t", vals=t["c6"].to_numpy())
+            # chromosomes of the two genes (<chromosome 1>&<chromosome 2>)
+            d = pc.dictionary_encode(t["c7"]).combine_chunks()
+            pairs = [x.split("&", 1) for x in d.dictionary.to_pylist()]
+            code = d.indices.to_numpy()
+            chrom = [""] * g.num_vertices()
+            for side, iv in ((0, ia), (1, ib)):
+                vs, first = numpy.unique(iv, return_index=True)
+                for v, e in zip(vs, first):
+                    pr = pairs[code[e]]
+                    chrom[v] = pr[side] if len(pr) == 2 else ""
+            g.vp.chromosome = g.new_vp("string", vals=chrom)
+            del d, pairs, code, chrom
+        del t, ia, ib
+        yield alt, g