History of emp_microbiome
The changes to the source code of this entry in the repository, newest first, each with the differences it made to the entry's files. They are also listed on GitLab.
[entries] Add the date when each entry was added, and remove icon_hash beff7c9
The dates are those of the commits that first added the entries' files (2020-08-11 for the entries of the initial commit); renamed entries keep the date of their first name. Changed: none
diff --git a/entries/emp_microbiome/__init__.py b/entries/emp_microbiome/__init__.py
index 06ea849..5666aa7 100644
--- a/entries/emp_microbiome/__init__.py
+++ b/entries/emp_microbiome/__init__.py
@@ -56,7 +56,7 @@ tags = ['Biological', 'Ecological', 'Weighted', 'Bipartite', 'Metadata']
url = ['https://earthmicrobiome.org/', 'https://doi.org/10.5281/zenodo.890000']
citation = [('L. R. Thompson, J. G. Sanders, D. McDonald, A. Amir, J. Ladau, K. J. Locey, R. J. Prill, A. Tripathi, S. M. Gibbons, G. Ackermann, et al., "A communal catalogue reveals Earth\'s multiscale microbial diversity." Nature 551, 457-463 (2017)',
'https://doi.org/10.1038/nature24621')]
-icon_hash = None
+date_added = '2026-10-31'
# or_gg_13_8 (8 million nodes, 158 million edges, 75 metadata columns) needs
# an estimated 30-40 GB of memory to parse
huge = True
[entries] Prefix the properties in the descriptions with (Node), (Edge) or (Graph) bc3dbbd
The lists of properties at the end of the descriptions marked edge properties with "Edge" but left node properties unmarked, or grouped them in "Nodes: `a`, ...; `b`, ..." items, so the kind of a property was often unclear. Every item now starts with (Node), (Edge) or (Graph), followed by the property names, a colon and the explanation, one item per property or group of properties; qualifications (kinds of nodes, networks) follow the names in parentheses. Properties that were only named got a short explanation, and a few items that named the wrong kind of property were corrected.
diff --git a/entries/emp_microbiome/__init__.py b/entries/emp_microbiome/__init__.py
index 4ca84d5..06ea849 100644
--- a/entries/emp_microbiome/__init__.py
+++ b/entries/emp_microbiome/__init__.py
@@ -34,24 +34,24 @@ Nodes are samples and microbial taxa (sequence variants or OTUs), and an edge co
The networks have the following properties:
-- `is_sample`: whether the node is a sample (otherwise it is a taxon).
-- `name`: the identifier of the sample (in lower case in `or_gg_13_8`, as given in its table), or of the taxon (the sequence itself for the Deblur tables, the OTU identifier otherwise).
-- `taxonomy`: the taxonomy of the taxon (from kingdom to species, separated by `; `; empty for samples).
-- The sample metadata, all the columns of the mapping file of the release (empty, NaN or false for taxa; strings unless said otherwise):
- - `barcode_sequence`, `linker_primer_sequence`, `description`, `host_subject_id`: the barcode and primer sequences, the description of the sample and the identifier of its host individual, as given by the study.
- - `study_id`, `title`, `principal_investigator`, `doi`, `ebi_accession`: the EMP study of the sample, its title, principal investigator, the DOI of its publication, and its accession in the European Nucleotide Archive.
- - `target_gene`, `target_subfragment`, `pcr_primers`, `illumina_technology`, `extraction_center`, `run_center`, `run_date`, `read_length_bp` (number): the sequencing of the sample.
- - `sequences_split_libraries`, `observations_closed_ref_greengenes`, `observations_closed_ref_silva`, `observations_open_ref_greengenes`, `observations_deblur_90bp`, `observations_deblur_100bp`, `observations_deblur_150bp` (numbers): the numbers of sequences of the sample after demultiplexing, and in each observation table.
- - `emp_release1`, `qc_filtered`, `subset_10k`, `subset_5k`, `subset_2k` (booleans): whether the sample is in release 1, whether it passed the quality filters of the EMP, and whether it belongs to the even subsets of 10,000, 5,000 and 2,000 samples used in the paper.
- - `sample_taxid`, `sample_scientific_name`: the NCBI taxon and name of the kind of sample (e.g. `soil metagenome`).
- - `host_taxid`, `host_common_name_provided`, `host_common_name`, `host_scientific_name`, `host_superkingdom`, `host_kingdom`, `host_phylum`, `host_class`, `host_order`, `host_family`, `host_genus`, `host_species`: the host of the sample and its taxonomy.
- - `collection_timestamp`: the date of collection of the sample (as given).
- - `country`, `latitude_deg`, `longitude_deg`, `depth_m`, `altitude_m`, `elevation_m` (numbers, except the country): the location of the sample (NaN if unknown).
- - `env_biome`, `env_feature`, `env_material`, `envo_biome_0` ... `envo_biome_5`: the environment of the sample (ENVO terms), and the levels of the hierarchy of its biome.
- - `empo_0`, `empo_1`, `empo_2`, `empo_3`: the habitat of the sample in the levels of the EMP ontology (e.g. `EMP sample` / `Free-living` / `Saline` / `Water (saline)`).
- - `adiv_observed_otus`, `adiv_chao1`, `adiv_shannon`, `adiv_faith_pd` (numbers): the alpha diversity of the sample (observed OTUs, Chao1, Shannon, Faith's phylogenetic diversity), as computed by the EMP.
- - `temperature_deg_c`, `ph`, `salinity_psu`, `oxygen_mg_per_l`, `phosphate_umol_per_l`, `ammonium_umol_per_l`, `nitrate_umol_per_l`, `sulfate_umol_per_l` (numbers): physicochemical measurements of the sample (NaN if not given).
-- Edge `count`: the number of reads of the taxon in the sample."""
+- (Node) `is_sample`: whether the node is a sample (otherwise it is a taxon).
+- (Node) `name`: the identifier of the sample (in lower case in `or_gg_13_8`, as given in its table), or of the taxon (the sequence itself for the Deblur tables, the OTU identifier otherwise).
+- (Node) `taxonomy`: the taxonomy of the taxon (from kingdom to species, separated by `; `; empty for samples).
+- (Node) The sample metadata, all the columns of the mapping file of the release (empty, NaN or false for taxa; strings unless said otherwise):
+ - (Node) `barcode_sequence`, `linker_primer_sequence`, `description`, `host_subject_id`: the barcode and primer sequences, the description of the sample and the identifier of its host individual, as given by the study.
+ - (Node) `study_id`, `title`, `principal_investigator`, `doi`, `ebi_accession`: the EMP study of the sample, its title, principal investigator, the DOI of its publication, and its accession in the European Nucleotide Archive.
+ - (Node) `target_gene`, `target_subfragment`, `pcr_primers`, `illumina_technology`, `extraction_center`, `run_center`, `run_date`, `read_length_bp` (number): the sequencing of the sample.
+ - (Node) `sequences_split_libraries`, `observations_closed_ref_greengenes`, `observations_closed_ref_silva`, `observations_open_ref_greengenes`, `observations_deblur_90bp`, `observations_deblur_100bp`, `observations_deblur_150bp` (numbers): the numbers of sequences of the sample after demultiplexing, and in each observation table.
+ - (Node) `emp_release1`, `qc_filtered`, `subset_10k`, `subset_5k`, `subset_2k` (booleans): whether the sample is in release 1, whether it passed the quality filters of the EMP, and whether it belongs to the even subsets of 10,000, 5,000 and 2,000 samples used in the paper.
+ - (Node) `sample_taxid`, `sample_scientific_name`: the NCBI taxon and name of the kind of sample (e.g. `soil metagenome`).
+ - (Node) `host_taxid`, `host_common_name_provided`, `host_common_name`, `host_scientific_name`, `host_superkingdom`, `host_kingdom`, `host_phylum`, `host_class`, `host_order`, `host_family`, `host_genus`, `host_species`: the host of the sample and its taxonomy.
+ - (Node) `collection_timestamp`: the date of collection of the sample (as given).
+ - (Node) `country`, `latitude_deg`, `longitude_deg`, `depth_m`, `altitude_m`, `elevation_m` (numbers, except the country): the location of the sample (NaN if unknown).
+ - (Node) `env_biome`, `env_feature`, `env_material`, `envo_biome_0` ... `envo_biome_5`: the environment of the sample (ENVO terms), and the levels of the hierarchy of its biome.
+ - (Node) `empo_0`, `empo_1`, `empo_2`, `empo_3`: the habitat of the sample in the levels of the EMP ontology (e.g. `EMP sample` / `Free-living` / `Saline` / `Water (saline)`).
+ - (Node) `adiv_observed_otus`, `adiv_chao1`, `adiv_shannon`, `adiv_faith_pd` (numbers): the alpha diversity of the sample (observed OTUs, Chao1, Shannon, Faith's phylogenetic diversity), as computed by the EMP.
+ - (Node) `temperature_deg_c`, `ph`, `salinity_psu`, `oxygen_mg_per_l`, `phosphate_umol_per_l`, `ammonium_umol_per_l`, `nitrate_umol_per_l`, `sulfate_umol_per_l` (numbers): physicochemical measurements of the sample (NaN if not given).
+- (Edge) `count`: the number of reads of the taxon in the sample."""
tags = ['Biological', 'Ecological', 'Weighted', 'Bipartite', 'Metadata']
url = ['https://earthmicrobiome.org/', 'https://doi.org/10.5281/zenodo.890000']
citation = [('L. R. Thompson, J. G. Sanders, D. McDonald, A. Amir, J. Ladau, K. J. Locey, R. J. Prill, A. Tripathi, S. M. Gibbons, G. Ackermann, et al., "A communal catalogue reveals Earth\'s multiscale microbial diversity." Nature 551, 457-463 (2017)',
[entries] New entry: emp_microbiome b5f6220
diff --git a/entries/emp_microbiome/__init__.py b/entries/emp_microbiome/__init__.py
new file mode 100644
index 0000000..4ca84d5
--- /dev/null
+++ b/entries/emp_microbiome/__init__.py
@@ -0,0 +1,170 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2026 Tiago de Paula Peixoto <tiago@skewed.de>
+#
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option) any
+# later version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along
+# with this program. If not, see <http://www.gnu.org/licenses/>.
+
+import csv
+import math
+
+import h5py
+
+from .. import *
+
+title = "Earth Microbiome Project sample-microbe networks (2017)"
+description = r"""Bipartite networks of environmental samples and the microbes (bacteria and archaea) found in them, from release 1 of the Earth Microbiome Project (EMP), a collaborative effort that collected and sequenced (16S rRNA gene amplicons, V4 region) about 27,000 samples from 97 studies of all kinds of environments: soils, sediments, fresh and salt water, plants, animal bodies and guts, built environments, etc., with a common protocol, and classified them with the EMP ontology of habitats (EMPO).
+
+Nodes are samples and microbial taxa (sequence variants or OTUs), and an edge connects a sample to each taxon observed in it, with the number of reads of the taxon in the sample as weight. There is one network for each of the published observation tables of the release, which differ in how the reads were grouped into taxa:
+
+- `deblur_90bp`, `deblur_100bp`, `deblur_150bp`: exact sequence variants (sub-OTUs) obtained with Deblur from the reads trimmed to 90, 100 and 150 base pairs (the 150 bp table contains fewer samples, those with longer reads).
+- `cr_gg_13_8`, `cr_silva_123`: OTUs of closed-reference picking (97% identity) against the Greengenes 13.8 and SILVA 123 references.
+- `or_gg_13_8`: OTUs of open-reference picking against Greengenes 13.8 (without the OTUs found in a single sample).
+
+The networks have the following properties:
+
+- `is_sample`: whether the node is a sample (otherwise it is a taxon).
+- `name`: the identifier of the sample (in lower case in `or_gg_13_8`, as given in its table), or of the taxon (the sequence itself for the Deblur tables, the OTU identifier otherwise).
+- `taxonomy`: the taxonomy of the taxon (from kingdom to species, separated by `; `; empty for samples).
+- The sample metadata, all the columns of the mapping file of the release (empty, NaN or false for taxa; strings unless said otherwise):
+ - `barcode_sequence`, `linker_primer_sequence`, `description`, `host_subject_id`: the barcode and primer sequences, the description of the sample and the identifier of its host individual, as given by the study.
+ - `study_id`, `title`, `principal_investigator`, `doi`, `ebi_accession`: the EMP study of the sample, its title, principal investigator, the DOI of its publication, and its accession in the European Nucleotide Archive.
+ - `target_gene`, `target_subfragment`, `pcr_primers`, `illumina_technology`, `extraction_center`, `run_center`, `run_date`, `read_length_bp` (number): the sequencing of the sample.
+ - `sequences_split_libraries`, `observations_closed_ref_greengenes`, `observations_closed_ref_silva`, `observations_open_ref_greengenes`, `observations_deblur_90bp`, `observations_deblur_100bp`, `observations_deblur_150bp` (numbers): the numbers of sequences of the sample after demultiplexing, and in each observation table.
+ - `emp_release1`, `qc_filtered`, `subset_10k`, `subset_5k`, `subset_2k` (booleans): whether the sample is in release 1, whether it passed the quality filters of the EMP, and whether it belongs to the even subsets of 10,000, 5,000 and 2,000 samples used in the paper.
+ - `sample_taxid`, `sample_scientific_name`: the NCBI taxon and name of the kind of sample (e.g. `soil metagenome`).
+ - `host_taxid`, `host_common_name_provided`, `host_common_name`, `host_scientific_name`, `host_superkingdom`, `host_kingdom`, `host_phylum`, `host_class`, `host_order`, `host_family`, `host_genus`, `host_species`: the host of the sample and its taxonomy.
+ - `collection_timestamp`: the date of collection of the sample (as given).
+ - `country`, `latitude_deg`, `longitude_deg`, `depth_m`, `altitude_m`, `elevation_m` (numbers, except the country): the location of the sample (NaN if unknown).
+ - `env_biome`, `env_feature`, `env_material`, `envo_biome_0` ... `envo_biome_5`: the environment of the sample (ENVO terms), and the levels of the hierarchy of its biome.
+ - `empo_0`, `empo_1`, `empo_2`, `empo_3`: the habitat of the sample in the levels of the EMP ontology (e.g. `EMP sample` / `Free-living` / `Saline` / `Water (saline)`).
+ - `adiv_observed_otus`, `adiv_chao1`, `adiv_shannon`, `adiv_faith_pd` (numbers): the alpha diversity of the sample (observed OTUs, Chao1, Shannon, Faith's phylogenetic diversity), as computed by the EMP.
+ - `temperature_deg_c`, `ph`, `salinity_psu`, `oxygen_mg_per_l`, `phosphate_umol_per_l`, `ammonium_umol_per_l`, `nitrate_umol_per_l`, `sulfate_umol_per_l` (numbers): physicochemical measurements of the sample (NaN if not given).
+- Edge `count`: the number of reads of the taxon in the sample."""
+tags = ['Biological', 'Ecological', 'Weighted', 'Bipartite', 'Metadata']
+url = ['https://earthmicrobiome.org/', 'https://doi.org/10.5281/zenodo.890000']
+citation = [('L. R. Thompson, J. G. Sanders, D. McDonald, A. Amir, J. Ladau, K. J. Locey, R. J. Prill, A. Tripathi, S. M. Gibbons, G. Ackermann, et al., "A communal catalogue reveals Earth\'s multiscale microbial diversity." Nature 551, 457-463 (2017)',
+ 'https://doi.org/10.1038/nature24621')]
+icon_hash = None
+# or_gg_13_8 (8 million nodes, 158 million edges, 75 metadata columns) needs
+# an estimated 30-40 GB of memory to parse
+huge = True
+upstream_license = ('Creative Commons Attribution 4.0 International (CC BY 4.0)',
+ 'https://creativecommons.org/licenses/by/4.0/')
+upstream_prefix = 'https://zenodo.org/records/890000/files'
+MAPPING = "emp_qiime_mapping_release1_20170912.tsv"
+files = [(("emp_deblur_90bp.release1.biom", MAPPING), "deblur_90bp", None),
+ (("emp_deblur_100bp.release1.biom", MAPPING), "deblur_100bp", None),
+ (("emp_deblur_150bp.release1.biom", MAPPING), "deblur_150bp", None),
+ (("emp_cr_gg_13_8.release1.biom", MAPPING), "cr_gg_13_8", None),
+ (("emp_cr_silva_16S_123.release1.biom", MAPPING), "cr_silva_123", None),
+ (("emp_or_gg_13_8.no_sample_singletons.biom", MAPPING), "or_gg_13_8", None)]
+
+def fetch_upstream(force=False):
+ return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
+ force)
+
+# check only: number of taxa (rows of the table) to read
+_check_rows = None
+
+# all the columns of the mapping file (besides the sample identifier), as
+# numbers, booleans or strings (the others, including identifiers such as
+# study_id and the taxonomy identifiers)
+NUMBERS = ["read_length_bp", "sequences_split_libraries",
+ "observations_closed_ref_greengenes", "observations_closed_ref_silva",
+ "observations_open_ref_greengenes", "observations_deblur_90bp",
+ "observations_deblur_100bp", "observations_deblur_150bp",
+ "latitude_deg", "longitude_deg", "depth_m", "altitude_m",
+ "elevation_m", "adiv_observed_otus", "adiv_chao1", "adiv_shannon",
+ "adiv_faith_pd", "temperature_deg_c", "ph", "salinity_psu",
+ "oxygen_mg_per_l", "phosphate_umol_per_l", "ammonium_umol_per_l",
+ "nitrate_umol_per_l", "sulfate_umol_per_l"]
+BOOLS = ["emp_release1", "qc_filtered", "subset_10k", "subset_5k", "subset_2k"]
+# property names of the columns in camel case
+RENAME = {"BarcodeSequence": "barcode_sequence",
+ "LinkerPrimerSequence": "linker_primer_sequence",
+ "Description": "description"}
+
+def to_float(x):
+ try:
+ return float(x)
+ except ValueError:
+ return math.nan
+
+def decode(x):
+ return x.decode() if isinstance(x, bytes) else str(x)
+
+@cache_network()
+@coerce_props()
+@annotate()
+def parse(alts=None):
+ name = __name__.split(".")[-1]
+ mapping = None
+ for (fbiom, fmap), alt, fmt in files:
+ if alts is not None and alt not in alts:
+ continue
+ if mapping is None:
+ with open_upstream_file(name, fmap, "r") as f:
+ reader = csv.DictReader(f, delimiter="\t")
+ mapping = {r["#SampleID"]: r for r in reader}
+ columns = [c for c in reader.fieldnames if c != "#SampleID"]
+ assert set(NUMBERS) | set(BOOLS) <= set(columns)
+ # the sample identifiers of the open-reference table are in
+ # lower case (they are unique in lower case)
+ lower = {k.lower(): r for k, r in mapping.items()}
+ assert len(lower) == len(mapping)
+ STRINGS = [c for c in columns if c not in NUMBERS and c not in BOOLS]
+ with open_upstream_file(name, fbiom, "rb", seekable=True) as fb:
+ hf = h5py.File(fb, "r")
+ sids = [decode(x) for x in hf["sample/ids"][:]]
+ nobs = len(hf["observation/ids"])
+ if _check_rows is not None:
+ nobs = min(nobs, _check_rows)
+ obs = [decode(x) for x in hf["observation/ids"][:nobs]]
+ if "observation/metadata/taxonomy" in hf:
+ tax = ["; ".join(decode(y) for y in x)
+ for x in hf["observation/metadata/taxonomy"][:nobs]]
+ else:
+ tax = [""] * nobs
+ indptr = hf["observation/matrix/indptr"][:nobs + 1].astype("int64")
+ nnz = int(indptr[-1])
+ indices = hf["observation/matrix/indices"][:nnz]
+ data = hf["observation/matrix/data"][:nnz]
+ ns = len(sids)
+ g = Graph(directed=False)
+ g.add_vertex(ns + nobs)
+ rows = numpy.repeat(numpy.arange(ns, ns + nobs, dtype="int64"),
+ numpy.diff(indptr))
+ g.add_edge_list(numpy.stack([indices.astype("int64"), rows], axis=1))
+ del rows, indices, indptr
+ g.ep.count = g.new_ep("double", vals=data)
+ del data
+ g.vp.is_sample = g.new_vp("bool", vals=numpy.arange(ns + nobs) < ns)
+ g.vp.name = g.new_vp("string", vals=sids + obs)
+ del obs
+ g.vp.taxonomy = g.new_vp("string", vals=[""] * ns + tax)
+ del tax
+ ms = [mapping.get(s) or lower.get(s.lower(), {}) for s in sids]
+ for p in STRINGS:
+ q = RENAME.get(p, p)
+ g.vp[q] = g.new_vp("string")
+ for i, m in enumerate(ms):
+ g.vp[q][i] = m.get(p, "")
+ for p in NUMBERS:
+ g.vp[p] = g.new_vp("double", vals=[to_float(m.get(p, "nan")) for m in ms] +
+ [math.nan] * nobs)
+ for p in BOOLS:
+ g.vp[p] = g.new_vp("bool")
+ g.vp[p].a[:ns] = [m.get(p) == "True" for m in ms]
+ yield alt, g