logo Netzschleuder network catalog, repository, and centrifuge

History of nih_open_citations

The changes to the source code of this entry in the repository, newest first, each with the differences it made to the entry's files. They are also listed on GitLab.

[entries] Add the date when each entry was added, and remove icon_hash by Tiago de Paula Peixoto beff7c9
The dates are those of the commits that first added the entries' files
(2020-08-11 for the entries of the initial commit); renamed entries keep
the date of their first name.

Changed: none
diff --git a/entries/nih_open_citations/__init__.py b/entries/nih_open_citations/__init__.py
index df1534b..f992cbf 100644
--- a/entries/nih_open_citations/__init__.py
+++ b/entries/nih_open_citations/__init__.py
@@ -50,7 +50,7 @@ url = ['https://icite.od.nih.gov/',
 citation = [('B. I. Hutchins, K. L. Baker, M. T. Davis, M. A. Diwersy, E. Haque, R. M. Harriman, T. A. Hoppe, S. A. Leicht, P. Meyer, G. M. Santangelo, "The NIH Open Citation Collection: A public access, broad coverage resource." PLOS Biology 17(10), e3000385 (2019)', 'https://doi.org/10.1371/journal.pbio.3000385'),
             ('B. I. Hutchins, X. Yuan, J. M. Anderson, G. M. Santangelo, "Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level." PLOS Biology 14(9), e1002541 (2016)', 'https://doi.org/10.1371/journal.pbio.1002541'),
             ('iCite, B. I. Hutchins, G. Santangelo, "iCite Database Snapshots (NIH Open Citation Collection)." figshare (2019-2026)', 'https://doi.org/10.35092/yhjc.c.4586573')]
-icon_hash = None
+date_added = '2026-10-27'
 upstream_license = ('CC0 1.0 (snapshots until 2025-09); CC BY 4.0 (snapshot 2026-09)',
                     'https://doi.org/10.35092/yhjc34177140.v1')
 
[entries] Prefix the properties in the descriptions with (Node), (Edge) or (Graph) by Tiago de Paula Peixoto bc3dbbd
The lists of properties at the end of the descriptions marked edge
properties with "Edge" but left node properties unmarked, or grouped
them in "Nodes: `a`, ...; `b`, ..." items, so the kind of a property
was often unclear. Every item now starts with (Node), (Edge) or (Graph),
followed by the property names, a colon and the explanation, one item
per property or group of properties; qualifications (kinds of nodes,
networks) follow the names in parentheses. Properties that were only
named got a short explanation, and a few items that named the wrong
kind of property were corrected.
diff --git a/entries/nih_open_citations/__init__.py b/entries/nih_open_citations/__init__.py
index 46b81d9..df1534b 100644
--- a/entries/nih_open_citations/__init__.py
+++ b/entries/nih_open_citations/__init__.py
@@ -32,18 +32,18 @@ The nodes have the metadata of the latest snapshot (`2026-09`), also in the netw
 
 The networks have the following properties:
 
-- `pmid`: the PubMed identifier of the article.
-- `doi`: the DOI of the article (empty if unknown).
-- `title`: the title of the article.
-- `year`: the publication year (-1 if unknown).
-- `journal`: the journal, as the ISO abbreviation of its name.
-- `is_research_article`: whether the publication types of the article are those of a primary research article.
-- `relative_citation_ratio`: the Relative Citation Ratio (RCR), the field- and time-normalized citation rate of the article, benchmarked to NIH-funded articles (NaN if not computed).
-- `nih_percentile`: the percentile rank of the RCR among NIH-funded articles (NaN if not computed).
-- `field_citation_rate`: the citation rate of the field of the article, estimated from its co-citation network (NaN if not computed).
-- `human`, `animal`, `molecular_cellular`: the fractions of the MeSH terms of the article in the categories human, animal and molecular/cellular biology (NaN if unknown).
-- `apt`: the Approximate Potential to Translate, the predicted probability that the article will be cited by a clinical article (NaN if unknown).
-- `is_clinical`: whether the article is a clinical article."""
+- (Node) `pmid`: the PubMed identifier of the article.
+- (Node) `doi`: the DOI of the article (empty if unknown).
+- (Node) `title`: the title of the article.
+- (Node) `year`: the publication year (-1 if unknown).
+- (Node) `journal`: the journal, as the ISO abbreviation of its name.
+- (Node) `is_research_article`: whether the publication types of the article are those of a primary research article.
+- (Node) `relative_citation_ratio`: the Relative Citation Ratio (RCR), the field- and time-normalized citation rate of the article, benchmarked to NIH-funded articles (NaN if not computed).
+- (Node) `nih_percentile`: the percentile rank of the RCR among NIH-funded articles (NaN if not computed).
+- (Node) `field_citation_rate`: the citation rate of the field of the article, estimated from its co-citation network (NaN if not computed).
+- (Node) `human`, `animal`, `molecular_cellular`: the fractions of the MeSH terms of the article in the categories human, animal and molecular/cellular biology (NaN if unknown).
+- (Node) `apt`: the Approximate Potential to Translate, the predicted probability that the article will be cited by a clinical article (NaN if unknown).
+- (Node) `is_clinical`: whether the article is a clinical article."""
 tags = ['Informational', 'Citation', 'Unweighted', 'Metadata']
 url = ['https://icite.od.nih.gov/',
        'https://nih.figshare.com/collections/iCite_Database_Snapshots_NIH_Open_Citation_Collection_/4586573']
[entries] nih_open_citations: do not catch the errors of the caller when opening the citation tables by Tiago de Paula Peixoto 036da59
The NotImplementedError of zipfile (for the Deflate64 archive) was caught
around the yield of open_citations(), so that the same error raised in
the caller's block would have made it yield again.

Changed: none
diff --git a/entries/nih_open_citations/__init__.py b/entries/nih_open_citations/__init__.py
index 10e7ea6..46b81d9 100644
--- a/entries/nih_open_citations/__init__.py
+++ b/entries/nih_open_citations/__init__.py
@@ -89,20 +89,23 @@ def open_citations(name, fname):
     """Open the citation table `fname` ("<archive>:zip:<member>"). The
     archive of the initial snapshot is compressed with Deflate64, which
     zipfile cannot read: it is then read with the 7z command line tool."""
-    try:
-        with open_upstream_file(name, fname, "rb") as f:
+    # (only the opening is guarded, not the caller's block)
+    with ExitStack() as stack:
+        try:
+            f = stack.enter_context(open_upstream_file(name, fname, "rb"))
+        except NotImplementedError:
+            f = None
+        if f is not None:
             yield f
-        return
-    except NotImplementedError:
-        pass
-    archive, kind, member, named = split_archive_member(fname)
-    with subprocess.Popen(["7z", "e", "-so", f"cache/upstream/{name}/{archive}",
-                           member], stdout=subprocess.PIPE,
-                          stderr=subprocess.DEVNULL) as p:
-        yield p.stdout
-        p.stdout.close()
-    if p.returncode not in (0, None, -13):    # -13: closed early (SIGPIPE)
-        raise RuntimeError(f"7z failed with code {p.returncode} for {fname}")
+    if f is None:
+        archive, kind, member, named = split_archive_member(fname)
+        with subprocess.Popen(["7z", "e", "-so", f"cache/upstream/{name}/{archive}",
+                               member], stdout=subprocess.PIPE,
+                              stderr=subprocess.DEVNULL) as p:
+            yield p.stdout
+            p.stdout.close()
+        if p.returncode not in (0, None, -13):    # -13: closed early (SIGPIPE)
+            raise RuntimeError(f"7z failed with code {p.returncode} for {fname}")
 
 def read_citations(name, fname):
     """The citations "<citing>,<referenced>" (PMIDs), in blocks of (citing,
@@ -154,7 +157,6 @@ def read_metadata(name, fname):
 @coerce_props()
 @annotate()
 def parse(alts=None):
-    global files
     name = __name__.split(".")[-1]
     for (cfile, mfile), alt, fmt in files:
         if alts is not None and alt not in alts:
[entries] Store the networks of more than 300 million edges of the new large entries only in the gt format by Tiago de Paula Peixoto 68260af
The networks of the large datasets added recently with more than about
300 million edges (published counts, or estimates where the networks
have not been parsed yet) are listed in `skip_conversion`, so that they
are not converted to GraphML, GML and CSV, which would take about 5 TB.
diff --git a/entries/nih_open_citations/__init__.py b/entries/nih_open_citations/__init__.py
index 9ba704e..10e7ea6 100644
--- a/entries/nih_open_citations/__init__.py
+++ b/entries/nih_open_citations/__init__.py
@@ -62,6 +62,10 @@ huge = True
 # The figshare API (the server cannot download from ndownloader.figshare.com):
 # the file ids of open_citation_collection.zip of each snapshot (one per
 # year), and of icite_metadata.zip of the latest one
+# networks of more than about 300 million edges: only stored and served in the
+# gt format
+skip_conversion = ['2019-10', '2020-09', '2021-09', '2022-09', '2023-09',
+                   '2024-09', '2025-09', '2026-09']
 upstream_prefix = 'https://api.figshare.com/v2/file/download'
 _snapshots = {"2019-10": 17508128, "2020-09": 25191392, "2021-09": 31006669,
               "2022-09": 37773108, "2023-09": 42570418, "2024-09": 49594377,
[entries] New entry: nih_open_citations by Tiago de Paula Peixoto 7d89907
diff --git a/entries/nih_open_citations/__init__.py b/entries/nih_open_citations/__init__.py
new file mode 100644
index 0000000..9ba704e
--- /dev/null
+++ b/entries/nih_open_citations/__init__.py
@@ -0,0 +1,207 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2026 Tiago de Paula Peixoto <tiago@skewed.de>
+#
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option) any
+# later version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import pyarrow
+import pyarrow.csv
+import pyarrow.compute as pc
+
+from .. import *
+
+
+title = "NIH Open Citation Collection of PubMed articles (iCite, 2019-2026)"
+description = """Citation networks of the articles indexed in PubMed, from the NIH Open Citation Collection, a public-domain collection of citations between PubMed articles compiled by the Office of Portfolio Analysis of the US National Institutes of Health (NIH), and distributed with its bibliometric web service iCite. The citations were gathered from MEDLINE, PubMed Central, Crossref and other open sources, and the references were matched to PubMed identifiers (PMIDs), partly with machine learning methods. Each node is a PubMed article, and a directed edge from article A to article B means that A cites B. Only the articles that take part in some citation are included.
+
+iCite publishes a snapshot of its database every month (84 snapshots from October 2019 to September 2026, each containing all the citations collected until then); the networks here are one snapshot per year: the initial snapshot of October 2019 (`2019-10`), and the September snapshots of 2020 to 2026 (`2020-09`, ..., `2026-09`).
+
+The nodes have the metadata of the latest snapshot (`2026-09`), also in the networks of earlier snapshots (articles that are absent from it have empty or missing values). The metrics of iCite are described in the documentation of iCite. The author lists, the coordinates of the articles in the "triangle of biomedicine" and some derived counts of the metadata are left out.
+
+The networks have the following properties:
+
+- `pmid`: the PubMed identifier of the article.
+- `doi`: the DOI of the article (empty if unknown).
+- `title`: the title of the article.
+- `year`: the publication year (-1 if unknown).
+- `journal`: the journal, as the ISO abbreviation of its name.
+- `is_research_article`: whether the publication types of the article are those of a primary research article.
+- `relative_citation_ratio`: the Relative Citation Ratio (RCR), the field- and time-normalized citation rate of the article, benchmarked to NIH-funded articles (NaN if not computed).
+- `nih_percentile`: the percentile rank of the RCR among NIH-funded articles (NaN if not computed).
+- `field_citation_rate`: the citation rate of the field of the article, estimated from its co-citation network (NaN if not computed).
+- `human`, `animal`, `molecular_cellular`: the fractions of the MeSH terms of the article in the categories human, animal and molecular/cellular biology (NaN if unknown).
+- `apt`: the Approximate Potential to Translate, the predicted probability that the article will be cited by a clinical article (NaN if unknown).
+- `is_clinical`: whether the article is a clinical article."""
+tags = ['Informational', 'Citation', 'Unweighted', 'Metadata']
+url = ['https://icite.od.nih.gov/',
+       'https://nih.figshare.com/collections/iCite_Database_Snapshots_NIH_Open_Citation_Collection_/4586573']
+citation = [('B. I. Hutchins, K. L. Baker, M. T. Davis, M. A. Diwersy, E. Haque, R. M. Harriman, T. A. Hoppe, S. A. Leicht, P. Meyer, G. M. Santangelo, "The NIH Open Citation Collection: A public access, broad coverage resource." PLOS Biology 17(10), e3000385 (2019)', 'https://doi.org/10.1371/journal.pbio.3000385'),
+            ('B. I. Hutchins, X. Yuan, J. M. Anderson, G. M. Santangelo, "Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level." PLOS Biology 14(9), e1002541 (2016)', 'https://doi.org/10.1371/journal.pbio.1002541'),
+            ('iCite, B. I. Hutchins, G. Santangelo, "iCite Database Snapshots (NIH Open Citation Collection)." figshare (2019-2026)', 'https://doi.org/10.35092/yhjc.c.4586573')]
+icon_hash = None
+upstream_license = ('CC0 1.0 (snapshots until 2025-09); CC BY 4.0 (snapshot 2026-09)',
+                    'https://doi.org/10.35092/yhjc34177140.v1')
+
+# About 0.4 to 0.92 billion edges per network (estimated from the sizes of
+# the citation tables): parsing the largest one (2026-09) needs an estimated
+# 50-60 GB of memory
+huge = True
+
+# The figshare API (the server cannot download from ndownloader.figshare.com):
+# the file ids of open_citation_collection.zip of each snapshot (one per
+# year), and of icite_metadata.zip of the latest one
+upstream_prefix = 'https://api.figshare.com/v2/file/download'
+_snapshots = {"2019-10": 17508128, "2020-09": 25191392, "2021-09": 31006669,
+              "2022-09": 37773108, "2023-09": 42570418, "2024-09": 49594377,
+              "2025-09": 58460521, "2026-09": 69733404}
+META_FILE = '69732534:zip:icite_metadata.csv'
+files = [((f'{fid}:zip:open_citation_collection.csv', META_FILE), alt, None)
+         for alt, fid in _snapshots.items()]
+
+def fetch_upstream(force=False):
+    # check=False: the archives are not opened to check them, since zipfile
+    # cannot read the one compressed with Deflate64 (see open_citations())
+    return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
+                                force, check=False)
+
+# For checks only: if set, only (about) this many citations are read, and
+# only this many lines of the metadata
+_check_rows = None
+
+@contextmanager
+def open_citations(name, fname):
+    """Open the citation table `fname` ("<archive>:zip:<member>"). The
+    archive of the initial snapshot is compressed with Deflate64, which
+    zipfile cannot read: it is then read with the 7z command line tool."""
+    try:
+        with open_upstream_file(name, fname, "rb") as f:
+            yield f
+        return
+    except NotImplementedError:
+        pass
+    archive, kind, member, named = split_archive_member(fname)
+    with subprocess.Popen(["7z", "e", "-so", f"cache/upstream/{name}/{archive}",
+                           member], stdout=subprocess.PIPE,
+                          stderr=subprocess.DEVNULL) as p:
+        yield p.stdout
+        p.stdout.close()
+    if p.returncode not in (0, None, -13):    # -13: closed early (SIGPIPE)
+        raise RuntimeError(f"7z failed with code {p.returncode} for {fname}")
+
+def read_citations(name, fname):
+    """The citations "<citing>,<referenced>" (PMIDs), in blocks of (citing,
+    referenced) arrays."""
+    n = 0
+    with open_citations(name, fname) as f:
+        reader = pyarrow.csv.open_csv(
+            f,
+            read_options=pyarrow.csv.ReadOptions(block_size=1 << 26),
+            convert_options=pyarrow.csv.ConvertOptions(
+                column_types={"citing": pyarrow.int64(),
+                              "referenced": pyarrow.int64()}))
+        for batch in reader:
+            if _check_rows is not None and n >= _check_rows:
+                break
+            n += batch.num_rows
+            yield [batch.column(k).to_numpy() for k in ["citing", "referenced"]]
+
+STR_PROPS = ["doi", "title", "journal"]
+FLOAT_PROPS = ["relative_citation_ratio", "nih_percentile",
+               "field_citation_rate", "human", "animal",
+               "molecular_cellular", "apt"]
+BOOL_PROPS = ["is_research_article", "is_clinical"]
+
+def read_metadata(name, fname):
+    """The metadata, in record batches with the columns used."""
+    cols = ["pmid", "year"] + STR_PROPS + FLOAT_PROPS + BOOL_PROPS
+    types = dict({"pmid": pyarrow.int64(), "year": pyarrow.int16()},
+                 **{k: pyarrow.string() for k in STR_PROPS},
+                 **{k: pyarrow.float64() for k in FLOAT_PROPS},
+                 **{k: pyarrow.bool_() for k in BOOL_PROPS})
+    n = 0
+    with open_upstream_file(name, fname, "rb") as f:
+        reader = pyarrow.csv.open_csv(
+            f,
+            read_options=pyarrow.csv.ReadOptions(block_size=1 << 26),
+            parse_options=pyarrow.csv.ParseOptions(newlines_in_values=True),
+            convert_options=pyarrow.csv.ConvertOptions(
+                include_columns=cols, column_types=types,
+                true_values=["True"], false_values=["False"],
+                strings_can_be_null=False))
+        for batch in reader:
+            if _check_rows is not None and n >= _check_rows:
+                break
+            n += batch.num_rows
+            yield batch
+
+@cache_network()
+@coerce_props()
+@annotate()
+def parse(alts=None):
+    global files
+    name = __name__.split(".")[-1]
+    for (cfile, mfile), alt, fmt in files:
+        if alts is not None and alt not in alts:
+            continue
+
+        # The articles that take part in some citation (a first pass), in
+        # the order of their PMIDs
+        used = numpy.zeros(0, dtype="bool")
+        for citing, cited in read_citations(name, cfile):
+            m = max(citing.max(), cited.max()) + 1
+            if m > len(used):
+                used = numpy.concatenate((used, numpy.zeros(m - len(used),
+                                                            dtype="bool")))
+            used[citing] = True
+            used[cited] = True
+        pmids = numpy.flatnonzero(used)
+        index = numpy.cumsum(used) - 1
+        del used
+
+        g = Graph(directed=True)
+        g.add_vertex(len(pmids))
+        g.vp.pmid = g.new_vp("int64_t", vals=pmids)
+        for citing, cited in read_citations(name, cfile):
+            g.add_edge_list(numpy.column_stack((index[citing], index[cited])))
+        del index
+
+        # The metadata of the latest snapshot
+        for k in STR_PROPS:
+            g.vp[k] = g.new_vp("string")
+        g.vp.year = g.new_vp("int16_t", val=-1)
+        for k in FLOAT_PROPS:
+            g.vp[k] = g.new_vp("double", val=numpy.nan)
+        for k in BOOL_PROPS:
+            g.vp[k] = g.new_vp("bool")
+        for batch in read_metadata(name, mfile):
+            v = lookup(pmids, numpy.arange(len(pmids)),
+                       batch.column("pmid").to_numpy())
+            m = v >= 0
+            if not m.any():
+                continue
+            mask = pyarrow.array(m)
+            v = v[m]
+            g.vp.year.a[v] = pc.fill_null(batch.column("year").filter(mask), -1).to_numpy()
+            for k in FLOAT_PROPS:
+                g.vp[k].a[v] = pc.fill_null(batch.column(k).filter(mask),
+                                            numpy.nan).to_numpy()
+            for k in BOOL_PROPS:
+                g.vp[k].a[v] = pc.fill_null(batch.column(k).filter(mask),
+                                            False).to_numpy(zero_copy_only=False)
+            for k in STR_PROPS:
+                p = g.vp[k]
+                for u, s in zip(v.tolist(), batch.column(k).filter(mask).to_pylist()):
+                    p[u] = s
+        yield alt, g