History of human_disease_comorbidity
The changes to the source code of this entry in the repository, newest first, each with the differences it made to the entry's files. They are also listed on GitLab.
[entries] human_disease_comorbidity: leave out the incomplete last line of truncated files 11c7cd4
The archived copies of WhiteNet5.net and BlackNet5.net, the only ones left, are truncated, ending with an incomplete line that failed the parse; the description says which disease pairs these two networks lack.
diff --git a/entries/human_disease_comorbidity/__init__.py b/entries/human_disease_comorbidity/__init__.py
index b2086d4..b950332 100644
--- a/entries/human_disease_comorbidity/__init__.py
+++ b/entries/human_disease_comorbidity/__init__.py
@@ -34,6 +34,8 @@ The networks have the following properties:
- `phi` (edges): the $\\phi$-correlation between the two diseases.
- `t` (edges): the $t$-test value of the $\\phi$-correlation.
+The files are obtained from the Internet Archive, the only remaining copies. Two of them are truncated there, so that their networks lack the disease pairs whose first code is larger than the last one included: `white_5` ends after the pairs of code 585, and `black_5` after those of code 890.1 (the incomplete last line of each file is left out).
+
The data are free for academic use only (according to the source, for non-academic purposes C. A. Hidalgo should be contacted)."""
tags = ['Biological', 'Public Health', 'Weighted', 'Metadata']
url = 'https://web.archive.org/web/20101116170623/http://barabasilab.neu.edu/projects/hudine/resource/data/data.html'
@@ -79,6 +81,9 @@ def parse(alts=None):
# phi-correlation, t-test value
with open_upstream_file(name, fname, "rb") as f:
df = pandas.read_csv(f, sep="\t", header=None, dtype={0: str, 1: str})
+ # The archived copies of two files (WhiteNet5, BlackNet5) are
+ # truncated, ending with an incomplete line
+ df = df.dropna()
g = Graph(directed=False)
eprops = [("cooccurrence", "int"), ("relative_risk", "double"),
("relative_risk_low", "double"),
[entries] New entry: human_disease_comorbidity b09eb63
diff --git a/entries/human_disease_comorbidity/__init__.py b/entries/human_disease_comorbidity/__init__.py
new file mode 100644
index 0000000..b2086d4
--- /dev/null
+++ b/entries/human_disease_comorbidity/__init__.py
@@ -0,0 +1,96 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2020-2026 Tiago de Paula Peixoto <tiago@skewed.de>
+#
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option) any
+# later version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along
+# with this program. If not, see <http://www.gnu.org/licenses/>.
+
+from .. import *
+import pandas
+
+title = "Human disease comorbidity networks, Medicare (1990-1993)"
+description = """Networks of comorbidity between diseases, from the Human Disease Network (HuDiNe) project of Hidalgo, Blumm, Barabási and Christakis. The data were compiled from the Medicare inpatient claims (MedPAR records) of 1990 to 1993 of the elderly Americans (aged 65 or older) enrolled in Medicare and alive during the whole period: 32,341,347 hospital claims of 13,039,018 patients. Each claim lists a primary and up to nine secondary diagnoses, coded according to the ICD-9-CM classification. Nodes are diseases (ICD-9 codes), and an undirected edge connects two diseases that were diagnosed together in at least one patient, with properties giving the number of such patients and the strength of the comorbidity, measured by the relative risk and by the $\\phi$-correlation (Pearson's correlation for binary variables).
+
+Diseases are defined at two levels of resolution of the ICD-9 codes: 3 digits (the main disease categories; networks ending in `_3`), and up to 5 digits (the full codes, which add details to the category; networks ending in `_5`). For each resolution there are networks for all patients (`all`), and for patients of each sex (`female`, `male`), race (`white`, `black`), and their combinations (e.g. `black_female`). Of the 13,039,018 patients, 7,598,529 are female and 5,440,490 male, and 11,745,416 are white and 983,095 black.
+
+The networks have the following properties:
+
+- `name`: the ICD-9-CM code of the disease (e.g. "250" or "250.00").
+- `prevalence`: the number of patients diagnosed with the disease in the population of the network.
+- `cooccurrence` (edges): the number of patients diagnosed with both diseases.
+- `relative_risk` (edges): the relative risk $RR_{ij} = C_{ij} N / (P_i P_j)$, where $C_{ij}$ is the co-occurrence, $P_i$ and $P_j$ the prevalences and $N$ the number of patients, i.e. the ratio between the observed co-occurrence and the one expected if the diseases were independent.
+- `relative_risk_low`, `relative_risk_high` (edges): the lower and upper limits of the 99% confidence interval of the relative risk.
+- `phi` (edges): the $\\phi$-correlation between the two diseases.
+- `t` (edges): the $t$-test value of the $\\phi$-correlation.
+
+The data are free for academic use only (according to the source, for non-academic purposes C. A. Hidalgo should be contacted)."""
+tags = ['Biological', 'Public Health', 'Weighted', 'Metadata']
+url = 'https://web.archive.org/web/20101116170623/http://barabasilab.neu.edu/projects/hudine/resource/data/data.html'
+citation = [('C. A. Hidalgo, N. Blumm, A.-L. Barabási, N. A. Christakis, "A dynamic network approach for the study of human phenotypes." PLoS Computational Biology 5(4), e1000353 (2009)', 'https://doi.org/10.1371/journal.pcbi.1000353')]
+icon_hash = '56a85c8e26855e083a2f78af'
+upstream_license = ('Free for academic use only', 'https://web.archive.org/web/20101116170623/http://barabasilab.neu.edu/projects/hudine/resource/data/data.html')
+# The HuDiNe site is offline; the files are obtained from the Internet
+# Archive.
+upstream_prefix = 'https://web.archive.org/web/2010id_/http://barabasilab.neu.edu/projects/hudine/resource/datasets'
+groups = {'all': 'All', 'female': 'Female', 'male': 'Male', 'white': 'White',
+ 'black': 'Black', 'white_female': 'WhiteFemale',
+ 'white_male': 'WhiteMale', 'black_female': 'BlackFemale',
+ 'black_male': 'BlackMale'}
+files = [(f'{fn}Net{d}.net', f'{alt}_{d}', None)
+ for d in (3, 5) for alt, fn in groups.items()]
+timestamps = {'AllNet3': 20101116162015, 'AllNet5': 20101116162031,
+ 'BlackFemaleNet3': 20101116162002, 'BlackFemaleNet5': 20101116165812,
+ 'BlackMaleNet3': 20101116162635, 'BlackMaleNet5': 20101116165919,
+ 'BlackNet3': 20101116162612, 'BlackNet5': 20101116165055,
+ 'FemaleNet3': 20101116162505, 'FemaleNet5': 20101116162723,
+ 'MaleNet3': 20101116162524, 'MaleNet5': 20101116163538,
+ 'WhiteFemaleNet3': 20101116161947, 'WhiteFemaleNet5': 20101116165302,
+ 'WhiteMaleNet3': 20101116162653, 'WhiteMaleNet5': 20101116170044,
+ 'WhiteNet3': 20101116162545, 'WhiteNet5': 20101116164500}
+archived_urls = {f'{k}.net': f'https://web.archive.org/web/{ts}id_/http://barabasilab.neu.edu/projects/hudine/resource/datasets/{k}.net'
+ for k, ts in timestamps.items()}
+
+def fetch_upstream(force=False):
+ return fetch_archived_files(__name__.split(".")[-1], archived_urls, force,
+ tries=4)
+
+@cache_network()
+@coerce_props(vforce={'name': 'string'})
+@annotate()
+def parse(alts=None):
+ global files
+ name = __name__.split(".")[-1]
+ for fname, alt, fmt in files:
+ if alts is not None and alt not in alts:
+ continue
+ # Columns: code 1, code 2, prevalence 1, prevalence 2, co-occurrence,
+ # relative risk, its 99% confidence interval (left, right),
+ # phi-correlation, t-test value
+ with open_upstream_file(name, fname, "rb") as f:
+ df = pandas.read_csv(f, sep="\t", header=None, dtype={0: str, 1: str})
+ g = Graph(directed=False)
+ eprops = [("cooccurrence", "int"), ("relative_risk", "double"),
+ ("relative_risk_low", "double"),
+ ("relative_risk_high", "double"), ("phi", "double"),
+ ("t", "double")]
+ for p, t in eprops:
+ g.ep[p] = g.new_ep(t)
+ g.vp.name = g.add_edge_list(df[[0, 1, 4, 5, 6, 7, 8, 9]].itertuples(index=False),
+ hashed=True, hash_type="string",
+ eprops=[g.ep[p] for p, t in eprops])
+ prev = dict(zip(df[0], df[2]))
+ prev.update(zip(df[1], df[3]))
+ g.vp.prevalence = g.new_vp("int", vals=[prev[g.vp.name[v]]
+ for v in g.vertices()])
+ yield alt, g