History of physician_shared_patients
The changes to the source code of this entry in the repository, newest first, each with the differences it made to the entry's files. They are also listed on GitLab.
[entries] Add the date when each entry was added, and remove icon_hash beff7c9
The dates are those of the commits that first added the entries' files (2020-08-11 for the entries of the initial commit); renamed entries keep the date of their first name. Changed: none
diff --git a/entries/physician_shared_patients/__init__.py b/entries/physician_shared_patients/__init__.py
index 48af259..5b3a68c 100644
--- a/entries/physician_shared_patients/__init__.py
+++ b/entries/physician_shared_patients/__init__.py
@@ -57,7 +57,7 @@ citation = [('Centers for Medicare & Medicaid Services, "Physician Shared Patien
'https://www.nber.org/research/data/national-plan-and-provider-enumeration-system-nppesnpi'),
('National Uniform Claim Committee, "Health Care Provider Taxonomy Code Set." Version 26.1 (2026)',
'https://www.nucc.org/index.php/code-sets-mainmenu-41/provider-taxonomy-mainmenu-40/csv-mainmenu-57')]
-icon_hash = None
+date_added = '2026-10-28'
upstream_license = ("Public domain (US government work)", "https://www.usa.gov/government-copyright")
upstream_prefix = 'https://downloads.cms.gov/foia'
[entries] Prefix the properties in the descriptions with (Node), (Edge) or (Graph) bc3dbbd
The lists of properties at the end of the descriptions marked edge properties with "Edge" but left node properties unmarked, or grouped them in "Nodes: `a`, ...; `b`, ..." items, so the kind of a property was often unclear. Every item now starts with (Node), (Edge) or (Graph), followed by the property names, a colon and the explanation, one item per property or group of properties; qualifications (kinds of nodes, networks) follow the names in parentheses. Properties that were only named got a short explanation, and a few items that named the wrong kind of property were corrected.
diff --git a/entries/physician_shared_patients/__init__.py b/entries/physician_shared_patients/__init__.py
index 3943490..48af259 100644
--- a/entries/physician_shared_patients/__init__.py
+++ b/entries/physician_shared_patients/__init__.py
@@ -33,19 +33,19 @@ The metadata of the providers (name, type, specialty, location) are taken from t
The networks have the following properties:
-- `npi`: the National Provider Identifier of the provider, as given (a ten-digit number; a few identifiers are invalid, e.g. `D038101571`).
-- `in_nppes`: whether the NPI is in the NPPES file of October 2026.
-- `entity_type`: `individual` or `organization` (empty if not in the NPPES or deactivated).
-- `name`: the name of the provider (first, middle and last name for individuals, the legal business name for organizations).
-- `credential` (individuals): the credentials, as given (e.g. `M.D.`).
-- `taxonomy_code`: the primary Health Care Provider Taxonomy code (e.g. `207R00000X`; the first one if none is marked as primary).
-- `specialty`: its description (classification and specialization, e.g. `Internal Medicine`, `Internal Medicine, Cardiovascular Disease`).
-- `city`, `state`, `zip5`: the location of the practice (city, two-letter state code, and the first five digits of the ZIP code; the state may also be a foreign one, with the country in `country`).
-- `country`: the country code of the practice location (`US` for the United States).
-- Edge `pairs`: the number of times a claim of the target followed a claim of the source within the window (the "pair count").
-- Edge `beneficiaries`: the number of distinct beneficiaries shared by the two providers (at least 11).
-- Edge `same_day`: the number of these sharing events in which both claims were on the same day.
-- `start` (graph), `end` (graph): the date range of the claims considered, as given by CMS (e.g. `2009-01-01` and `2010-02-01` for `2009_days30`)."""
+- (Node) `npi`: the National Provider Identifier of the provider, as given (a ten-digit number; a few identifiers are invalid, e.g. `D038101571`).
+- (Node) `in_nppes`: whether the NPI is in the NPPES file of October 2026.
+- (Node) `entity_type`: `individual` or `organization` (empty if not in the NPPES or deactivated).
+- (Node) `name`: the name of the provider (first, middle and last name for individuals, the legal business name for organizations).
+- (Node) `credential` (individuals): the credentials, as given (e.g. `M.D.`).
+- (Node) `taxonomy_code`: the primary Health Care Provider Taxonomy code (e.g. `207R00000X`; the first one if none is marked as primary).
+- (Node) `specialty`: its description (classification and specialization, e.g. `Internal Medicine`, `Internal Medicine, Cardiovascular Disease`).
+- (Node) `city`, `state`, `zip5`: the location of the practice (city, two-letter state code, and the first five digits of the ZIP code; the state may also be a foreign one, with the country in `country`).
+- (Node) `country`: the country code of the practice location (`US` for the United States).
+- (Edge) `pairs`: the number of times a claim of the target followed a claim of the source within the window (the "pair count").
+- (Edge) `beneficiaries`: the number of distinct beneficiaries shared by the two providers (at least 11).
+- (Edge) `same_day`: the number of these sharing events in which both claims were on the same day.
+- (Node) `start` (graph), `end` (graph): the date range of the claims considered, as given by CMS (e.g. `2009-01-01` and `2010-02-01` for `2009_days30`)."""
tags = ['Social', 'Public Health', 'Offline', 'Weighted', 'Metadata']
url = ['https://www.cms.gov/Regulations-and-Guidance/Legislation/FOIA/Referral-Data-FAQs',
'https://downloads.cms.gov/foia/physician_shared_patient_patterns_technical_requirements.pdf',
[entries] physician_shared_patients: keep the invalid NPIs, as strings 8d0270e
Some lines have identifiers that are not numbers (e.g. "D038101571" in the file of 2011 with a 30-day window), which failed. The NPIs are now stored as strings, as given, also keeping the leading zeros of invalid ones.
diff --git a/entries/physician_shared_patients/__init__.py b/entries/physician_shared_patients/__init__.py
index 0b5cec2..3943490 100644
--- a/entries/physician_shared_patients/__init__.py
+++ b/entries/physician_shared_patients/__init__.py
@@ -33,7 +33,7 @@ The metadata of the providers (name, type, specialty, location) are taken from t
The networks have the following properties:
-- `npi`: the National Provider Identifier of the provider (a ten-digit number; invalid identifiers with leading zeros are given without them).
+- `npi`: the National Provider Identifier of the provider, as given (a ten-digit number; a few identifiers are invalid, e.g. `D038101571`).
- `in_nppes`: whether the NPI is in the NPPES file of October 2026.
- `entity_type`: `individual` or `organization` (empty if not in the NPPES or deactivated).
- `name`: the name of the provider (first, middle and last name for individuals, the legal business name for organizations).
@@ -163,9 +163,25 @@ _nppes_cols = ["entity_type", "name", "credential", "taxonomy_code",
# For checks only: if set, only (about) this many lines are read
_check_rows = None
-def read_pairs(name, fname):
+def read_pairs(name, fname, other):
"""The lines "NPI1,NPI2,pair count,beneficiary count,same day count"
- (fixed-width numbers padded with spaces), as blocks of arrays."""
+ (fixed-width numbers padded with spaces), as blocks of arrays. The NPIs
+ are given as numbers; the few that are not numbers without leading zeros
+ (e.g. "D038101571") are given negative codes, and the dict `other` maps
+ their codes to the identifiers as given."""
+ codes = {}
+ def npi_numbers(c):
+ valid = pc.match_substring_regex(c, "^[1-9][0-9]*$")
+ if pc.all(valid).as_py():
+ return pc.cast(c, pyarrow.int64()).to_numpy()
+ x = pc.cast(pc.if_else(valid, c, "0"), pyarrow.int64()).to_numpy().copy()
+ for i in numpy.flatnonzero(~valid.to_numpy(zero_copy_only=False)):
+ npi = c[i].as_py()
+ if npi not in codes:
+ codes[npi] = -1 - len(codes)
+ other[codes[npi]] = npi
+ x[i] = codes[npi]
+ return x
cols = ["npi1", "npi2", "pairs", "benes", "same_day"]
n = 0
with open_upstream_file(name, fname, "rb") as f:
@@ -179,9 +195,9 @@ def read_pairs(name, fname):
if _check_rows is not None and n >= _check_rows:
break
n += batch.num_rows
- yield [pc.cast(pc.utf8_trim_whitespace(c),
- pyarrow.int64() if k < 2 else pyarrow.int32()).to_numpy()
- for k, c in enumerate(batch.columns)]
+ cs = [pc.utf8_trim_whitespace(c) for c in batch.columns]
+ yield ([npi_numbers(c) for c in cs[:2]] +
+ [pc.cast(c, pyarrow.int32()).to_numpy() for c in cs[2:]])
@cache_network()
@coerce_props()
@@ -193,7 +209,8 @@ def parse(alts=None):
if alts is not None and alt not in alts:
continue
fname = fnames[0]
- blocks = list(read_pairs(name, fname))
+ other = {}
+ blocks = list(read_pairs(name, fname, other))
st = numpy.concatenate([b[0] for b in blocks] + [b[1] for b in blocks])
for b in blocks:
del b[:2]
@@ -204,7 +221,8 @@ def parse(alts=None):
m = len(idx) // 2
g = Graph(directed=True)
g.add_vertex(len(npis))
- g.vp.npi = g.new_vp("int64_t", vals=npis)
+ g.vp.npi = g.new_vp("string", vals=[other[x] if x < 0 else str(x)
+ for x in npis.tolist()])
# the metadata of the providers, from the NPPES
meta = (pl.DataFrame({"npi": npis}).with_row_index("v")
.join(read_nppes(name, fnames[1], fnames[2]), on="npi", how="left")
[entries] physician_shared_patients: add the NPPES data of the providers 8ff64de
The providers were identified only by their NPI. They now have their name, type, primary taxonomy and specialty, city, state, ZIP code and country from the NPPES file (the NBER copy of October 2026, since the CMS download site is down, with the NUCC taxonomy codes), as of the download date. The networks are otherwise unchanged. Changed: upstream physician_shared_patients
diff --git a/entries/physician_shared_patients/__init__.py b/entries/physician_shared_patients/__init__.py
index 917dfca..0b5cec2 100644
--- a/entries/physician_shared_patients/__init__.py
+++ b/entries/physician_shared_patients/__init__.py
@@ -27,20 +27,36 @@ description = r"""Networks of health care providers (physicians, other practitio
Each node is a provider, identified by its National Provider Identifier (NPI), and a directed edge from provider A to provider B means that, for some beneficiaries, a claim of B followed a claim of A within a given time window (30, 60, 90 or 180 days) after it. A provider is counted when it appears on a claim in any role of "treatment association" (billing organization, attending, operating, rendering or ordering physician, ...), except as referring physician or as supplier of durable medical equipment. Relationships from claims of the same day are assigned to the lesser of the two NPIs (as source). The identifiers are as given by CMS: a few thousand edges (e.g. 4,410 in `2009_days30`) involve identifiers that are not valid NPIs (with a leading zero), which are kept as they are. Pairs of providers that shared fewer than 11 distinct beneficiaries in the year were left out by CMS, to protect the privacy of the patients, so that the networks are thresholded at this value (the unthresholded data are not public). Since the windows extend beyond the end of the year, the claims considered for a year run until one, two, three or six months into the next one; the data of 2015 only cover claims until September or October 2015 (shorter windows than the others for the 60, 90 and 180 day periods).
-There is one network for each year and time window, named `<year>_days<window>` (e.g. `2014_days30`, `2014_days180`); the networks of longer windows contain those of shorter windows of the same year. No metadata of the providers (names, specialties, locations, which are in the National Plan and Provider Enumeration System, NPPES) are included; the providers can be looked up by their NPI.
+There is one network for each year and time window, named `<year>_days<window>` (e.g. `2014_days30`, `2014_days180`); the networks of longer windows contain those of shorter windows of the same year.
+
+The metadata of the providers (name, type, specialty, location) are taken from the National Plan and Provider Enumeration System (NPPES), the registry of the NPIs, in its monthly file of October 2026 (data until 11 October 2026, from the mirror of the National Bureau of Economic Research): they are therefore those of the download date, not of the years of the networks (providers move, change names and specialties), and they are missing for the identifiers that are not in the NPPES (invalid identifiers), and empty for the deactivated NPIs (whose data the NPPES does not disclose). The specialty is given by the primary taxonomy code of the provider, described by the Health Care Provider Taxonomy code set of the National Uniform Claim Committee (NUCC, version 26.1).
The networks have the following properties:
- `npi`: the National Provider Identifier of the provider (a ten-digit number; invalid identifiers with leading zeros are given without them).
+- `in_nppes`: whether the NPI is in the NPPES file of October 2026.
+- `entity_type`: `individual` or `organization` (empty if not in the NPPES or deactivated).
+- `name`: the name of the provider (first, middle and last name for individuals, the legal business name for organizations).
+- `credential` (individuals): the credentials, as given (e.g. `M.D.`).
+- `taxonomy_code`: the primary Health Care Provider Taxonomy code (e.g. `207R00000X`; the first one if none is marked as primary).
+- `specialty`: its description (classification and specialization, e.g. `Internal Medicine`, `Internal Medicine, Cardiovascular Disease`).
+- `city`, `state`, `zip5`: the location of the practice (city, two-letter state code, and the first five digits of the ZIP code; the state may also be a foreign one, with the country in `country`).
+- `country`: the country code of the practice location (`US` for the United States).
- Edge `pairs`: the number of times a claim of the target followed a claim of the source within the window (the "pair count").
- Edge `beneficiaries`: the number of distinct beneficiaries shared by the two providers (at least 11).
- Edge `same_day`: the number of these sharing events in which both claims were on the same day.
- `start` (graph), `end` (graph): the date range of the claims considered, as given by CMS (e.g. `2009-01-01` and `2010-02-01` for `2009_days30`)."""
-tags = ['Social', 'Public Health', 'Offline', 'Weighted']
+tags = ['Social', 'Public Health', 'Offline', 'Weighted', 'Metadata']
url = ['https://www.cms.gov/Regulations-and-Guidance/Legislation/FOIA/Referral-Data-FAQs',
- 'https://downloads.cms.gov/foia/physician_shared_patient_patterns_technical_requirements.pdf']
+ 'https://downloads.cms.gov/foia/physician_shared_patient_patterns_technical_requirements.pdf',
+ 'https://www.nber.org/research/data/national-plan-and-provider-enumeration-system-nppesnpi',
+ 'https://www.nucc.org/index.php/code-sets-mainmenu-41/provider-taxonomy-mainmenu-40/csv-mainmenu-57']
citation = [('Centers for Medicare & Medicaid Services, "Physician Shared Patient Patterns." Technical requirements (2015)',
- 'https://downloads.cms.gov/foia/physician_shared_patient_patterns_technical_requirements.pdf')]
+ 'https://downloads.cms.gov/foia/physician_shared_patient_patterns_technical_requirements.pdf'),
+ ('Centers for Medicare & Medicaid Services, "National Plan and Provider Enumeration System (NPPES) Downloadable File." October 2026, mirrored by the National Bureau of Economic Research',
+ 'https://www.nber.org/research/data/national-plan-and-provider-enumeration-system-nppesnpi'),
+ ('National Uniform Claim Committee, "Health Care Provider Taxonomy Code Set." Version 26.1 (2026)',
+ 'https://www.nucc.org/index.php/code-sets-mainmenu-41/provider-taxonomy-mainmenu-40/csv-mainmenu-57')]
icon_hash = None
upstream_license = ("Public domain (US government work)", "https://www.usa.gov/government-copyright")
@@ -59,13 +75,90 @@ def _range(year, window):
return "2015-01-01", _ranges_2015[window]
return f"{year}-01-01", f"{year + 1}-{_ranges[window][1]}-01"
-files = [(f"physician-shared-patient-patterns-{y}-days{w}.zip:"
- f"physician-shared-patient-patterns-{y}-days{w}.txt", f"{y}_days{w}", None)
+# The NPPES file of October 2026 (as mirrored by the NBER, which keeps the
+# monthly files; the download site of CMS only offers the current one), and
+# the NUCC taxonomy code set
+nppes_prefix = 'https://data.nber.org/npi/zip/2026'
+nppes_file = "npi_raw_202610.zip:npidata_pfile_20050523-20261011.csv"
+nucc_prefix = 'https://www.nucc.org/images/stories/CSV'
+nucc_file = "nucc_taxonomy_261.csv"
+
+files = [((f"physician-shared-patient-patterns-{y}-days{w}.zip:"
+ f"physician-shared-patient-patterns-{y}-days{w}.txt", nppes_file, nucc_file),
+ f"{y}_days{w}", None)
for y in years for w in windows]
def fetch_upstream(force=False):
- return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
- force)
+ name = __name__.split(".")[-1]
+ fetch_upstream_files(name, upstream_prefix,
+ [(f[0], alt, fmt) for f, alt, fmt in files], force)
+ fetch_upstream_files(name, nppes_prefix, [(nppes_file, None, None)], force)
+ fetch_upstream_files(name, nucc_prefix, [(nucc_file, None, None)], force)
+
+_nppes = None
+
+def read_nppes(name, fname, nucc):
+ """The NPPES data of the providers (a polars DataFrame by NPI), read once
+ for all networks."""
+ global _nppes
+ if _nppes is not None:
+ return _nppes
+ import polars as pl
+ with open_upstream_file(name, nucc, "rb") as f:
+ tax = pl.read_csv(f.read(), infer_schema=False, encoding="utf8-lossy")
+ tax = tax.select(pl.col("Code").str.strip_chars().alias("taxonomy_code"),
+ pl.when(pl.col("Specialization").fill_null("").str.strip_chars() != "")
+ .then(pl.col("Classification").str.strip_chars() + ", " +
+ pl.col("Specialization").str.strip_chars())
+ .otherwise(pl.col("Classification").str.strip_chars())
+ .alias("specialty"))
+ P = "Healthcare Provider Primary Taxonomy Switch_"
+ T = "Healthcare Provider Taxonomy Code_"
+ loc = "Provider Business Practice Location Address "
+ cols = {"NPI": "npi", "Entity Type Code": "etype",
+ "Provider Organization Name (Legal Business Name)": "org",
+ "Provider Last Name (Legal Name)": "last",
+ "Provider First Name": "first", "Provider Middle Name": "middle",
+ "Provider Credential Text": "credential",
+ loc + "City Name": "city", loc + "State Name": "state",
+ loc + "Postal Code": "zip",
+ loc + "Country Code (If outside U.S.)": "country"}
+ cols |= {f"{T}{i}": f"t{i}" for i in range(1, 16)}
+ cols |= {f"{P}{i}": f"p{i}" for i in range(1, 16)}
+ parts = []
+ n = 0
+ with open_upstream_file(name, fname, "rb") as f:
+ reader = pyarrow.csv.open_csv(
+ f, read_options=pyarrow.csv.ReadOptions(block_size=1 << 26),
+ convert_options=pyarrow.csv.ConvertOptions(
+ include_columns=list(cols),
+ column_types={c: pyarrow.string() for c in cols}))
+ for batch in reader:
+ df = pl.from_arrow(pyarrow.Table.from_batches([batch])).rename(cols)
+ n += len(df)
+ prim = pl.coalesce([pl.when(pl.col(f"p{i}") == "Y").then(pl.col(f"t{i}"))
+ for i in range(1, 16)] + [pl.col("t1")])
+ s = lambda c: pl.col(c).fill_null("").str.strip_chars()
+ name_ = pl.when(s("etype") == "2").then(s("org")).otherwise(
+ pl.concat_str([s("first"), s("middle"), s("last")], separator=" ")
+ .str.replace_all(r"\s+", " ").str.strip_chars())
+ parts.append(df.select(
+ pl.col("npi").cast(pl.Int64),
+ pl.col("etype").replace_strict({"1": "individual", "2": "organization"},
+ default="").alias("entity_type"),
+ name_.alias("name"), s("credential").alias("credential"),
+ prim.fill_null("").str.strip_chars().alias("taxonomy_code"),
+ s("city").alias("city"), s("state").alias("state"),
+ s("zip").str.slice(0, 5).alias("zip5"),
+ s("country").alias("country")))
+ if _check_rows is not None and n >= _check_rows:
+ break
+ _nppes = (pl.concat(parts).join(tax, on="taxonomy_code", how="left")
+ .with_columns(pl.col("specialty").fill_null("")))
+ return _nppes
+
+_nppes_cols = ["entity_type", "name", "credential", "taxonomy_code",
+ "specialty", "city", "state", "zip5", "country"]
# For checks only: if set, only (about) this many lines are read
_check_rows = None
@@ -94,10 +187,12 @@ def read_pairs(name, fname):
@coerce_props()
@annotate()
def parse(alts=None):
+ import polars as pl
name = __name__.split(".")[-1]
- for fname, alt, fmt in files:
+ for fnames, alt, fmt in files:
if alts is not None and alt not in alts:
continue
+ fname = fnames[0]
blocks = list(read_pairs(name, fname))
st = numpy.concatenate([b[0] for b in blocks] + [b[1] for b in blocks])
for b in blocks:
@@ -110,7 +205,15 @@ def parse(alts=None):
g = Graph(directed=True)
g.add_vertex(len(npis))
g.vp.npi = g.new_vp("int64_t", vals=npis)
+ # the metadata of the providers, from the NPPES
+ meta = (pl.DataFrame({"npi": npis}).with_row_index("v")
+ .join(read_nppes(name, fnames[1], fnames[2]), on="npi", how="left")
+ .sort("v"))
del npis
+ g.vp.in_nppes = g.new_vp("bool", vals=meta["entity_type"].is_not_null().to_numpy())
+ for c in _nppes_cols:
+ g.vp[c] = g.new_vp("string", vals=meta[c].fill_null("").to_list())
+ del meta
g.add_edge_list(numpy.column_stack((idx[:m], idx[m:])))
del idx
for k, p in zip(["pairs", "beneficiaries", "same_day"], props):
[entries] New entry: physician_shared_patients 6113cb2
diff --git a/entries/physician_shared_patients/__init__.py b/entries/physician_shared_patients/__init__.py
new file mode 100644
index 0000000..917dfca
--- /dev/null
+++ b/entries/physician_shared_patients/__init__.py
@@ -0,0 +1,122 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2026 Tiago de Paula Peixoto <tiago@skewed.de>
+#
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option) any
+# later version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along
+# with this program. If not, see <http://www.gnu.org/licenses/>.
+
+import pyarrow
+import pyarrow.csv
+import pyarrow.compute as pc
+
+from .. import *
+
+title = "Medicare physician shared patient patterns (2009-2015)"
+description = r"""Networks of health care providers (physicians, other practitioners and organizations such as hospitals, laboratories and home health agencies) of the US Medicare program that took part in the care of the same patients, released by the Centers for Medicare & Medicaid Services (CMS) in response to a Freedom of Information Act request (the data are also known as "DocGraph", after the project that requested them). They were computed by CMS from the Medicare fee-for-service claims (and the Medicare Advantage encounter claims) of each calendar year, from 2009 to 2015.
+
+Each node is a provider, identified by its National Provider Identifier (NPI), and a directed edge from provider A to provider B means that, for some beneficiaries, a claim of B followed a claim of A within a given time window (30, 60, 90 or 180 days) after it. A provider is counted when it appears on a claim in any role of "treatment association" (billing organization, attending, operating, rendering or ordering physician, ...), except as referring physician or as supplier of durable medical equipment. Relationships from claims of the same day are assigned to the lesser of the two NPIs (as source). The identifiers are as given by CMS: a few thousand edges (e.g. 4,410 in `2009_days30`) involve identifiers that are not valid NPIs (with a leading zero), which are kept as they are. Pairs of providers that shared fewer than 11 distinct beneficiaries in the year were left out by CMS, to protect the privacy of the patients, so that the networks are thresholded at this value (the unthresholded data are not public). Since the windows extend beyond the end of the year, the claims considered for a year run until one, two, three or six months into the next one; the data of 2015 only cover claims until September or October 2015 (shorter windows than the others for the 60, 90 and 180 day periods).
+
+There is one network for each year and time window, named `<year>_days<window>` (e.g. `2014_days30`, `2014_days180`); the networks of longer windows contain those of shorter windows of the same year. No metadata of the providers (names, specialties, locations, which are in the National Plan and Provider Enumeration System, NPPES) are included; the providers can be looked up by their NPI.
+
+The networks have the following properties:
+
+- `npi`: the National Provider Identifier of the provider (a ten-digit number; invalid identifiers with leading zeros are given without them).
+- Edge `pairs`: the number of times a claim of the target followed a claim of the source within the window (the "pair count").
+- Edge `beneficiaries`: the number of distinct beneficiaries shared by the two providers (at least 11).
+- Edge `same_day`: the number of these sharing events in which both claims were on the same day.
+- `start` (graph), `end` (graph): the date range of the claims considered, as given by CMS (e.g. `2009-01-01` and `2010-02-01` for `2009_days30`)."""
+tags = ['Social', 'Public Health', 'Offline', 'Weighted']
+url = ['https://www.cms.gov/Regulations-and-Guidance/Legislation/FOIA/Referral-Data-FAQs',
+ 'https://downloads.cms.gov/foia/physician_shared_patient_patterns_technical_requirements.pdf']
+citation = [('Centers for Medicare & Medicaid Services, "Physician Shared Patient Patterns." Technical requirements (2015)',
+ 'https://downloads.cms.gov/foia/physician_shared_patient_patterns_technical_requirements.pdf')]
+icon_hash = None
+upstream_license = ("Public domain (US government work)", "https://www.usa.gov/government-copyright")
+
+upstream_prefix = 'https://downloads.cms.gov/foia'
+windows = [30, 60, 90, 180]
+years = range(2009, 2016)
+
+# the date ranges of the claims, from the technical requirements of CMS
+_ranges = {30: ("01", "02"), 60: ("01", "03"), 90: ("01", "04"),
+ 180: ("01", "07")}
+_ranges_2015 = {30: "2015-09-01", 60: "2015-10-01", 90: "2015-10-01",
+ 180: "2015-10-01"}
+
+def _range(year, window):
+ if year == 2015:
+ return "2015-01-01", _ranges_2015[window]
+ return f"{year}-01-01", f"{year + 1}-{_ranges[window][1]}-01"
+
+files = [(f"physician-shared-patient-patterns-{y}-days{w}.zip:"
+ f"physician-shared-patient-patterns-{y}-days{w}.txt", f"{y}_days{w}", None)
+ for y in years for w in windows]
+
+def fetch_upstream(force=False):
+ return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
+ force)
+
+# For checks only: if set, only (about) this many lines are read
+_check_rows = None
+
+def read_pairs(name, fname):
+ """The lines "NPI1,NPI2,pair count,beneficiary count,same day count"
+ (fixed-width numbers padded with spaces), as blocks of arrays."""
+ cols = ["npi1", "npi2", "pairs", "benes", "same_day"]
+ n = 0
+ with open_upstream_file(name, fname, "rb") as f:
+ reader = pyarrow.csv.open_csv(
+ f,
+ read_options=pyarrow.csv.ReadOptions(column_names=cols,
+ block_size=1 << 26),
+ convert_options=pyarrow.csv.ConvertOptions(
+ column_types={c: pyarrow.string() for c in cols}))
+ for batch in reader:
+ if _check_rows is not None and n >= _check_rows:
+ break
+ n += batch.num_rows
+ yield [pc.cast(pc.utf8_trim_whitespace(c),
+ pyarrow.int64() if k < 2 else pyarrow.int32()).to_numpy()
+ for k, c in enumerate(batch.columns)]
+
+@cache_network()
+@coerce_props()
+@annotate()
+def parse(alts=None):
+ name = __name__.split(".")[-1]
+ for fname, alt, fmt in files:
+ if alts is not None and alt not in alts:
+ continue
+ blocks = list(read_pairs(name, fname))
+ st = numpy.concatenate([b[0] for b in blocks] + [b[1] for b in blocks])
+ for b in blocks:
+ del b[:2]
+ props = [numpy.concatenate([b[k] for b in blocks]) for k in (0, 1, 2)]
+ del blocks
+ npis, idx = numpy.unique(st, return_inverse=True)
+ del st
+ m = len(idx) // 2
+ g = Graph(directed=True)
+ g.add_vertex(len(npis))
+ g.vp.npi = g.new_vp("int64_t", vals=npis)
+ del npis
+ g.add_edge_list(numpy.column_stack((idx[:m], idx[m:])))
+ del idx
+ for k, p in zip(["pairs", "beneficiaries", "same_day"], props):
+ g.ep[k] = g.new_ep("int32_t", vals=p)
+ del props
+ year, window = alt.split("_days")
+ g.gp.start, g.gp.end = [g.new_gp("string", v)
+ for v in _range(int(year), int(window))]
+ yield alt, g