History of open_payments
The changes to the source code of this entry in the repository, newest first, each with the differences it made to the entry's files. They are also listed on GitLab.
[entries] Add the date when each entry was added, and remove icon_hash beff7c9
The dates are those of the commits that first added the entries' files (2020-08-11 for the entries of the initial commit); renamed entries keep the date of their first name. Changed: none
diff --git a/entries/open_payments/__init__.py b/entries/open_payments/__init__.py
index c229c96..969bbba 100644
--- a/entries/open_payments/__init__.py
+++ b/entries/open_payments/__init__.py
@@ -48,7 +48,7 @@ url = ['https://openpaymentsdata.cms.gov/',
'https://download.cms.gov/openpayments/']
citation = [('Centers for Medicare & Medicaid Services, "Open Payments: General Payment Data." Publication of June 30, 2026 (program years 2019-2025)',
'https://openpaymentsdata.cms.gov/')]
-icon_hash = None
+date_added = '2026-10-28'
upstream_license = ("Public domain (US government work)", "https://www.usa.gov/government-works")
# The files of the publication of June 30, 2026 (as listed in
[entries] Prefix the properties in the descriptions with (Node), (Edge) or (Graph) bc3dbbd
The lists of properties at the end of the descriptions marked edge properties with "Edge" but left node properties unmarked, or grouped them in "Nodes: `a`, ...; `b`, ..." items, so the kind of a property was often unclear. Every item now starts with (Node), (Edge) or (Graph), followed by the property names, a colon and the explanation, one item per property or group of properties; qualifications (kinds of nodes, networks) follow the names in parentheses. Properties that were only named got a short explanation, and a few items that named the wrong kind of property were corrected.
diff --git a/entries/open_payments/__init__.py b/entries/open_payments/__init__.py
index aa490f3..c229c96 100644
--- a/entries/open_payments/__init__.py
+++ b/entries/open_payments/__init__.py
@@ -25,24 +25,24 @@ Each network is a bipartite multigraph of the paying companies and the recipient
The networks have the following properties:
-- `is_company`: whether the node is a paying company (otherwise it is a recipient).
-- `id`: the Open Payments identifier of the company, the profile identifier of the physician or practitioner, or the identifier of the teaching hospital.
-- `name`: the name of the company or of the teaching hospital, or the name of the physician or practitioner (first, middle and last names, and suffix).
-- `kind`: `Company`, `Physician`, `Non-Physician Practitioner` or `Teaching Hospital`.
-- `npi` (physicians and practitioners): the National Provider Identifier of the recipient (0 if not given, and for the other nodes).
-- `ccn` (teaching hospitals): the CMS Certification Number of the hospital (empty for the other nodes).
-- `city` (recipients), `state`, `country`: the location of the recipient (its primary business address), or the state and country of the company.
-- `primary_type` (physicians and practitioners): the first primary type of the recipient (e.g. `Medical Doctor`, `Nurse Practitioner`).
-- `specialty` (physicians and practitioners): the first specialty of the recipient, as a path of the provider taxonomy (e.g. `Allopathic & Osteopathic Physicians|Family Medicine`).
-- Edge `time`: the date of the payment (of the first payment if several are aggregated), as the Unix time of its beginning (midnight UTC).
-- Edge `amount`: the total amount of the payment, in US dollars.
-- Edge `payments`: the number of payments included in the record.
-- Edge `form`: the form of payment (e.g. `Cash or cash equivalent`, `In-kind items and services`, `Stock`).
-- Edge `nature`: the nature of the payment (e.g. `Food and Beverage`, `Consulting Fee`, `Travel and Lodging`, `Royalty or License`).
-- Edge `product`: the name of the (first) drug, device or supply associated with the payment (empty if none).
-- Edge `product_type`: the type of that product (`Drug`, `Biological`, `Device`, `Medical Supply`, or empty).
-- Edge `third_party`: the indicator of payments to a third party at the request of the recipient (e.g. `No Third Party Payment`, `Entity`, `Individual`).
-- Edge `record_id`: the identifier of the record in Open Payments."""
+- (Node) `is_company`: whether the node is a paying company (otherwise it is a recipient).
+- (Node) `id`: the Open Payments identifier of the company, the profile identifier of the physician or practitioner, or the identifier of the teaching hospital.
+- (Node) `name`: the name of the company or of the teaching hospital, or the name of the physician or practitioner (first, middle and last names, and suffix).
+- (Node) `kind`: `Company`, `Physician`, `Non-Physician Practitioner` or `Teaching Hospital`.
+- (Node) `npi` (physicians and practitioners): the National Provider Identifier of the recipient (0 if not given, and for the other nodes).
+- (Node) `ccn` (teaching hospitals): the CMS Certification Number of the hospital (empty for the other nodes).
+- (Node) `city` (recipients), `state`, `country`: the location of the recipient (its primary business address), or the state and country of the company.
+- (Node) `primary_type` (physicians and practitioners): the first primary type of the recipient (e.g. `Medical Doctor`, `Nurse Practitioner`).
+- (Node) `specialty` (physicians and practitioners): the first specialty of the recipient, as a path of the provider taxonomy (e.g. `Allopathic & Osteopathic Physicians|Family Medicine`).
+- (Edge) `time`: the date of the payment (of the first payment if several are aggregated), as the Unix time of its beginning (midnight UTC).
+- (Edge) `amount`: the total amount of the payment, in US dollars.
+- (Edge) `payments`: the number of payments included in the record.
+- (Edge) `form`: the form of payment (e.g. `Cash or cash equivalent`, `In-kind items and services`, `Stock`).
+- (Edge) `nature`: the nature of the payment (e.g. `Food and Beverage`, `Consulting Fee`, `Travel and Lodging`, `Royalty or License`).
+- (Edge) `product`: the name of the (first) drug, device or supply associated with the payment (empty if none).
+- (Edge) `product_type`: the type of that product (`Drug`, `Biological`, `Device`, `Medical Supply`, or empty).
+- (Edge) `third_party`: the indicator of payments to a third party at the request of the recipient (e.g. `No Third Party Payment`, `Entity`, `Individual`).
+- (Edge) `record_id`: the identifier of the record in Open Payments."""
tags = ['Economic', 'Public Health', 'Bipartite', 'Multigraph', 'Weighted', 'Temporal', 'Timestamps', 'Metadata']
url = ['https://openpaymentsdata.cms.gov/',
'https://download.cms.gov/openpayments/']
[entries] New entry: open_payments 88c399f
diff --git a/entries/open_payments/__init__.py b/entries/open_payments/__init__.py
new file mode 100644
index 0000000..aa490f3
--- /dev/null
+++ b/entries/open_payments/__init__.py
@@ -0,0 +1,184 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2026 Tiago de Paula Peixoto <tiago@skewed.de>
+#
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option) any
+# later version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along
+# with this program. If not, see <http://www.gnu.org/licenses/>.
+
+from .. import *
+
+title = "Open Payments: industry payments to US health care providers (2019-2025)"
+description = r"""Payments and other transfers of value from manufacturers of drugs, medical devices and medical supplies, and from group purchasing organizations, to physicians, other health care practitioners (physician assistants, nurse practitioners, ...) and teaching hospitals in the United States, as reported by the companies to the Open Payments program of the Centers for Medicare & Medicaid Services (CMS), which publishes them under the Physician Payments Sunshine Act. The networks contain the "general payments" (payments not made in connection with a research agreement: food and beverage, consulting fees, travel, speaker fees, royalties, education, gifts, ...), with one network per program year (the calendar year of the payments), from 2019 to 2025, named after the year (e.g. `2024`).
+
+Each network is a bipartite multigraph of the paying companies and the recipients, with a directed edge from a company to a recipient for each payment record, with its date, amount, form, nature and associated product. A record may aggregate several payments of small value (with the number of payments given). The companies are identified by their Open Payments identifier, the physicians and practitioners by their Open Payments profile identifier, and the teaching hospitals by their Open Payments identifier; only the companies and recipients with some payment in the year are included. The records are those of the publication of June 2026 (the data are refreshed by CMS at least once a year, with corrections and disputes resolved by the recipients, so that the data of a year change between publications). The years 2013 to 2018, which are no longer part of the current publication of CMS, are not included, and neither are the research payments, the ownership and investment interests, the addresses of the recipients and companies, the payments to third parties (only their flag), and the products other than the first one listed with each payment.
+
+The networks have the following properties:
+
+- `is_company`: whether the node is a paying company (otherwise it is a recipient).
+- `id`: the Open Payments identifier of the company, the profile identifier of the physician or practitioner, or the identifier of the teaching hospital.
+- `name`: the name of the company or of the teaching hospital, or the name of the physician or practitioner (first, middle and last names, and suffix).
+- `kind`: `Company`, `Physician`, `Non-Physician Practitioner` or `Teaching Hospital`.
+- `npi` (physicians and practitioners): the National Provider Identifier of the recipient (0 if not given, and for the other nodes).
+- `ccn` (teaching hospitals): the CMS Certification Number of the hospital (empty for the other nodes).
+- `city` (recipients), `state`, `country`: the location of the recipient (its primary business address), or the state and country of the company.
+- `primary_type` (physicians and practitioners): the first primary type of the recipient (e.g. `Medical Doctor`, `Nurse Practitioner`).
+- `specialty` (physicians and practitioners): the first specialty of the recipient, as a path of the provider taxonomy (e.g. `Allopathic & Osteopathic Physicians|Family Medicine`).
+- Edge `time`: the date of the payment (of the first payment if several are aggregated), as the Unix time of its beginning (midnight UTC).
+- Edge `amount`: the total amount of the payment, in US dollars.
+- Edge `payments`: the number of payments included in the record.
+- Edge `form`: the form of payment (e.g. `Cash or cash equivalent`, `In-kind items and services`, `Stock`).
+- Edge `nature`: the nature of the payment (e.g. `Food and Beverage`, `Consulting Fee`, `Travel and Lodging`, `Royalty or License`).
+- Edge `product`: the name of the (first) drug, device or supply associated with the payment (empty if none).
+- Edge `product_type`: the type of that product (`Drug`, `Biological`, `Device`, `Medical Supply`, or empty).
+- Edge `third_party`: the indicator of payments to a third party at the request of the recipient (e.g. `No Third Party Payment`, `Entity`, `Individual`).
+- Edge `record_id`: the identifier of the record in Open Payments."""
+tags = ['Economic', 'Public Health', 'Bipartite', 'Multigraph', 'Weighted', 'Temporal', 'Timestamps', 'Metadata']
+url = ['https://openpaymentsdata.cms.gov/',
+ 'https://download.cms.gov/openpayments/']
+citation = [('Centers for Medicare & Medicaid Services, "Open Payments: General Payment Data." Publication of June 30, 2026 (program years 2019-2025)',
+ 'https://openpaymentsdata.cms.gov/')]
+icon_hash = None
+upstream_license = ("Public domain (US government work)", "https://www.usa.gov/government-works")
+
+# The files of the publication of June 30, 2026 (as listed in
+# https://openpaymentsdata.cms.gov/api/1/metastore/schemas/dataset/items):
+# the publication date is part of their paths, but the files of a
+# publication are removed when the next one is published (once or twice a
+# year), and older program years are dropped from the publications
+upstream_prefix = 'https://download.cms.gov/openpayments'
+publication = "P06302026_06032026"
+years = range(2019, 2026)
+files = [(f"PGYR{y}_{publication}/OP_DTL_GNRL_PGYR{y}_{publication}.csv", str(y), None)
+ for y in years]
+
+def fetch_upstream(force=False):
+ return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
+ force)
+
+_cols = {
+ "Covered_Recipient_Type": "rtype",
+ "Teaching_Hospital_CCN": "ccn",
+ "Teaching_Hospital_ID": "hospital_id",
+ "Teaching_Hospital_Name": "hospital_name",
+ "Covered_Recipient_Profile_ID": "profile_id",
+ "Covered_Recipient_NPI": "npi",
+ "Covered_Recipient_First_Name": "first",
+ "Covered_Recipient_Middle_Name": "middle",
+ "Covered_Recipient_Last_Name": "last",
+ "Covered_Recipient_Name_Suffix": "suffix",
+ "Recipient_City": "city",
+ "Recipient_State": "state",
+ "Recipient_Country": "country",
+ "Covered_Recipient_Primary_Type_1": "primary_type",
+ "Covered_Recipient_Specialty_1": "specialty",
+ "Applicable_Manufacturer_or_Applicable_GPO_Making_Payment_ID": "company_id",
+ "Applicable_Manufacturer_or_Applicable_GPO_Making_Payment_Name": "company_name",
+ "Applicable_Manufacturer_or_Applicable_GPO_Making_Payment_State": "company_state",
+ "Applicable_Manufacturer_or_Applicable_GPO_Making_Payment_Country": "company_country",
+ "Total_Amount_of_Payment_USDollars": "amount",
+ "Date_of_Payment": "date",
+ "Number_of_Payments_Included_in_Total_Amount": "payments",
+ "Form_of_Payment_or_Transfer_of_Value": "form",
+ "Nature_of_Payment_or_Transfer_of_Value": "nature",
+ "Third_Party_Payment_Recipient_Indicator": "third_party",
+ "Record_ID": "record_id",
+ "Indicate_Drug_or_Biological_or_Device_or_Medical_Supply_1": "product_type",
+ "Name_of_Drug_or_Biological_or_Device_or_Medical_Supply_1": "product",
+}
+
+_kinds = {"Covered Recipient Physician": "Physician",
+ "Covered Recipient Non-Physician Practitioner": "Non-Physician Practitioner",
+ "Covered Recipient Teaching Hospital": "Teaching Hospital"}
+
+def _join(*cols):
+ import polars as pl
+ return pl.concat_str([pl.col(c) for c in cols], separator=" ",
+ ignore_nulls=True).str.replace_all(r"\s+", " ").str.strip_chars()
+
+# For checks only: if set, only (about) this many rows are read
+_check_rows = None
+
+@cache_network()
+@coerce_props(vforce={"ccn": "string"})
+@annotate()
+def parse(alts=None):
+ import polars as pl
+ name = __name__.split(".")[-1]
+ for fname, alt, fmt in files:
+ if alts is not None and alt not in alts:
+ continue
+ with open_upstream_file(name, fname, "rb") as f:
+ df = pl.read_csv(f, columns=list(_cols), infer_schema=False,
+ n_rows=_check_rows)
+ df = df.rename(_cols).with_columns(pl.all().fill_null(""))
+ hosp = df["rtype"] == "Covered Recipient Teaching Hospital"
+ df = df.with_columns(
+ key=pl.when(hosp).then(pl.lit("h") + pl.col("hospital_id"))
+ .otherwise(pl.lit("p") + pl.col("profile_id")),
+ rid=pl.when(hosp).then(pl.col("hospital_id")).otherwise(pl.col("profile_id")),
+ rname=pl.when(hosp).then(pl.col("hospital_name"))
+ .otherwise(_join("first", "middle", "last", "suffix")),
+ kind=pl.col("rtype").replace_strict(_kinds, default=pl.col("rtype")))
+
+ # the companies, in the order of their identifiers
+ comp = (df.select(["company_id", "company_name", "company_state",
+ "company_country"])
+ .unique(subset="company_id", keep="first", maintain_order=True)
+ .with_columns(n=pl.col("company_id").cast(pl.Int64, strict=False))
+ .sort(["n", "company_id"]).drop("n")
+ .with_row_index("s"))
+ # the recipients (with the attributes of their first record), in
+ # the order of their kinds and identifiers
+ rec = (df.select(["key", "rid", "rname", "kind", "npi", "ccn", "city",
+ "state", "country", "primary_type", "specialty"])
+ .unique(subset="key", keep="first", maintain_order=True)
+ .with_columns(n=pl.col("rid").cast(pl.Int64, strict=False))
+ .sort(["kind", "n", "rid"]).drop("n")
+ .with_row_index("t"))
+ nc = len(comp)
+ df = (df.join(comp.select("company_id", "s"), on="company_id", how="left",
+ maintain_order="left")
+ .join(rec.select("key", "t"), on="key", how="left",
+ maintain_order="left"))
+
+ g = Graph(directed=True)
+ g.add_vertex(nc + len(rec))
+ g.vp.is_company = g.new_vp("bool", vals=numpy.arange(g.num_vertices()) < nc)
+ g.vp.id = g.new_vp("string", vals=comp["company_id"].to_list() + rec["rid"].to_list())
+ g.vp.name = g.new_vp("string", vals=comp["company_name"].to_list() + rec["rname"].to_list())
+ g.vp.kind = g.new_vp("string", vals=["Company"] * nc + rec["kind"].to_list())
+ npi = rec["npi"].cast(pl.Int64, strict=False).fill_null(0).to_numpy()
+ g.vp.npi = g.new_vp("int64_t", vals=numpy.concatenate((numpy.zeros(nc, dtype="int64"), npi)))
+ g.vp.ccn = g.new_vp("string", vals=[""] * nc + rec["ccn"].to_list())
+ g.vp.city = g.new_vp("string", vals=[""] * nc + rec["city"].to_list())
+ g.vp.state = g.new_vp("string", vals=comp["company_state"].to_list() + rec["state"].to_list())
+ g.vp.country = g.new_vp("string", vals=comp["company_country"].to_list() + rec["country"].to_list())
+ g.vp.primary_type = g.new_vp("string", vals=[""] * nc + rec["primary_type"].to_list())
+ g.vp.specialty = g.new_vp("string", vals=[""] * nc + rec["specialty"].to_list())
+ del comp, rec
+
+ g.add_edge_list(numpy.column_stack((df["s"].to_numpy().astype("int64"),
+ df["t"].to_numpy().astype("int64") + nc)))
+ t = (df["date"].str.strptime(pl.Date, "%m/%d/%Y", strict=False)
+ .cast(pl.Datetime("us")).dt.epoch("s"))
+ if t.null_count() > 0:
+ raise ValueError(f"{alt}: {t.null_count()} invalid dates")
+ g.ep.time = g.new_ep("int64_t", vals=t.to_numpy())
+ g.ep.amount = g.new_ep("double", vals=df["amount"].cast(pl.Float64).to_numpy())
+ g.ep.payments = g.new_ep("int32_t", vals=df["payments"].cast(pl.Int32).to_numpy())
+ for c in ["form", "nature", "product", "product_type", "third_party"]:
+ g.ep[c] = g.new_ep("string", vals=df[c].to_list())
+ g.ep.record_id = g.new_ep("int64_t", vals=df["record_id"].cast(pl.Int64).to_numpy())
+ del df
+ yield alt, g