logo Netzschleuder network catalog, repository, and centrifuge

History of mind_news

The changes to the source code of this entry in the repository, newest first, each with the differences it made to the entry's files. They are also listed on GitLab.

[entries] Add the date when each entry was added, and remove icon_hash by Tiago de Paula Peixoto beff7c9
The dates are those of the commits that first added the entries' files
(2020-08-11 for the entries of the initial commit); renamed entries keep
the date of their first name.

Changed: none
diff --git a/entries/mind_news/__init__.py b/entries/mind_news/__init__.py
index a1f2d64..ca7b552 100644
--- a/entries/mind_news/__init__.py
+++ b/entries/mind_news/__init__.py
@@ -45,7 +45,7 @@ tags = ['Social', 'Online', 'Informational', 'Preferences', 'Unweighted', 'Multi
 url = ['https://msnews.github.io/',
        'https://huggingface.co/datasets/Recommenders/MIND']
 citation = [('F. Wu, Y. Qiao, J.-H. Chen, C. Wu, T. Qi, J. Lian, D. Liu, X. Xie, J. Gao, W. Wu, M. Zhou, "MIND: A Large-scale Dataset for News Recommendation." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 3597-3606 (2020)', 'https://doi.org/10.18653/v1/2020.acl-main.331')]
-icon_hash = None
+date_added = '2026-10-28'
 upstream_license = ("Microsoft Research License Terms (non-commercial research use; no distribution)", "https://github.com/msnews/MIND/blob/master/MSR%20License_Data.pdf")
 restricted = True
 # The original download location (Azure blob storage, linked from
[entries] Prefix the properties in the descriptions with (Node), (Edge) or (Graph) by Tiago de Paula Peixoto bc3dbbd
The lists of properties at the end of the descriptions marked edge
properties with "Edge" but left node properties unmarked, or grouped
them in "Nodes: `a`, ...; `b`, ..." items, so the kind of a property
was often unclear. Every item now starts with (Node), (Edge) or (Graph),
followed by the property names, a colon and the explanation, one item
per property or group of properties; qualifications (kinds of nodes,
networks) follow the names in parentheses. Properties that were only
named got a short explanation, and a few items that named the wrong
kind of property were corrected.
diff --git a/entries/mind_news/__init__.py b/entries/mind_news/__init__.py
index 637719c..a1f2d64 100644
--- a/entries/mind_news/__init__.py
+++ b/entries/mind_news/__init__.py
@@ -32,10 +32,15 @@ Both networks contain all the users and all the news articles of the data set. L
 
 The networks have the following properties:
 
-- Nodes: `is_user`, whether the node is a user (otherwise a news article); `name`, the anonymized id of the user, or the id of the news article in the data set.
-- News articles: `category` and `subcategory`, the category and subcategory of the article on Microsoft News; `title`, the title of the article; `url`, the address of the article at the time of the release of the data set (empty for users).
-- Edges: `time`, the time of the impression, as Unix time (the times are given without a time zone, and taken here as UTC; -1 for the click histories); `impression`, the number of the impression within its set (-1 for the click histories); `split`, the set of the impression: 0 for training, 1 for validation, 2 for test (-1 for the click histories).
-- `impressions`, edges: `clicked`, 1 if the user clicked the article, 0 if not, -1 if unknown (test set)."""
+- (Node) `is_user`: whether the node is a user (otherwise a news article).
+- (Node) `name`: the anonymized id of the user, or the id of the news article in the data set.
+- (Node) `category`, `subcategory` (news articles): the category and subcategory of the article on Microsoft News (empty for users).
+- (Node) `title` (news articles): the title of the article (empty for users).
+- (Node) `url` (news articles): the address of the article at the time of the release of the data set (empty for users).
+- (Edge) `time`: the time of the impression, as Unix time (the times are given without a time zone, and taken here as UTC; -1 for the click histories).
+- (Edge) `impression`: the number of the impression within its set (-1 for the click histories).
+- (Edge) `split`: the set of the impression: 0 for training, 1 for validation, 2 for test (-1 for the click histories).
+- (Edge) `clicked` (`impressions`): 1 if the user clicked the article, 0 if not, -1 if unknown (test set)."""
 tags = ['Social', 'Online', 'Informational', 'Preferences', 'Unweighted', 'Multigraph', 'Timestamps', 'Metadata']
 url = ['https://msnews.github.io/',
        'https://huggingface.co/datasets/Recommenders/MIND']
[entries] Remove unneeded code from the entries by Tiago de Paula Peixoto 2c69f19
Unneeded "global files" statements, unused imports and variables,
f-strings without placeholders, duplicate keys in foodweb_little_rock
(keeping the order and the values used), and the second definition of
get_lines() in twitter_higgs. Nothing changes in the networks or
descriptions.

Changed: none
diff --git a/entries/mind_news/__init__.py b/entries/mind_news/__init__.py
index 87e064f..637719c 100644
--- a/entries/mind_news/__init__.py
+++ b/entries/mind_news/__init__.py
@@ -126,7 +126,6 @@ def read_behaviors(name, fname):
 @coerce_props()
 @annotate()
 def parse(alts=None):
-    global files
     name = __name__.split(".")[-1]
     for fnames, alt, fmt in files:
         if alts is not None and alt not in alts:
[entries] Remove the remarks on the distribution of the data from the descriptions of beer_reviews, bluesky_failla, citibike, divvy, facebook100, goodreads, hiv_transmission, hollywood_actors, mind_news, movielens_10m_tags, movielens_32m, movielens_ratings, netflix_prize, opensky_flights, opensky_flights_full, regulondb, statsbomb, yelp_restaurants and yelp_reviews by Tiago de Paula Peixoto 8be09c8
The descriptions said whether the data may be distributed (access
tokens, restrictions, redistribution bans, take-downs); the site shows
restricted entries itself, and the descriptions now only describe the
data.
diff --git a/entries/mind_news/__init__.py b/entries/mind_news/__init__.py
index 612d0e1..87e064f 100644
--- a/entries/mind_news/__init__.py
+++ b/entries/mind_news/__init__.py
@@ -35,9 +35,7 @@ The networks have the following properties:
 - Nodes: `is_user`, whether the node is a user (otherwise a news article); `name`, the anonymized id of the user, or the id of the news article in the data set.
 - News articles: `category` and `subcategory`, the category and subcategory of the article on Microsoft News; `title`, the title of the article; `url`, the address of the article at the time of the release of the data set (empty for users).
 - Edges: `time`, the time of the impression, as Unix time (the times are given without a time zone, and taken here as UTC; -1 for the click histories); `impression`, the number of the impression within its set (-1 for the click histories); `split`, the set of the impression: 0 for training, 1 for validation, 2 for test (-1 for the click histories).
-- `impressions`, edges: `clicked`, 1 if the user clicked the article, 0 if not, -1 if unknown (test set).
-
-The files of this entry are available only with an access token, since the license of the data set (Microsoft Research License Terms) does not allow distributing it."""
+- `impressions`, edges: `clicked`, 1 if the user clicked the article, 0 if not, -1 if unknown (test set)."""
 tags = ['Social', 'Online', 'Informational', 'Preferences', 'Unweighted', 'Multigraph', 'Timestamps', 'Metadata']
 url = ['https://msnews.github.io/',
        'https://huggingface.co/datasets/Recommenders/MIND']
[entries] New entry: mind_news by Tiago de Paula Peixoto ae50f34
diff --git a/entries/mind_news/__init__.py b/entries/mind_news/__init__.py
new file mode 100644
index 0000000..612d0e1
--- /dev/null
+++ b/entries/mind_news/__init__.py
@@ -0,0 +1,241 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2020-2026 Tiago de Paula Peixoto <tiago@skewed.de>
+#
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option) any
+# later version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from .. import *
+
+from array import array
+import hashlib
+import calendar
+
+title = "MIND news clicks of Microsoft News users (2019)"
+description = """Bipartite networks of the users and news articles of the Microsoft News website, from the large version of the MIND data set (Microsoft News Dataset), compiled by Wu et al. from the anonymized behavior logs of Microsoft News. One million users who had clicked at least five news articles in the six weeks from October 12 to November 22, 2019 were sampled at random, and their ids were replaced by anonymous ones. The logs of the users consist of impressions: each impression is a list of news articles that were shown to a user at some time, with the ones that the user clicked marked. The impressions of November 9 to 14 form the training set of the data set, those of November 15 the validation set, and those of November 16 to 22 the test set, for which the clicks are not given. Each impression also comes with the click history of the user, the articles the user clicked before (in the first four weeks, from October 12 to November 8), without their times. The data set contains 4,979,946 impressions of 876,956 users and 130,379 news articles.
+
+- `clicks`: an edge connects a user to a news article that the user clicked: once for each click in an impression of the training or validation set (3,958,501 clicks, with their times), and once for each article in the user's click history (18,131,836 pairs, without times; the histories given with the impressions of a user were merged, keeping each user-article pair once).
+- `impressions`: an edge connects a user to a news article for each time the article was shown to the user in an impression (190,707,932 in all), with whether the user clicked it (unknown for the test set).
+
+Both networks contain all the users and all the news articles of the data set. Left out are the abstracts of the articles and the entities (linked to Wikidata) recognized in their titles and abstracts, with their embeddings, as well as the smaller versions of the data set (MIND-small, a sample of 50,000 of the users, and MIND-demo), which are subsets of this one.
+
+The networks have the following properties:
+
+- Nodes: `is_user`, whether the node is a user (otherwise a news article); `name`, the anonymized id of the user, or the id of the news article in the data set.
+- News articles: `category` and `subcategory`, the category and subcategory of the article on Microsoft News; `title`, the title of the article; `url`, the address of the article at the time of the release of the data set (empty for users).
+- Edges: `time`, the time of the impression, as Unix time (the times are given without a time zone, and taken here as UTC; -1 for the click histories); `impression`, the number of the impression within its set (-1 for the click histories); `split`, the set of the impression: 0 for training, 1 for validation, 2 for test (-1 for the click histories).
+- `impressions`, edges: `clicked`, 1 if the user clicked the article, 0 if not, -1 if unknown (test set).
+
+The files of this entry are available only with an access token, since the license of the data set (Microsoft Research License Terms) does not allow distributing it."""
+tags = ['Social', 'Online', 'Informational', 'Preferences', 'Unweighted', 'Multigraph', 'Timestamps', 'Metadata']
+url = ['https://msnews.github.io/',
+       'https://huggingface.co/datasets/Recommenders/MIND']
+citation = [('F. Wu, Y. Qiao, J.-H. Chen, C. Wu, T. Qi, J. Lian, D. Liu, X. Xie, J. Gao, W. Wu, M. Zhou, "MIND: A Large-scale Dataset for News Recommendation." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 3597-3606 (2020)', 'https://doi.org/10.18653/v1/2020.acl-main.331')]
+icon_hash = None
+upstream_license = ("Microsoft Research License Terms (non-commercial research use; no distribution)", "https://github.com/msnews/MIND/blob/master/MSR%20License_Data.pdf")
+restricted = True
+# The original download location (Azure blob storage, linked from
+# msnews.github.io) is no longer public; the files are those of the copy on
+# Hugging Face, pinned to a revision of the repository
+revision = '2115068f457b577f93a268fffe22373e45727d89'
+upstream_prefix = f'https://huggingface.co/datasets/Recommenders/MIND/resolve/{revision}'
+splits = ["train", "dev", "test"]
+_fnames = tuple(f'MINDlarge_{s}.zip:{m}' for s in splits
+                for m in ("behaviors.tsv", "news.tsv"))
+files = [(_fnames, 'clicks', None),
+         (_fnames, 'impressions', None)]
+
+# SHA-256 sums of the files (from the Git LFS pointers of the revision)
+sha256sums = {
+    "MINDlarge_train.zip": "7e7da0ecc45db1d8769eebf92301675e7af443b0d5266ac12b6e102bb8399233",
+    "MINDlarge_dev.zip": "67b0437b48b6baee5655b6063a56d29edbcc92308fc0140555b0ae27263cf9e9",
+    "MINDlarge_test.zip": "0cb0af2e61b4e02d1c5779ac29a732b556f95b15f2452075ec7e550f057a15f0",
+}
+
+def fetch_upstream(force=False):
+    # The files are checked against their SHA-256 sums, since a download
+    # that breaks off is not always reported as an error
+    name = __name__.split(".")[-1]
+    for fname, sha in sha256sums.items():
+        path = f"cache/upstream/{name}/{fname}"
+        if not force and os.path.exists(path):
+            continue
+        os.makedirs(os.path.dirname(path), exist_ok=True)
+        def download():
+            h = hashlib.sha256()
+            with open_url(f"{upstream_prefix}/{fname}") as f, \
+                 open(path + ".part", "wb") as out:
+                while True:
+                    buf = f.read(1 << 24)
+                    if not buf:
+                        break
+                    h.update(buf)
+                    out.write(buf)
+            if h.hexdigest() != sha:
+                raise IOError(f"unexpected content for {fname}")
+        retry_download(download, tries=3)
+        os.replace(path + ".part", path)
+
+# For checks only: if set, only this many impressions are read from each file
+_check_rows = None
+
+def parse_time(s):
+    "Unix time of a time stamp such as '11/15/2019 8:55:22 AM' (taken as UTC)."
+    d, t, ampm = s.split(" ")
+    mo, dd, yy = d.split("/")
+    hh, mi, ss = t.split(":")
+    hh = int(hh) % 12 + (12 if ampm == "PM" else 0)
+    return calendar.timegm((int(yy), int(mo), int(dd), hh, int(mi), int(ss)))
+
+def read_news(name, fnames):
+    "News id -> (category, subcategory, title, url), from all the splits."
+    news = {}
+    for fname in fnames:
+        if not fname.endswith("news.tsv"):
+            continue
+        with open_upstream_file(name, fname, "r") as f:
+            for line in f:
+                row = line.rstrip("\n").split("\t")
+                if len(row) < 6:
+                    continue
+                nid, cat, subcat, ntitle, abstract, nurl = row[:6]
+                news.setdefault(nid, (cat, subcat, ntitle, nurl))
+    return news
+
+def read_behaviors(name, fname):
+    "Rows (impression id, user id, time, history, impressions) of a file."
+    with open_upstream_file(name, fname, "r") as f:
+        for i, line in enumerate(f):
+            if _check_rows is not None and i >= _check_rows:
+                break
+            row = line.rstrip("\n").split("\t")
+            if len(row) != 5:
+                continue
+            yield row
+
+@cache_network()
+@coerce_props()
+@annotate()
+def parse(alts=None):
+    global files
+    name = __name__.split(".")[-1]
+    for fnames, alt, fmt in files:
+        if alts is not None and alt not in alts:
+            continue
+        news = read_news(name, fnames)
+        news_idx = {nid: i for i, nid in enumerate(news)}
+        def get_news(nid):
+            i = news_idx.get(nid)
+            if i is None:
+                # news without metadata
+                i = news_idx[nid] = len(news_idx)
+            return i
+        user_idx = {}
+        # 32-bit arrays, to save memory (the impressions have ~2*10^8 rows)
+        src = array("i")
+        tgt = array("i")
+        times = array("i")
+        imps = array("i")
+        splt = array("b")
+        clicked = array("b")
+        seen_hist = set()
+        for fname in fnames:
+            if not fname.endswith("behaviors.tsv"):
+                continue
+            split = splits.index(fname.split("_")[1].split(".")[0])
+            for imp, uid, t, hist, items in read_behaviors(name, fname):
+                u = user_idx.setdefault(uid, len(user_idx))
+                t = parse_time(t)
+                imp = int(imp)
+                if alt == "clicks" and hist != "":
+                    key = (u, hash(hist))
+                    if key not in seen_hist:
+                        seen_hist.add(key)
+                        for nid in hist.split(" "):
+                            src.append(u)
+                            tgt.append(get_news(nid))
+                            times.append(-1)
+                            imps.append(-1)
+                            splt.append(-1)
+                for item in items.split(" "):
+                    if "-" in item:
+                        nid, c = item.split("-")
+                        c = int(c)
+                    else:
+                        nid, c = item, -1
+                    if alt == "clicks" and c != 1:
+                        continue
+                    src.append(u)
+                    tgt.append(get_news(nid))
+                    times.append(t)
+                    imps.append(imp)
+                    splt.append(split)
+                    if alt == "impressions":
+                        clicked.append(c)
+        del seen_hist
+        src = numpy.frombuffer(src, dtype="int32")
+        tgt = numpy.frombuffer(tgt, dtype="int32")
+        times = numpy.frombuffer(times, dtype="int32")
+        imps = numpy.frombuffer(imps, dtype="int32")
+        splt = numpy.frombuffer(splt, dtype="int8")
+        if alt == "clicks":
+            # The click histories of a user can repeat in the splits: keep
+            # each (user, news) pair of the histories once
+            N_n = len(news_idx)
+            hist = times == -1
+            key = src[hist].astype("int64") * N_n + tgt[hist]
+            _, first = numpy.unique(key, return_index=True)
+            keep = numpy.ones(len(src), dtype="bool")
+            hidx = numpy.where(hist)[0]
+            keep[hidx] = False
+            keep[hidx[first]] = True
+            src, tgt, times, imps, splt = (x[keep] for x in (src, tgt, times, imps, splt))
+            del keep, hist, key, hidx, first
+        else:
+            clicked = numpy.frombuffer(clicked, dtype="int8")
+        N_u = len(user_idx)
+        g = Graph(directed=False)
+        g.add_vertex(N_u + len(news_idx))
+        g.vp.is_user = g.new_vp("bool")
+        g.vp.is_user.a[:N_u] = True
+        g.vp.name = g.new_vp("string")
+        for uid, u in user_idx.items():
+            g.vp.name[u] = uid
+        del user_idx
+        g.vp.category = g.new_vp("string")
+        g.vp.subcategory = g.new_vp("string")
+        g.vp.title = g.new_vp("string")
+        g.vp.url = g.new_vp("string")
+        for nid, i in news_idx.items():
+            v = N_u + i
+            g.vp.name[v] = nid
+            if nid in news:
+                (g.vp.category[v], g.vp.subcategory[v], g.vp.title[v],
+                 g.vp.url[v]) = news[nid]
+        # Edges added in chunks, to keep the memory use low
+        step = 10 ** 7
+        for i in range(0, len(src), step):
+            sl = slice(i, i + step)
+            g.add_edge_list(numpy.column_stack((src[sl], N_u + tgt[sl])))
+        del src, tgt
+        g.ep.time = g.new_ep("int64_t")
+        g.ep.time.a = times
+        g.ep.impression = g.new_ep("int")
+        g.ep.impression.a = imps
+        g.ep.split = g.new_ep("int16_t")
+        g.ep.split.a = splt
+        if alt == "impressions":
+            g.ep.clicked = g.new_ep("int16_t")
+            g.ep.clicked.a = clicked
+        yield alt, g