History of movielens_100k
The changes to the source code of this entry in the repository, newest first, each with the differences it made to the entry's files. They are also listed on GitLab.
[entries] Restrict the entries whose upstream terms forbid redistribution ddb6c8c
movielens_100k (the MovieLens license forbids redistribution without permission), and yelp_reviews and yelp_restaurants (the Yelp Dataset terms forbid sharing the data with third parties) are now restricted: their files are available only with an access token, as their descriptions now say. yelp_reviews gets the Yelp Dataset terms as its license.
diff --git a/entries/movielens_100k/__init__.py b/entries/movielens_100k/__init__.py
index 6f279cd..2929ca0 100644
--- a/entries/movielens_100k/__init__.py
+++ b/entries/movielens_100k/__init__.py
@@ -26,12 +26,15 @@ description = """A bipartite network of the tags that users of the online movie
The network has the following properties:
- Nodes: `is_tag`, whether the node is a tag (otherwise it is a movie); `name`, the tag string, or the title of the movie (including its year of release, as entered in MovieLens); `movie_id`, the MovieLens movie id (-1 for tags); `genres`, the list of genres of the movie.
-- Edges: `user`, the anonymized id of the user who applied the tag; `time`, the time when the tag was applied, as Unix time."""
+- Edges: `user`, the anonymized id of the user who applied the tag; `time`, the time when the tag was applied, as Unix time.
+
+The files of this entry are available only with an access token, since the MovieLens license forbids redistributing the data without permission."""
tags = ['Informational', 'Folksonomy', 'Unweighted', 'Multigraph', 'Timestamps', 'Metadata']
url = 'https://grouplens.org/datasets/movielens/10m/'
citation = [('F. M. Harper, J. A. Konstan, "The MovieLens Datasets: History and Context." ACM Transactions on Interactive Intelligent Systems 5(4), 19 (2015)', 'https://doi.org/10.1145/2827872')]
icon_hash = '5759d552bd3e93b53c695600'
upstream_license = ("MovieLens usage license (research use; no redistribution without permission)", "https://files.grouplens.org/datasets/movielens/ml-10m-README.html")
+restricted = True
upstream_prefix = 'https://files.grouplens.org/datasets/movielens'
files = [(('ml-10m.zip:ml-10M100K/tags.dat', 'ml-10m.zip:ml-10M100K/movies.dat'),
None, None)]
[entries] Move more KONECT-derived entries to their original sources bf9dc0e
The entries now download the data from where KONECT obtained them (live, or via the Wayback Machine), using the same release as KONECT where it is still available, and gain the metadata KONECT dropped; descriptions and citations are rewritten from the original sources: - Linton Freeman's UCINET data page: bible_nouns, bison, cattle, hens, highschool, kangaroo, macaques, moreno_sheep, physician_trust (the three relations and the physicians' attributes), rhesus_monkey, train_terrorists, windsurfers (now the networks `interaction` and `closeness`), college_freshmen, crime, interactome_yeast (from the primary source), moreno_taro (undirected), residence_hall, zebras - other pages, archived: contiguous_usa, advogato, google, github, digg_reply, twitter, bookcrossing, dbtropes_feature (without KONECT's spurious "ItemName" tropes) - movielens_100k (in fact the MovieLens 10M tag assignments) - dblp release 2017-05-02: dblp_author_paper, dblp_cite, dblp_coauthor - DBpedia 3.9: dbpedia_country, dbpedia_genre, dbpedia_location, dbpedia_occupation, dbpedia_producer, dbpedia_recordlabel, dbpedia_starring, dbpedia_team, dbpedia_writer, dbpedia_all, dbpedia_link - Discogs releases dump of October 2015: discogs_affiliation, discogs_genre and discogs_label (now with genre and style networks)
diff --git a/entries/movielens_100k/__init__.py b/entries/movielens_100k/__init__.py
index 990cb0e..6f279cd 100644
--- a/entries/movielens_100k/__init__.py
+++ b/entries/movielens_100k/__init__.py
@@ -18,19 +18,53 @@
from .. import *
-title = "MovieLens 100K (1998)"
-description = """Three bipartite networks that make up the MovieLens 100K Dataset, a stable benchmark dataset of 100,000 ratings from 1000 users on 1700 movies. These data capture the tag-movie, user-movie, and user-tag networks. (Also available from MovieLens are 1M, 10M and 20M folksonomy datasets.)[^icon]
-[^icon]: Description obtained from the [ICON](https://icon.colorado.edu) project."""
-tags = ['Informational', 'Folksonomy', 'Unweighted', 'Multigraph', 'Timestamps']
-url = 'http://konect.cc/networks/movielens-10m_ti'
-citation = [('GroupLens Research, "MovieLens data sets."', 'http://grouplens.org/datasets/movielens/')]
+import hashlib
+
+title = "MovieLens 10M tag assignments (2005-2009)"
+description = """A bipartite network of the tags that users of the online movie recommender service MovieLens applied to movies, from the MovieLens 10M data set released by GroupLens Research (University of Minnesota) in January 2009. MovieLens users can annotate movies with free-text tags, typically a single word or short phrase whose meaning is determined by the user. The data set contains 95,580 tag applications made between December 2005 and January 2009 by 4,009 users, who were selected at random (all selected users had rated at least 20 movies) and anonymized. Each node is a tag (a distinct tag string, case-sensitive) or a movie, and each edge is one application of the tag to the movie by some user, so that the same tag applied to the same movie by several users appears as multiple edges. Only movies that received at least one tag are included. (Despite the entry's name, these data are not from the MovieLens 100K data set, which contains no tags.)
+
+The network has the following properties:
+
+- Nodes: `is_tag`, whether the node is a tag (otherwise it is a movie); `name`, the tag string, or the title of the movie (including its year of release, as entered in MovieLens); `movie_id`, the MovieLens movie id (-1 for tags); `genres`, the list of genres of the movie.
+- Edges: `user`, the anonymized id of the user who applied the tag; `time`, the time when the tag was applied, as Unix time."""
+tags = ['Informational', 'Folksonomy', 'Unweighted', 'Multigraph', 'Timestamps', 'Metadata']
+url = 'https://grouplens.org/datasets/movielens/10m/'
+citation = [('F. M. Harper, J. A. Konstan, "The MovieLens Datasets: History and Context." ACM Transactions on Interactive Intelligent Systems 5(4), 19 (2015)', 'https://doi.org/10.1145/2827872')]
icon_hash = '5759d552bd3e93b53c695600'
-upstream_prefix = 'http://konect.cc/files'
-files = [('download.tsv.movielens-10m_ti.tar.bz2', None, 'konect')]
+upstream_license = ("MovieLens usage license (research use; no redistribution without permission)", "https://files.grouplens.org/datasets/movielens/ml-10m-README.html")
+upstream_prefix = 'https://files.grouplens.org/datasets/movielens'
+files = [(('ml-10m.zip:ml-10M100K/tags.dat', 'ml-10m.zip:ml-10M100K/movies.dat'),
+ None, None)]
+
+# The server of files.grouplens.org does not send a certificate chain that
+# can be verified against the usual trust stores (it ends in an emSign root
+# that is missing from them). The file is then obtained from a pinned
+# Internet Archive snapshot instead; in both cases its content is checked
+# against the MD5 sum published by GroupLens.
+archived_url = 'https://web.archive.org/web/20220331225744id_/https://files.grouplens.org/datasets/movielens/ml-10m.zip'
+md5sum = 'ce571fd55effeba0271552578f2648bd'
def fetch_upstream(force=False):
- return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
- force)
+ name = __name__.split(".")[-1]
+ if not force:
+ try:
+ with open_upstream_file(name, "ml-10m.zip", "rb"):
+ return
+ except (FileNotFoundError, IOError):
+ pass
+ def download(url):
+ with open_url(url) as f:
+ return f.read()
+ try:
+ data = download(f"{upstream_prefix}/ml-10m.zip")
+ except urllib.error.URLError as e:
+ if not isinstance(e.reason, ssl.SSLCertVerificationError):
+ raise
+ data = retry_download(download, archived_url, tries=3)
+ if hashlib.md5(data).hexdigest() != md5sum:
+ raise RuntimeError("unexpected content for ml-10m.zip")
+ with open_upstream_file(name, "ml-10m.zip", "wb") as output:
+ output.write(data)
@cache_network()
@coerce_props()
@@ -41,9 +75,38 @@ def parse(alts=None):
for fnames, alt, fmt in files:
if alts is not None and alt not in alts:
continue
- if isinstance(fnames, str):
- fnames = [fnames]
- with ExitStack() as stack:
- fs = [stack.enter_context(open_upstream_file(name, fn, "rb")) for fn in fnames]
- g = parse_graph(fs, fmt, directed=False)
+ with open_upstream_file(name, fnames[0], "r") as f:
+ # UserID::MovieID::Tag::Timestamp
+ rows = [line.rstrip("\n").split("::") for line in f if line.strip()]
+ movies = {}
+ with open_upstream_file(name, fnames[1], "r") as f:
+ # MovieID::Title::Genres
+ for line in f:
+ if not line.strip():
+ continue
+ mid, mtitle, genres = line.rstrip("\n").split("::")
+ movies[int(mid)] = (mtitle, genres.split("|"))
+ g = Graph(directed=False)
+ g.vp.is_tag = g.new_vp("bool")
+ g.vp.name = g.new_vp("string")
+ g.vp.movie_id = g.new_vp("int")
+ g.vp.genres = g.new_vp("vector<string>")
+ g.ep.user = g.new_ep("int")
+ g.ep.time = g.new_ep("int64_t")
+ tag_v = {}
+ movie_v = {}
+ elist = []
+ for user, mid, tag, t in rows:
+ if tag not in tag_v:
+ v = tag_v[tag] = g.add_vertex()
+ g.vp.is_tag[v] = True
+ g.vp.name[v] = tag
+ g.vp.movie_id[v] = -1
+ mid = int(mid)
+ if mid not in movie_v:
+ v = movie_v[mid] = g.add_vertex()
+ g.vp.movie_id[v] = mid
+ g.vp.name[v], g.vp.genres[v] = movies[mid]
+ elist.append((int(tag_v[tag]), int(movie_v[mid]), int(user), int(t)))
+ g.add_edge_list(elist, eprops=[g.ep.user, g.ep.time])
yield alt, g
Update copyright information 73a4f0d
diff --git a/entries/movielens_100k/__init__.py b/entries/movielens_100k/__init__.py
index 559e153..990cb0e 100644
--- a/entries/movielens_100k/__init__.py
+++ b/entries/movielens_100k/__init__.py
@@ -1,7 +1,7 @@
#! /usr/bin/env python
# -*- coding: utf-8 -*-
#
-# Copyright (C) 2020-2024 Tiago de Paula Peixoto <tiago@skewed.de>
+# Copyright (C) 2020-2026 Tiago de Paula Peixoto <tiago@skewed.de>
#
# This program is free software: you can redistribute it and/or modify it under
# the terms of the GNU Affero General Public License as published by the Free
Update copyright information 3d288ea
diff --git a/entries/movielens_100k/__init__.py b/entries/movielens_100k/__init__.py
index 17240b5..559e153 100644
--- a/entries/movielens_100k/__init__.py
+++ b/entries/movielens_100k/__init__.py
@@ -1,7 +1,7 @@
#! /usr/bin/env python
# -*- coding: utf-8 -*-
#
-# Copyright (C) 2020-2023 Tiago de Paula Peixoto <tiago@skewed.de>
+# Copyright (C) 2020-2024 Tiago de Paula Peixoto <tiago@skewed.de>
#
# This program is free software: you can redistribute it and/or modify it under
# the terms of the GNU Affero General Public License as published by the Free
Update copyright information 8d005ae
diff --git a/entries/movielens_100k/__init__.py b/entries/movielens_100k/__init__.py
index 5ea51dc..17240b5 100644
--- a/entries/movielens_100k/__init__.py
+++ b/entries/movielens_100k/__init__.py
@@ -1,7 +1,7 @@
#! /usr/bin/env python
# -*- coding: utf-8 -*-
#
-# Copyright (C) 2021 Tiago de Paula Peixoto <tiago@skewed.de>
+# Copyright (C) 2020-2023 Tiago de Paula Peixoto <tiago@skewed.de>
#
# This program is free software: you can redistribute it and/or modify it under
# the terms of the GNU Affero General Public License as published by the Free
Update copyright info db2efcc
diff --git a/entries/movielens_100k/__init__.py b/entries/movielens_100k/__init__.py
index a346c82..5ea51dc 100644
--- a/entries/movielens_100k/__init__.py
+++ b/entries/movielens_100k/__init__.py
@@ -1,7 +1,7 @@
#! /usr/bin/env python
# -*- coding: utf-8 -*-
#
-# Copyright (C) 2020 Tiago de Paula Peixoto <tiago@skewed.de>
+# Copyright (C) 2021 Tiago de Paula Peixoto <tiago@skewed.de>
#
# This program is free software: you can redistribute it and/or modify it under
# the terms of the GNU Affero General Public License as published by the Free
Un-orphan KONECT datasets 894eba5
KONECT has been brought back from the dead for the time being.
diff --git a/entries/movielens_100k/__init__.py b/entries/movielens_100k/__init__.py
index 532545c..a346c82 100644
--- a/entries/movielens_100k/__init__.py
+++ b/entries/movielens_100k/__init__.py
@@ -26,7 +26,6 @@ url = 'http://konect.cc/networks/movielens-10m_ti'
citation = [('GroupLens Research, "MovieLens data sets."', 'http://grouplens.org/datasets/movielens/')]
icon_hash = '5759d552bd3e93b53c695600'
upstream_prefix = 'http://konect.cc/files'
-orphan = True
files = [('download.tsv.movielens-10m_ti.tar.bz2', None, 'konect')]
def fetch_upstream(force=False):
Update citations to use DOIs e878874
diff --git a/entries/movielens_100k/__init__.py b/entries/movielens_100k/__init__.py
index 55265f3..532545c 100644
--- a/entries/movielens_100k/__init__.py
+++ b/entries/movielens_100k/__init__.py
@@ -23,7 +23,7 @@ description = """Three bipartite networks that make up the MovieLens 100K Datase
[^icon]: Description obtained from the [ICON](https://icon.colorado.edu) project."""
tags = ['Informational', 'Folksonomy', 'Unweighted', 'Multigraph', 'Timestamps']
url = 'http://konect.cc/networks/movielens-10m_ti'
-citation = [('GroupLens Research, "MovieLens data sets." http://grouplens.org/datasets/movielens/ (2006)', 'http://grouplens.org/datasets/movielens/')]
+citation = [('GroupLens Research, "MovieLens data sets."', 'http://grouplens.org/datasets/movielens/')]
icon_hash = '5759d552bd3e93b53c695600'
upstream_prefix = 'http://konect.cc/files'
orphan = True
Add support for 'orphan' datasets and tag konect entries e330b80
diff --git a/entries/movielens_100k/__init__.py b/entries/movielens_100k/__init__.py
index 03429f3..55265f3 100644
--- a/entries/movielens_100k/__init__.py
+++ b/entries/movielens_100k/__init__.py
@@ -26,6 +26,7 @@ url = 'http://konect.cc/networks/movielens-10m_ti'
citation = [('GroupLens Research, "MovieLens data sets." http://grouplens.org/datasets/movielens/ (2006)', 'http://grouplens.org/datasets/movielens/')]
icon_hash = '5759d552bd3e93b53c695600'
upstream_prefix = 'http://konect.cc/files'
+orphan = True
files = [('download.tsv.movielens-10m_ti.tar.bz2', None, 'konect')]
def fetch_upstream(force=False):
Change license from GPLv3 to AGPLv3 d754be5
diff --git a/entries/movielens_100k/__init__.py b/entries/movielens_100k/__init__.py
index 4065496..03429f3 100644
--- a/entries/movielens_100k/__init__.py
+++ b/entries/movielens_100k/__init__.py
@@ -3,18 +3,18 @@
#
# Copyright (C) 2020 Tiago de Paula Peixoto <tiago@skewed.de>
#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option) any
+# later version.
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
+# details.
#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
+# You should have received a copy of the GNU Affero General Public License along
+# with this program. If not, see <http://www.gnu.org/licenses/>.
from .. import *
Add explicit attribution to descriptions obtained from ICON 1645314
diff --git a/entries/movielens_100k/__init__.py b/entries/movielens_100k/__init__.py
index 8a3fc85..4065496 100644
--- a/entries/movielens_100k/__init__.py
+++ b/entries/movielens_100k/__init__.py
@@ -19,7 +19,8 @@
from .. import *
title = "MovieLens 100K (1998)"
-description = "Three bipartite networks that make up the MovieLens 100K Dataset, a stable benchmark dataset of 100,000 ratings from 1000 users on 1700 movies. These data capture the tag-movie, user-movie, and user-tag networks. (Also available from MovieLens are 1M, 10M and 20M folksonomy datasets.)"
+description = """Three bipartite networks that make up the MovieLens 100K Dataset, a stable benchmark dataset of 100,000 ratings from 1000 users on 1700 movies. These data capture the tag-movie, user-movie, and user-tag networks. (Also available from MovieLens are 1M, 10M and 20M folksonomy datasets.)[^icon]
+[^icon]: Description obtained from the [ICON](https://icon.colorado.edu) project."""
tags = ['Informational', 'Folksonomy', 'Unweighted', 'Multigraph', 'Timestamps']
url = 'http://konect.cc/networks/movielens-10m_ti'
citation = [('GroupLens Research, "MovieLens data sets." http://grouplens.org/datasets/movielens/ (2006)', 'http://grouplens.org/datasets/movielens/')]
Initial commit cc85872
diff --git a/entries/movielens_100k/__init__.py b/entries/movielens_100k/__init__.py
new file mode 100644
index 0000000..8a3fc85
--- /dev/null
+++ b/entries/movielens_100k/__init__.py
@@ -0,0 +1,48 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2020 Tiago de Paula Peixoto <tiago@skewed.de>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+from .. import *
+
+title = "MovieLens 100K (1998)"
+description = "Three bipartite networks that make up the MovieLens 100K Dataset, a stable benchmark dataset of 100,000 ratings from 1000 users on 1700 movies. These data capture the tag-movie, user-movie, and user-tag networks. (Also available from MovieLens are 1M, 10M and 20M folksonomy datasets.)"
+tags = ['Informational', 'Folksonomy', 'Unweighted', 'Multigraph', 'Timestamps']
+url = 'http://konect.cc/networks/movielens-10m_ti'
+citation = [('GroupLens Research, "MovieLens data sets." http://grouplens.org/datasets/movielens/ (2006)', 'http://grouplens.org/datasets/movielens/')]
+icon_hash = '5759d552bd3e93b53c695600'
+upstream_prefix = 'http://konect.cc/files'
+files = [('download.tsv.movielens-10m_ti.tar.bz2', None, 'konect')]
+
+def fetch_upstream(force=False):
+ return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
+ force)
+
+@cache_network()
+@coerce_props()
+@annotate()
+def parse(alts=None):
+ global files
+ name = __name__.split(".")[-1]
+ for fnames, alt, fmt in files:
+ if alts is not None and alt not in alts:
+ continue
+ if isinstance(fnames, str):
+ fnames = [fnames]
+ with ExitStack() as stack:
+ fs = [stack.enter_context(open_upstream_file(name, fn, "rb")) for fn in fnames]
+ g = parse_graph(fs, fmt, directed=False)
+ yield alt, g