History of wikilens_ratings
The changes to the source code of this entry in the repository, newest first, each with the differences it made to the entry's files. They are also listed on GitLab.
[entries] Add the date when each entry was added, and remove icon_hash beff7c9
The dates are those of the commits that first added the entries' files (2020-08-11 for the entries of the initial commit); renamed entries keep the date of their first name. Changed: none
diff --git a/entries/wikilens_ratings/__init__.py b/entries/wikilens_ratings/__init__.py
index bed2dd1..04e9f67 100644
--- a/entries/wikilens_ratings/__init__.py
+++ b/entries/wikilens_ratings/__init__.py
@@ -48,7 +48,7 @@ The network `links` has the following properties:
tags = ['Social', 'Online', 'Preferences', 'Bipartite', 'Weighted', 'Timestamps', 'Metadata']
url = 'https://grouplens.org/datasets/wikilens/'
citation = [('D. Frankowski, S. K. Lam, S. Sen, F. M. Harper, S. Yilek, M. Cassano, J. Riedl, "Recommenders everywhere: the WikiLens community-maintained recommender system." Proceedings of the 2007 International Symposium on Wikis (WikiSym), 47-60 (2007).', 'https://doi.org/10.1145/1296951.1296957')]
-icon_hash = None
+date_added = '2026-10-27'
upstream_license = None
upstream_prefix = 'https://files.grouplens.org/datasets/wikilens'
_dump = 'wikilens.20080202.tar.gz:wikilens.20080202.data/dump.txt.gz'
[entries] Prefix the properties in the descriptions with (Node), (Edge) or (Graph) bc3dbbd
The lists of properties at the end of the descriptions marked edge properties with "Edge" but left node properties unmarked, or grouped them in "Nodes: `a`, ...; `b`, ..." items, so the kind of a property was often unclear. Every item now starts with (Node), (Edge) or (Graph), followed by the property names, a colon and the explanation, one item per property or group of properties; qualifications (kinds of nodes, networks) follow the names in parentheses. Properties that were only named got a short explanation, and a few items that named the wrong kind of property were corrected.
diff --git a/entries/wikilens_ratings/__init__.py b/entries/wikilens_ratings/__init__.py
index 76768a5..bed2dd1 100644
--- a/entries/wikilens_ratings/__init__.py
+++ b/entries/wikilens_ratings/__init__.py
@@ -31,20 +31,20 @@ The network `links` is directed: nodes are wiki pages and an edge $i \to j$ mean
The network `ratings` has the following properties:
-- `name`: the name of the user page, or of the item (its wiki page name, or its URN if it is not a wiki page; empty if unknown).
-- `is_user`: whether the node is a user (otherwise an item).
-- `urn`: the URN of the item (empty for users).
-- `category`: the categories (item types) of the page of the item or user (e.g. `Movie`, `Book`, `Beer`, `User`).
-- Edge `rating`: the rating, from 0 to 5.
-- Edge `dimension`: the dimension of the rating (0 for all ratings but one, of dimension 5).
-- Edge `item_version`: the version of the item's wiki page that was rated (-1 if unknown).
-- Edge `time`: the time of the rating, in Unix time.
+- (Node) `name`: the name of the user page, or of the item (its wiki page name, or its URN if it is not a wiki page; empty if unknown).
+- (Node) `is_user`: whether the node is a user (otherwise an item).
+- (Node) `urn`: the URN of the item (empty for users).
+- (Node) `category`: the categories (item types) of the page of the item or user (e.g. `Movie`, `Book`, `Beer`, `User`).
+- (Edge) `rating`: the rating, from 0 to 5.
+- (Edge) `dimension`: the dimension of the rating (0 for all ratings but one, of dimension 5).
+- (Edge) `item_version`: the version of the item's wiki page that was rated (-1 if unknown).
+- (Edge) `time`: the time of the rating, in Unix time.
The network `links` has the following properties:
-- `name`: the name of the wiki page (empty if unknown).
-- `category`: the categories of the page.
-- `hits`: the number of visits of the page recorded by the wiki."""
+- (Node) `name`: the name of the wiki page (empty if unknown).
+- (Node) `category`: the categories of the page.
+- (Node) `hits`: the number of visits of the page recorded by the wiki."""
tags = ['Social', 'Online', 'Preferences', 'Bipartite', 'Weighted', 'Timestamps', 'Metadata']
url = 'https://grouplens.org/datasets/wikilens/'
citation = [('D. Frankowski, S. K. Lam, S. Sen, F. M. Harper, S. Yilek, M. Cassano, J. Riedl, "Recommenders everywhere: the WikiLens community-maintained recommender system." Proceedings of the 2007 International Symposium on Wikis (WikiSym), 47-60 (2007).', 'https://doi.org/10.1145/1296951.1296957')]
[entries] Remove unneeded code from the entries 2c69f19
Unneeded "global files" statements, unused imports and variables, f-strings without placeholders, duplicate keys in foodweb_little_rock (keeping the order and the values used), and the second definition of get_lines() in twitter_higgs. Nothing changes in the networks or descriptions. Changed: none
diff --git a/entries/wikilens_ratings/__init__.py b/entries/wikilens_ratings/__init__.py
index b952f30..76768a5 100644
--- a/entries/wikilens_ratings/__init__.py
+++ b/entries/wikilens_ratings/__init__.py
@@ -106,7 +106,6 @@ def _unix_time(s):
@coerce_props()
@annotate()
def parse(alts=None):
- global files
name = __name__.split(".")[-1]
tables = None
for fname, alt, fmt in files:
[entries] Move contact, convote_mentions, douban, email_company, epinions, flixster, football_league_results, hyves, lastfm, libimseti, livemocha, reuters, sa_companies, sister_cities, tripadvisor_ratings, wikilens_ratings and wikipedia_discussions_de to their original sources 8f097de
These entries downloaded KONECT's copies of the data. They now download the original data (live, from pinned Wayback Machine snapshots, or by re-extracting from public dumps), gaining the metadata KONECT dropped (names, identifiers, timestamps, attributes), with descriptions and citations rewritten from the original sources and factual errors fixed (years and periods of contact, email_company, douban, flixster, libimseti, sa_companies; the rated objects of epinions; the sign of convote_mentions; songs merged by title alone in lastfm). New networks: the trust network of epinions, the links of wikilens_ratings, the JSON crawl of tripadvisor_ratings, and current versions of sister_cities and wikipedia_discussions_de next to the ones KONECT used. asu_social_networks only loses its mention of KONECT. Changed: upstream contact convote_mentions douban email_company epinions flixster football_league_results hyves lastfm libimseti livemocha reuters sa_companies sister_cities tripadvisor_ratings wikilens_ratings wikipedia_discussions_de
diff --git a/entries/wikilens_ratings/__init__.py b/entries/wikilens_ratings/__init__.py
index fbba63e..b952f30 100644
--- a/entries/wikilens_ratings/__init__.py
+++ b/entries/wikilens_ratings/__init__.py
@@ -16,46 +16,167 @@
# You should have received a copy of the GNU Affero General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
-
+import gzip
+import zoneinfo
from .. import *
-title = 'WikiLens ratings (2004-2008)'
-description = """A bipartite network of the users of WikiLens and the items they rated, from the data of GroupLens, as distributed by KONECT. WikiLens was a research project of GroupLens (University of Minnesota): a community-maintained recommender system, a wiki-like site on which users could add items of any kind (e.g. books, beers, restaurants or films) and rate them. Each edge connects a user to an item, with the rating (from 0 to 5, in half stars) and its time. The data identify the users and items only by indices.
+title = 'WikiLens ratings and links (2004-2008)'
+description = r"""A bipartite network of the users of WikiLens and the items they rated, and the network of links among its wiki pages, from the database dump that GroupLens (University of Minnesota) published on 2 February 2008. WikiLens was a research project of GroupLens, a generalized, community-maintained recommender system: a wiki (based on PhpWiki) on which users could define item types (e.g. beers, books, restaurants or films) and categories, add items, and rate them and get recommendations. It ran from 2004 and was taken offline in 2009. The dump contains all the data that could be obtained by crawling the site; private data (user accounts, preferences, logs) were erased from it.
+
+The network `ratings` is bipartite: an edge connects a user to an item they rated, for all 26,937 ratings of the dump, with the rating (from 0 to 5, in half stars) and its time. Each user is identified by their user page and each rated object by its URN; most URNs are those of wiki pages (an item and its page), but a few name other objects (e.g. names in a list or titles of papers), and 10 ratings refer to URN ids that are absent from the dump, whose items have no name. Users are also wiki pages, which can themselves be rated as items; such a page is then both a user node and an item node. The times are given in the dump without a time zone, and are taken to be the local time of the University of Minnesota (US Central time, America/Chicago, with daylight saving time).
+
+The network `links` is directed: nodes are wiki pages and an edge $i \to j$ means that page $i$ links to page $j$, from the link cache of the wiki (7,430 links, including self-links). Only the pages with links are included; 9 links refer to page ids absent from the dump, whose nodes have no name. Many of the dump's other pages are spam or empty, and are left out.
-The network has the following properties:
+The network `ratings` has the following properties:
-- `is_ratee`: whether the node is a rated item (otherwise a user).
+- `name`: the name of the user page, or of the item (its wiki page name, or its URN if it is not a wiki page; empty if unknown).
+- `is_user`: whether the node is a user (otherwise an item).
+- `urn`: the URN of the item (empty for users).
+- `category`: the categories (item types) of the page of the item or user (e.g. `Movie`, `Book`, `Beer`, `User`).
- Edge `rating`: the rating, from 0 to 5.
-- Edge `time`: the time of the rating, in Unix time."""
-tags = ['Social', 'Online', 'Preferences', 'Weighted', 'Timestamps']
-url = ['http://konect.cc/networks/wikilens-ratings', 'https://grouplens.org/datasets/wikilens/']
-citation = [('D. Frankowski, S. K. Lam, S. Sen, F. M. Harper, S. Yilek, M. Cassano, J. Riedl, "Recommenders everywhere: the WikiLens community-maintained recommender system." Proceedings of the 2007 International Symposium on Wikis, 47-60 (2007)', 'https://doi.org/10.1145/1296951.1296957'),
- ('J. Kunegis, "KONECT -- The Koblenz Network Collection." Proceedings of the 22nd International Conference on World Wide Web Companion, 1343-1350 (2013)', 'https://doi.org/10.1145/2487788.2488173')]
+- Edge `dimension`: the dimension of the rating (0 for all ratings but one, of dimension 5).
+- Edge `item_version`: the version of the item's wiki page that was rated (-1 if unknown).
+- Edge `time`: the time of the rating, in Unix time.
+
+The network `links` has the following properties:
+
+- `name`: the name of the wiki page (empty if unknown).
+- `category`: the categories of the page.
+- `hits`: the number of visits of the page recorded by the wiki."""
+tags = ['Social', 'Online', 'Preferences', 'Bipartite', 'Weighted', 'Timestamps', 'Metadata']
+url = 'https://grouplens.org/datasets/wikilens/'
+citation = [('D. Frankowski, S. K. Lam, S. Sen, F. M. Harper, S. Yilek, M. Cassano, J. Riedl, "Recommenders everywhere: the WikiLens community-maintained recommender system." Proceedings of the 2007 International Symposium on Wikis (WikiSym), 47-60 (2007).', 'https://doi.org/10.1145/1296951.1296957')]
icon_hash = None
upstream_license = None
-upstream_prefix = 'http://konect.cc/files'
-files = [('download.tsv.wikilens-ratings.tar.bz2', None, 'konect')]
+upstream_prefix = 'https://files.grouplens.org/datasets/wikilens'
+_dump = 'wikilens.20080202.tar.gz:wikilens.20080202.data/dump.txt.gz'
+files = [(_dump, "ratings", None),
+ (_dump, "links", None)]
-def postprocess(alt, g):
- g.ep.rating = g.ep.weight.copy("double")
- del g.ep["weight"]
+# Limit on the number of ratings and links read, for checks (None: all)
+_check_rows = None
def fetch_upstream(force=False):
return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
force)
+# a tuple of n columns of a MySQL dump (numbers, NULL or quoted strings)
+def _row_re(n):
+ return re.compile(rb"\(" + rb",".join([rb"('(?:[^'\\]|\\.)*'|NULL|[^,()']*)"] * n)
+ + rb"\)", re.S)
+
+_COLUMNS = {"category": 2, "link": 2, "page": 4, "page_urn": 2, "rating": 6,
+ "urn": 2}
+
+def _decode(x):
+ # the dump is declared as latin1, but the strings are UTF-8
+ try:
+ return x.decode("utf8")
+ except UnicodeDecodeError:
+ return x.decode("latin1")
+
+def _read_tables(f):
+ "The rows of the tables of the dump that are used, as lists of values."
+ lines = defaultdict(list)
+ for line in f:
+ if line.startswith(b"INSERT INTO `"):
+ t = line.split(b"`")[1].decode()
+ if t in _COLUMNS:
+ lines[t].append(line)
+ tables = {}
+ for t, n in _COLUMNS.items():
+ rows = []
+ for m in sql_rows(iter(lines[t]), _row_re(n)):
+ rows.append([_decode(sql_unescape(x[1:-1])) if x.startswith(b"'")
+ else x.decode() for x in m.groups()])
+ if _check_rows is not None and t in ("rating", "link") and \
+ len(rows) >= _check_rows:
+ break
+ tables[t] = rows
+ return tables
+
+_TZ = zoneinfo.ZoneInfo("America/Chicago")
+
+def _unix_time(s):
+ return int(datetime.strptime(s, "%Y-%m-%d %H:%M:%S").replace(tzinfo=_TZ).timestamp())
+
@cache_network()
@coerce_props()
@annotate()
def parse(alts=None):
global files
name = __name__.split(".")[-1]
+ tables = None
for fname, alt, fmt in files:
if alts is not None and alt not in alts:
continue
- with open_upstream_file(name, fname, "rb") as f:
- g = parse_graph([f], fmt, directed=None)
- postprocess(alt, g)
+ if tables is None:
+ with open_upstream_file(name, fname, "rb") as fz:
+ with gzip.open(fz, "rb") as f:
+ tables = _read_tables(f)
+ pages = {int(i): pname for i, pname, hits, data in tables["page"]}
+ hits = {int(i): int(h) for i, pname, h, data in tables["page"]}
+ cats = defaultdict(list)
+ for item, cat in tables["category"]:
+ cats[int(item)].append(pages.get(int(cat), ""))
+ page_ids = {pname: i for i, pname in pages.items()}
+
+ if alt == "ratings":
+ urns = {int(i): u for i, u in tables["urn"]}
+ # the page of each URN
+ urn_page = {u: p for p, u in tables["page_urn"]}
+ users, items = {}, {}
+ edges = []
+ # dimension, raterpage, rateepage (URN id), ratingvalue,
+ # rateeversion, tstamp
+ for dim, rater, ratee, value, version, tstamp in tables["rating"]:
+ u = users.setdefault(int(rater), len(users))
+ i = items.setdefault(int(ratee), len(items))
+ edges.append((u, i, float(value), int(dim), int(version),
+ _unix_time(tstamp)))
+ nu = len(users)
+ g = Graph(directed=False)
+ g.add_vertex(nu + len(items))
+ g.vp.name = g.new_vp("string")
+ g.vp.is_user = g.new_vp("bool")
+ g.vp.urn = g.new_vp("string")
+ g.vp.category = g.new_vp("vector<string>")
+ for p, v in users.items():
+ g.vp.is_user[v] = True
+ g.vp.name[v] = pages.get(p, "")
+ g.vp.category[v] = cats.get(p, [])
+ for uid, i in items.items():
+ v = g.vertex(nu + i)
+ u = urns.get(uid, "")
+ g.vp.urn[v] = u
+ p = urn_page.get(u)
+ if p is not None:
+ g.vp.name[v] = p
+ g.vp.category[v] = cats.get(page_ids.get(p), [])
+ else:
+ g.vp.name[v] = u
+ g.ep.rating = g.new_ep("double")
+ g.ep.dimension = g.new_ep("int")
+ g.ep.item_version = g.new_ep("int")
+ g.ep.time = g.new_ep("int64_t")
+ g.add_edge_list([(u, nu + i, r, d, ver, t)
+ for u, i, r, d, ver, t in edges],
+ eprops=[g.ep.rating, g.ep.dimension,
+ g.ep.item_version, g.ep.time])
+ else:
+ g = Graph(directed=True)
+ g.vp.page = g.add_edge_list(numpy.array([(int(a), int(b)) for a, b in tables["link"]],
+ dtype="int64").reshape(-1, 2),
+ hashed=True)
+ g.vp.name = g.new_vp("string")
+ g.vp.category = g.new_vp("vector<string>")
+ g.vp.hits = g.new_vp("int64_t")
+ for v in g.vertices():
+ p = g.vp.page[v]
+ g.vp.name[v] = pages.get(p, "")
+ g.vp.category[v] = cats.get(p, [])
+ g.vp.hits[v] = hits.get(p, 0)
+ del g.vp["page"]
yield alt, g
[entries] New entry: wikilens_ratings 794ec3b
diff --git a/entries/wikilens_ratings/__init__.py b/entries/wikilens_ratings/__init__.py
new file mode 100644
index 0000000..fbba63e
--- /dev/null
+++ b/entries/wikilens_ratings/__init__.py
@@ -0,0 +1,61 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2026 Tiago de Paula Peixoto <tiago@skewed.de>
+#
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option) any
+# later version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along
+# with this program. If not, see <http://www.gnu.org/licenses/>.
+
+
+
+from .. import *
+
+
+title = 'WikiLens ratings (2004-2008)'
+description = """A bipartite network of the users of WikiLens and the items they rated, from the data of GroupLens, as distributed by KONECT. WikiLens was a research project of GroupLens (University of Minnesota): a community-maintained recommender system, a wiki-like site on which users could add items of any kind (e.g. books, beers, restaurants or films) and rate them. Each edge connects a user to an item, with the rating (from 0 to 5, in half stars) and its time. The data identify the users and items only by indices.
+
+The network has the following properties:
+
+- `is_ratee`: whether the node is a rated item (otherwise a user).
+- Edge `rating`: the rating, from 0 to 5.
+- Edge `time`: the time of the rating, in Unix time."""
+tags = ['Social', 'Online', 'Preferences', 'Weighted', 'Timestamps']
+url = ['http://konect.cc/networks/wikilens-ratings', 'https://grouplens.org/datasets/wikilens/']
+citation = [('D. Frankowski, S. K. Lam, S. Sen, F. M. Harper, S. Yilek, M. Cassano, J. Riedl, "Recommenders everywhere: the WikiLens community-maintained recommender system." Proceedings of the 2007 International Symposium on Wikis, 47-60 (2007)', 'https://doi.org/10.1145/1296951.1296957'),
+ ('J. Kunegis, "KONECT -- The Koblenz Network Collection." Proceedings of the 22nd International Conference on World Wide Web Companion, 1343-1350 (2013)', 'https://doi.org/10.1145/2487788.2488173')]
+icon_hash = None
+upstream_license = None
+upstream_prefix = 'http://konect.cc/files'
+files = [('download.tsv.wikilens-ratings.tar.bz2', None, 'konect')]
+
+def postprocess(alt, g):
+ g.ep.rating = g.ep.weight.copy("double")
+ del g.ep["weight"]
+
+def fetch_upstream(force=False):
+ return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
+ force)
+
+@cache_network()
+@coerce_props()
+@annotate()
+def parse(alts=None):
+ global files
+ name = __name__.split(".")[-1]
+ for fname, alt, fmt in files:
+ if alts is not None and alt not in alts:
+ continue
+ with open_upstream_file(name, fname, "rb") as f:
+ g = parse_graph([f], fmt, directed=None)
+ postprocess(alt, g)
+ yield alt, g