History of genius_expertise
The changes to the source code of this entry in the repository, newest first, each with the differences it made to the entry's files. They are also listed on GitLab.
[entries] Add the date when each entry was added, and remove icon_hash beff7c9
The dates are those of the commits that first added the entries' files (2020-08-11 for the entries of the initial commit); renamed entries keep the date of their first name. Changed: none
diff --git a/entries/genius_expertise/__init__.py b/entries/genius_expertise/__init__.py
index cfefb6c..7538ed2 100644
--- a/entries/genius_expertise/__init__.py
+++ b/entries/genius_expertise/__init__.py
@@ -53,7 +53,7 @@ tags = ['Social', 'Online', 'Collaboration', 'Multigraph', 'Timestamps', 'Unweig
url = ['https://www.cs.cornell.edu/~arb/data/genius-expertise/',
'https://github.com/cptq/genius-expertise']
citation = [('D. Lim and A. R. Benson, "Expertise and Dynamics within Crowdsourced Musical Knowledge Curation: A Case Study of the Genius Platform." Proceedings of the International AAAI Conference on Web and Social Media (ICWSM) 15, 373-384 (2021).', 'https://doi.org/10.1609/icwsm.v15i1.18068')]
-icon_hash = None
+date_added = '2026-10-27'
upstream_license = None
upstream_prefix = 'gdrive://'
_id = '1SKwubShEWa7CuAuSqTxwRu6_J4ZVkOvD'
[entries] Prefix the properties in the descriptions with (Node), (Edge) or (Graph) bc3dbbd
The lists of properties at the end of the descriptions marked edge properties with "Edge" but left node properties unmarked, or grouped them in "Nodes: `a`, ...; `b`, ..." items, so the kind of a property was often unclear. Every item now starts with (Node), (Edge) or (Graph), followed by the property names, a colon and the explanation, one item per property or group of properties; qualifications (kinds of nodes, networks) follow the names in parentheses. Properties that were only named got a short explanation, and a few items that named the wrong kind of property were corrected.
diff --git a/entries/genius_expertise/__init__.py b/entries/genius_expertise/__init__.py
index ae31f9a..cfefb6c 100644
--- a/entries/genius_expertise/__init__.py
+++ b/entries/genius_expertise/__init__.py
@@ -34,11 +34,21 @@ User names and the names of songs and artists are the identifiers of their pages
The networks have the following properties:
-- `name`: the name of the user, song or artist page; `is_song` (`annotations`, `artist_songs`), whether the node is a song.
-- Users (`follows`, `annotations`): `iq`, the user's IQ (Genius reputation points); `roles`, the roles of the user (e.g. "Editor", "Contributor"); `annotations_num`, `suggestions_num`, `transcriptions_num`, `questions_num`, `answers_num` and `pyongs_num`, the numbers of annotations, suggestions, transcriptions, questions, answers and pyongs (shares) of the user; `followers_num`, `following_num`, the numbers of followers and of followed users. Users without information have the value -1.
-- Songs (`annotations`, `artist_songs`): `title`, `primary_artist`, `release_date` (as given, e.g. "July 31, 2012"), `views`, `pyongs`, `contributors` (the number of contributors), `has_bio` (whether the song has a description), `tags` (genre tags). Songs without information have empty values and -1.
-- Artists (`artist_songs`): `iq`, `followers`, `roles`.
-- `annotations`, edges: `time`, the time of the edit (or of the annotation), in Unix time; `annotation`, the numeric identifier of the annotation; `type`, the type of the annotation ("reviewed", "unreviewed" or "verified"); `votes`, the votes of the annotation."""
+- (Node) `name`: the name of the user, song or artist page.
+- (Node) `is_song` (`annotations`, `artist_songs`): whether the node is a song.
+- (Node) `iq` (users, in `follows` and `annotations`): the user's IQ (Genius reputation points); -1 for users without information.
+- (Node) `roles` (users, in `follows` and `annotations`): the roles of the user (e.g. "Editor", "Contributor").
+- (Node) `annotations_num`, `suggestions_num`, `transcriptions_num`, `questions_num`, `answers_num`, `pyongs_num` (users, in `follows` and `annotations`): the numbers of annotations, suggestions, transcriptions, questions, answers and pyongs (shares) of the user; -1 for users without information.
+- (Node) `followers_num`, `following_num` (users, in `follows` and `annotations`): the numbers of followers and of followed users; -1 for users without information.
+- (Node) `title`, `primary_artist`, `release_date`, `views`, `pyongs` (songs, in `annotations` and `artist_songs`): the title, primary artist, release date (as given, e.g. "July 31, 2012"), views and pyongs of the song; empty or -1 for songs without information.
+- (Node) `contributors` (songs, in `annotations` and `artist_songs`): the number of contributors; -1 for songs without information.
+- (Node) `has_bio` (songs, in `annotations` and `artist_songs`): whether the song has a description.
+- (Node) `tags` (songs, in `annotations` and `artist_songs`): genre tags; empty for songs without information.
+- (Node) `iq`, `followers`, `roles` (artists, in `artist_songs`): the IQ, followers and roles of the artist.
+- (Edge) `time` (`annotations`): the time of the edit (or of the annotation), in Unix time.
+- (Edge) `annotation` (`annotations`): the numeric identifier of the annotation.
+- (Edge) `type` (`annotations`): the type of the annotation ("reviewed", "unreviewed" or "verified").
+- (Edge) `votes` (`annotations`): the votes of the annotation."""
tags = ['Social', 'Online', 'Collaboration', 'Multigraph', 'Timestamps', 'Unweighted', 'Metadata']
url = ['https://www.cs.cornell.edu/~arb/data/genius-expertise/',
'https://github.com/cptq/genius-expertise']
[entries] New entry: genius_expertise 49507ec
diff --git a/entries/genius_expertise/__init__.py b/entries/genius_expertise/__init__.py
new file mode 100644
index 0000000..ae31f9a
--- /dev/null
+++ b/entries/genius_expertise/__init__.py
@@ -0,0 +1,215 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2020-2026 Tiago de Paula Peixoto <tiago@skewed.de>
+#
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option) any
+# later version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along
+# with this program. If not, see <http://www.gnu.org/licenses/>.
+
+
+from .. import *
+
+import calendar
+import csv
+import json
+
+title = "Genius users, annotations and artists (2019-2020)"
+description = r"""Users, songs and artists of the lyrics annotation website Genius (genius.com), crawled between September 2019 and January 2020 by D. Lim and A. R. Benson to study the expertise and dynamics of crowdsourced knowledge curation. On Genius, users write annotations explaining passages of song lyrics; annotations are edited collaboratively and reviewed by moderators, and verified annotations are written by the artists themselves. Users can follow each other. There are three networks:
+
+- `follows`: a directed network of users, where an edge $i \to j$ means that user $i$ follows user $j$ (users who appear only in the user information file are included as isolated nodes; 1,219 users follow themselves, as given).
+- `annotations`: a bipartite multigraph of users and songs, where each edge is an edit of an annotation of the song's lyrics by the user. Reviewed annotations come with their edit histories, where each edit is an edge; unreviewed annotations have only their author, and verified annotations only the artist account that wrote them, each of which is a single edge with the time of the annotation.
+- `artist_songs`: a bipartite network of artists (as given by their Genius artist pages) and the songs they contributed to (e.g. as performers, writers or producers).
+
+User names and the names of songs and artists are the identifiers of their pages on the site (e.g. `https://genius.com/<name>`); artist and user accounts are distinct, so that artists in `artist_songs` are not identified with users. The user information file lists 2,513 users more than once, and the artist information file 2,543 artists, in part with different values; the first entry is used (for the artists, with the songs of all their entries). The times are given without time zone, and are interpreted as UTC. Left out are the texts of the annotations and their edits, the lyrics of the songs, the contribution shares of the users to each reviewed annotation computed by Genius, and the users who accepted the annotations.
+
+The networks have the following properties:
+
+- `name`: the name of the user, song or artist page; `is_song` (`annotations`, `artist_songs`), whether the node is a song.
+- Users (`follows`, `annotations`): `iq`, the user's IQ (Genius reputation points); `roles`, the roles of the user (e.g. "Editor", "Contributor"); `annotations_num`, `suggestions_num`, `transcriptions_num`, `questions_num`, `answers_num` and `pyongs_num`, the numbers of annotations, suggestions, transcriptions, questions, answers and pyongs (shares) of the user; `followers_num`, `following_num`, the numbers of followers and of followed users. Users without information have the value -1.
+- Songs (`annotations`, `artist_songs`): `title`, `primary_artist`, `release_date` (as given, e.g. "July 31, 2012"), `views`, `pyongs`, `contributors` (the number of contributors), `has_bio` (whether the song has a description), `tags` (genre tags). Songs without information have empty values and -1.
+- Artists (`artist_songs`): `iq`, `followers`, `roles`.
+- `annotations`, edges: `time`, the time of the edit (or of the annotation), in Unix time; `annotation`, the numeric identifier of the annotation; `type`, the type of the annotation ("reviewed", "unreviewed" or "verified"); `votes`, the votes of the annotation."""
+tags = ['Social', 'Online', 'Collaboration', 'Multigraph', 'Timestamps', 'Unweighted', 'Metadata']
+url = ['https://www.cs.cornell.edu/~arb/data/genius-expertise/',
+ 'https://github.com/cptq/genius-expertise']
+citation = [('D. Lim and A. R. Benson, "Expertise and Dynamics within Crowdsourced Musical Knowledge Curation: A Case Study of the Genius Platform." Proceedings of the International AAAI Conference on Web and Social Media (ICWSM) 15, 373-384 (2021).', 'https://doi.org/10.1609/icwsm.v15i1.18068')]
+icon_hash = None
+upstream_license = None
+upstream_prefix = 'gdrive://'
+_id = '1SKwubShEWa7CuAuSqTxwRu6_J4ZVkOvD'
+_d = f'{_id}:zip:genius-expertise'
+files = [((f'{_d}/follows.csv', f'{_d}/user_info.json'), 'follows', None),
+ ((f'{_d}/annotation_info.json', f'{_d}/user_info.json',
+ f'{_d}/song_info.json'), 'annotations', None),
+ ((f'{_d}/artist_info.json', f'{_d}/song_info.json'), 'artist_songs', None)]
+
+def fetch_upstream(force=False):
+ return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
+ force)
+
+def _jl(name, fname):
+ with open_upstream_file(name, fname, "rb") as f:
+ for line in f:
+ if line.strip():
+ yield json.loads(line)
+
+_months = {m: i + 1 for i, m in enumerate(["Jan", "Feb", "Mar", "Apr", "May", "Jun",
+ "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"])}
+
+def _time(x):
+ "Unix time of e.g. 'Aug 5, 2016 4:44:24 PM' (parsed without the locale)."
+ mon, day, year, hms, ampm = x.replace(",", "").split()
+ h, m, s = map(int, hms.split(":"))
+ assert ampm in ("AM", "PM") and 1 <= h <= 12
+ h = h % 12 + (12 if ampm == "PM" else 0)
+ return calendar.timegm((int(year), _months[mon], int(day), h, m, s))
+
+_ustats = ["Annotations", "Suggestions", "Transcriptions", "Questions",
+ "Answers", "Pyongs"]
+
+def _user_props(g, vs, name, fname):
+ "Set the user properties of the vertices `vs` (a dict name -> vertex)."
+ for s in _ustats + ["iq", "followers_num", "following_num"]:
+ p = s.lower() + "_num" if s in _ustats else s
+ g.vp[p] = g.new_vp("int64_t", val=-1)
+ g.vp.roles = g.new_vp("vector<string>")
+ seen = set()
+ for d in _jl(name, fname):
+ u = d["url_name"]
+ if u in seen or u not in vs: # duplicated users: keep the first
+ continue
+ seen.add(u)
+ v = vs[u]
+ g.vp.iq[v] = d["iq"]
+ for s in _ustats:
+ g.vp[s.lower() + "_num"][v] = d["stats"][s]
+ g.vp.followers_num[v] = d["followers_num"]
+ g.vp.following_num[v] = d["following_num"]
+ g.vp.roles[v] = d["roles"]
+
+def _song_props(g, vs, name, fname):
+ "Set the song properties of the vertices `vs` (a dict name -> vertex)."
+ for p in ["title", "primary_artist", "release_date"]:
+ g.vp[p] = g.new_vp("string")
+ for p in ["views", "pyongs", "contributors"]:
+ g.vp[p] = g.new_vp("int64_t", val=-1)
+ g.vp.has_bio = g.new_vp("bool")
+ g.vp.tags = g.new_vp("vector<string>")
+ for d in _jl(name, fname):
+ v = vs.get(d["url_name"])
+ if v is None:
+ continue
+ for p in ["title", "primary_artist", "release_date"]:
+ g.vp[p][v] = d[p] or ""
+ for p in ["views", "pyongs", "contributors"]:
+ if d[p] is not None:
+ g.vp[p][v] = d[p]
+ g.vp.has_bio[v] = bool(d["has_bio"])
+ g.vp.tags[v] = d["tags"] or []
+
+@cache_network()
+@coerce_props()
+@annotate()
+def parse(alts=None):
+ name = __name__.split(".")[-1]
+ for fnames, alt, fmt in files:
+ if alts is not None and alt not in alts:
+ continue
+ if alt == "follows":
+ g = Graph(directed=True)
+ with open_upstream_file(name, fnames[0], "r") as f:
+ rows = csv.reader(f)
+ assert next(rows) == ["follower", "following"]
+ g.vp.name = g.add_edge_list(rows, hashed=True, hash_type="string")
+ vs = {g.vp.name[v]: v for v in g.vertices()}
+ for d in _jl(name, fnames[1]):
+ u = d["url_name"]
+ if u not in vs:
+ v = g.add_vertex()
+ g.vp.name[v] = u
+ vs[u] = v
+ _user_props(g, vs, name, fnames[1])
+ elif alt == "annotations":
+ E = []
+ for d in _jl(name, fnames[0]):
+ aid = int(d["url"].removeprefix("https://genius.com/"))
+ song = d["song"]
+ if d["type"] == "reviewed":
+ for e in d["edits_lst"]:
+ u = e["name"].removeprefix("https://genius.com/")
+ E.append((u, song, _time(e["time"]), aid, "reviewed",
+ d["votes"]))
+ else:
+ u = d["user"] if d["type"] == "unreviewed" else d["artist"]
+ E.append((u, song, _time(d["time"]), aid, d["type"],
+ d["votes"]))
+ users = sorted(set(e[0] for e in E))
+ songs = sorted(set(e[1] for e in E))
+ vs = {u: i for i, u in enumerate(users)}
+ ss = {s: len(users) + i for i, s in enumerate(songs)}
+ g = Graph(directed=False)
+ g.add_vertex(len(users) + len(songs))
+ g.vp.name = g.new_vp("string", vals=users + songs)
+ g.vp.is_song = g.new_vp("bool")
+ g.vp.is_song.a[len(users):] = True
+ g.ep.time = g.new_ep("int64_t")
+ g.ep.annotation = g.new_ep("int64_t")
+ g.ep.type = g.new_ep("string")
+ g.ep.votes = g.new_ep("int64_t")
+ g.add_edge_list(((vs[u], ss[s], t, a, k, n) for u, s, t, a, k, n in E),
+ eprops=[g.ep.time, g.ep.annotation, g.ep.type,
+ g.ep.votes])
+ del E
+ _user_props(g, vs, name, fnames[1])
+ _song_props(g, ss, name, fnames[2])
+ else:
+ # artists listed more than once: the first entry is used for the
+ # properties, and the songs of all entries are merged
+ arts = []
+ aidx = {}
+ asongs = []
+ for d in _jl(name, fnames[0]):
+ i = aidx.get(d["url_name"])
+ if i is None:
+ aidx[d["url_name"]] = i = len(arts)
+ arts.append(d)
+ asongs.append({})
+ for s in d["songs"] or []:
+ asongs[i][s] = None
+ g = Graph(directed=False)
+ g.add_vertex(len(arts))
+ g.vp.name = g.new_vp("string", vals=[d["url_name"] for d in arts])
+ g.vp.iq = g.new_vp("int64_t", vals=[d["iq"] for d in arts])
+ g.vp.followers = g.new_vp("int64_t", vals=[d["followers"] for d in arts])
+ g.vp.roles = g.new_vp("vector<string>")
+ for v, d in enumerate(arts):
+ g.vp.roles[v] = d["roles"]
+ ss = {}
+ E = []
+ for v, songs in enumerate(asongs):
+ for s in songs:
+ if s not in ss:
+ ss[s] = len(arts) + len(ss)
+ E.append((v, ss[s]))
+ del arts, asongs
+ g.add_vertex(len(ss))
+ g.add_edge_list(E)
+ del E
+ for s, v in ss.items():
+ g.vp.name[v] = s
+ g.vp.is_song = g.new_vp("bool")
+ g.vp.is_song.a[g.num_vertices() - len(ss):] = True
+ for p in ["iq", "followers"]:
+ g.vp[p].a[g.num_vertices() - len(ss):] = -1
+ _song_props(g, ss, name, fnames[1])
+ yield alt, g