History of tolokers
The changes to the source code of this entry in the repository, newest first, each with the differences it made to the entry's files. They are also listed on GitLab.
[entries] Add the date when each entry was added, and remove icon_hash beff7c9
The dates are those of the commits that first added the entries' files (2020-08-11 for the entries of the initial commit); renamed entries keep the date of their first name. Changed: none
diff --git a/entries/tolokers/__init__.py b/entries/tolokers/__init__.py
index 9fc1b72..ea58762 100644
--- a/entries/tolokers/__init__.py
+++ b/entries/tolokers/__init__.py
@@ -34,7 +34,7 @@ tags = ['Social', 'Online', 'Collaboration', 'Unweighted', 'Projection', 'Metada
url = 'https://github.com/Toloka/TolokerGraph'
citation = [('D. Likhobaba, N. Pavlichenko, D. Ustalov, "Toloker Graph: Interaction of Crowd Annotators." Zenodo (2023)', 'https://doi.org/10.5281/zenodo.7620795'),
('O. Platonov, D. Kuznedelev, M. Diskin, A. Babenko, L. Prokhorenkova, "A critical look at the evaluation of GNNs under heterophily: Are we really making progress?" International Conference on Learning Representations (ICLR) (2023)', 'https://doi.org/10.48550/arXiv.2302.11640')]
-icon_hash = None
+date_added = '2026-10-27'
upstream_license = ('CC BY 4.0', 'https://creativecommons.org/licenses/by/4.0/')
upstream_prefix = 'https://raw.githubusercontent.com/Toloka/TolokerGraph/04745e8f1037ea39fb226537bdc6ae6bb7bf6619'
files = [(('edges.tsv', 'nodes.tsv'), None, None)]
[entries] Prefix the properties in the descriptions with (Node), (Edge) or (Graph) bc3dbbd
The lists of properties at the end of the descriptions marked edge properties with "Edge" but left node properties unmarked, or grouped them in "Nodes: `a`, ...; `b`, ..." items, so the kind of a property was often unclear. Every item now starts with (Node), (Edge) or (Graph), followed by the property names, a colon and the explanation, one item per property or group of properties; qualifications (kinds of nodes, networks) follow the names in parentheses. Properties that were only named got a short explanation, and a few items that named the wrong kind of property were corrected.
diff --git a/entries/tolokers/__init__.py b/entries/tolokers/__init__.py
index 783c8e5..9fc1b72 100644
--- a/entries/tolokers/__init__.py
+++ b/entries/tolokers/__init__.py
@@ -25,11 +25,11 @@ description = """A network of crowd workers ("Tolokers") of the Toloka crowdsour
The node indices are the annotator identifiers of the source data. The network has the following properties:
-- `approved_rate`, `skipped_rate`, `expired_rate`, `rejected_rate`: the fractions of the annotator's labels that were approved, of the tasks that the annotator skipped, of the tasks that expired, and of the labels that were rejected (they sum to 1).
-- `education`: the education level reported by the annotator, one of `none`, `basic`, `middle` and `high`.
-- `english_profile`: whether the annotator reported knowing English.
-- `english_tested`: whether the annotator passed the platform's English test.
-- `banned`: whether the annotator was banned from the project."""
+- (Node) `approved_rate`, `skipped_rate`, `expired_rate`, `rejected_rate`: the fractions of the annotator's labels that were approved, of the tasks that the annotator skipped, of the tasks that expired, and of the labels that were rejected (they sum to 1).
+- (Node) `education`: the education level reported by the annotator, one of `none`, `basic`, `middle` and `high`.
+- (Node) `english_profile`: whether the annotator reported knowing English.
+- (Node) `english_tested`: whether the annotator passed the platform's English test.
+- (Node) `banned`: whether the annotator was banned from the project."""
tags = ['Social', 'Online', 'Collaboration', 'Unweighted', 'Projection', 'Metadata']
url = 'https://github.com/Toloka/TolokerGraph'
citation = [('D. Likhobaba, N. Pavlichenko, D. Ustalov, "Toloker Graph: Interaction of Crowd Annotators." Zenodo (2023)', 'https://doi.org/10.5281/zenodo.7620795'),
[entries] Remove unneeded code from the entries 2c69f19
Unneeded "global files" statements, unused imports and variables, f-strings without placeholders, duplicate keys in foodweb_little_rock (keeping the order and the values used), and the second definition of get_lines() in twitter_higgs. Nothing changes in the networks or descriptions. Changed: none
diff --git a/entries/tolokers/__init__.py b/entries/tolokers/__init__.py
index 007fc30..783c8e5 100644
--- a/entries/tolokers/__init__.py
+++ b/entries/tolokers/__init__.py
@@ -47,7 +47,6 @@ def fetch_upstream(force=False):
@coerce_props()
@annotate()
def parse(alts=None):
- global files
name = __name__.split(".")[-1]
for fnames, alt, fmt in files:
if alts is not None and alt not in alts:
[entries] New entry: tolokers 9f37b0f
diff --git a/entries/tolokers/__init__.py b/entries/tolokers/__init__.py
new file mode 100644
index 0000000..007fc30
--- /dev/null
+++ b/entries/tolokers/__init__.py
@@ -0,0 +1,71 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2026 Tiago de Paula Peixoto <tiago@skewed.de>
+#
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option) any
+# later version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along
+# with this program. If not, see <http://www.gnu.org/licenses/>.
+
+
+from .. import *
+
+
+title = "Toloka crowd annotators (Tolokers, 2023)"
+description = """A network of crowd workers ("Tolokers") of the Toloka crowdsourcing platform who worked on one data labeling project, released by Toloka and used as a benchmark for node classification (as the `tolokers` dataset of Platonov et al.). Nodes are annotators, and an undirected edge connects two annotators who labeled the same task (at least one). Each annotator is described by the shares of their labels that were approved, skipped, expired or rejected, by their self-reported education level and knowledge of English, and by whether they passed the platform's English test; the classification target is whether the annotator was banned from the project. Which tasks were labeled, and by how many common annotators, is not included in the data.
+
+The node indices are the annotator identifiers of the source data. The network has the following properties:
+
+- `approved_rate`, `skipped_rate`, `expired_rate`, `rejected_rate`: the fractions of the annotator's labels that were approved, of the tasks that the annotator skipped, of the tasks that expired, and of the labels that were rejected (they sum to 1).
+- `education`: the education level reported by the annotator, one of `none`, `basic`, `middle` and `high`.
+- `english_profile`: whether the annotator reported knowing English.
+- `english_tested`: whether the annotator passed the platform's English test.
+- `banned`: whether the annotator was banned from the project."""
+tags = ['Social', 'Online', 'Collaboration', 'Unweighted', 'Projection', 'Metadata']
+url = 'https://github.com/Toloka/TolokerGraph'
+citation = [('D. Likhobaba, N. Pavlichenko, D. Ustalov, "Toloker Graph: Interaction of Crowd Annotators." Zenodo (2023)', 'https://doi.org/10.5281/zenodo.7620795'),
+ ('O. Platonov, D. Kuznedelev, M. Diskin, A. Babenko, L. Prokhorenkova, "A critical look at the evaluation of GNNs under heterophily: Are we really making progress?" International Conference on Learning Representations (ICLR) (2023)', 'https://doi.org/10.48550/arXiv.2302.11640')]
+icon_hash = None
+upstream_license = ('CC BY 4.0', 'https://creativecommons.org/licenses/by/4.0/')
+upstream_prefix = 'https://raw.githubusercontent.com/Toloka/TolokerGraph/04745e8f1037ea39fb226537bdc6ae6bb7bf6619'
+files = [(('edges.tsv', 'nodes.tsv'), None, None)]
+
+def fetch_upstream(force=False):
+ return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
+ force)
+
+@cache_network()
+@coerce_props()
+@annotate()
+def parse(alts=None):
+ global files
+ name = __name__.split(".")[-1]
+ for fnames, alt, fmt in files:
+ if alts is not None and alt not in alts:
+ continue
+ with ExitStack() as stack:
+ fs = [stack.enter_context(open_upstream_file(name, fn, "r")) for fn in fnames]
+
+ rows = list(csv.DictReader(fs[1], delimiter="\t"))
+ g = Graph(directed=False)
+ g.add_vertex(len(rows))
+ assert all(int(r["id"]) == i for i, r in enumerate(rows))
+ for k in ["approved_rate", "skipped_rate", "expired_rate",
+ "rejected_rate"]:
+ g.vp[k] = g.new_vp("double", vals=[float(r[k]) for r in rows])
+ g.vp.education = g.new_vp("string", vals=[r["education"] for r in rows])
+ for k in ["english_profile", "english_tested", "banned"]:
+ g.vp[k] = g.new_vp("bool", vals=[int(r[k]) for r in rows])
+
+ next(fs[0])
+ g.add_edge_list(read_ints(fs[0]).reshape(-1, 2))
+ yield alt, g