tolokers — Toloka crowd annotators (Tolokers, 2023)

Description

A network of crowd workers ("Tolokers") of the Toloka crowdsourcing platform who worked on one data labeling project, released by Toloka and used as a benchmark for node classification (as the tolokers dataset of Platonov et al.). Nodes are annotators, and an undirected edge connects two annotators who labeled the same task (at least one). Each annotator is described by the shares of their labels that were approved, skipped, expired or rejected, by their self-reported education level and knowledge of English, and by whether they passed the platform's English test; the classification target is whether the annotator was banned from the project. Which tasks were labeled, and by how many common annotators, is not included in the data.

The node indices are the annotator identifiers of the source data. The network has the following properties:

(Node) approved_rate, skipped_rate, expired_rate, rejected_rate: the fractions of the annotator's labels that were approved, of the tasks that the annotator skipped, of the tasks that expired, and of the labels that were rejected (they sum to 1).
(Node) education: the education level reported by the annotator, one of none, basic, middle and high.
(Node) english_profile: whether the annotator reported knowing English.
(Node) english_tested: whether the annotator passed the platform's English test.
(Node) banned: whether the annotator was banned from the project.

Tags

Social Online Collaboration Unweighted Projection Metadata

Added

2026-10-27

Citation

D. Likhobaba, N. Pavlichenko, D. Ustalov, "Toloker Graph: Interaction of Crowd Annotators." Zenodo (2023) 10.5281/zenodo.7620795 Sci-Hub
O. Platonov, D. Kuznedelev, M. Diskin, A. Babenko, L. Prokhorenkova, "A critical look at the evaluation of GNNs under heterophily: Are we really making progress?" International Conference on Learning Representations (ICLR) (2023) 10.48550/arXiv.2302.11640 Sci-Hub

Upstream URL OK

https://github.com/Toloka/TolokerGraph

Upstream license

CC BY 4.0, https://creativecommons.org/licenses/by/4.0/

Networks

Tip: hover over a column header for a description.

Column legend

$\left<k\right>$: Mean degree
$\sigma_k$: Standard deviation of the degrees
$\lambda_h$: Largest eigenvalue of the Hashimoto (non-backtracking) matrix
$\tau$: Random walk mixing time
$r$: Degree assortativity
$c$: Global clustering coefficient
$\oslash$: (Pseudo-) diameter
$S$: Relative size of the largest component
NPs: Node properties
EPs: Edge properties

Name	Nodes	Edges	$\left<k\right>$	$\sigma_k$	$\lambda_h$	$\tau$	$r$	$c$	$\oslash$	$S$	Kind	Mode	NPs	EPs	gt	GraphML	GML	csv
tolokers	11,758	519,000	88.28	161.82	390.61	14.32	-0.08	0.23	11	1.00	Undirected	Unipartite	approved_rate skipped_rate expired_rate rejected_rate education english_profile english_tested banned		872 KiB	1.9 MiB	1.9 MiB	1.7 MiB

Ridiculograms^*

^* These are automatically generated force-directed visualizations, and can be quite meaningless for networks both big and small. They should not be taken seriously as sources of scientific insight. See here for a discussion.