History of gab_users_fair
The changes to the source code of this entry in the repository, newest first, each with the differences it made to the entry's files. They are also listed on GitLab.
[entries] Add the date when each entry was added, and remove icon_hash beff7c9
The dates are those of the commits that first added the entries' files (2020-08-11 for the entries of the initial commit); renamed entries keep the date of their first name. Changed: none
diff --git a/entries/gab_users_fair/__init__.py b/entries/gab_users_fair/__init__.py
index 4955351..f0321ef 100644
--- a/entries/gab_users_fair/__init__.py
+++ b/entries/gab_users_fair/__init__.py
@@ -47,7 +47,7 @@ citation = [('G. Fair, R. Wesslen, "Shouting into the Void: A Database of the Al
'https://doi.org/10.1609/icwsm.v13i01.3258')]
upstream_license = ('Creative Commons Attribution 4.0 International (CC BY 4.0)',
'https://creativecommons.org/licenses/by/4.0/')
-icon_hash = None
+date_added = '2026-10-31'
# Zenodo record; a JSON array of the scraped users (one per line), 1.7 GB
# compressed
[entries] Prefix the properties in the descriptions with (Node), (Edge) or (Graph) bc3dbbd
The lists of properties at the end of the descriptions marked edge properties with "Edge" but left node properties unmarked, or grouped them in "Nodes: `a`, ...; `b`, ..." items, so the kind of a property was often unclear. Every item now starts with (Node), (Edge) or (Graph), followed by the property names, a colon and the explanation, one item per property or group of properties; qualifications (kinds of nodes, networks) follow the names in parentheses. Properties that were only named got a short explanation, and a few items that named the wrong kind of property were corrected.
diff --git a/entries/gab_users_fair/__init__.py b/entries/gab_users_fair/__init__.py
index d405949..4955351 100644
--- a/entries/gab_users_fair/__init__.py
+++ b/entries/gab_users_fair/__init__.py
@@ -29,8 +29,18 @@ The posts described by Fair and Wesslen (35.9 million) are not in the published
The network has the following properties:
-- Nodes: `user_id`, the id of the user on Gab; `username`; `display_name`, the name shown on the profile; `bio`, the biography of the profile (empty if none); `has_record`, whether the user's profile was scraped (otherwise the user only appears in the lists of others); `created`, the month of creation of the account (Unix time of its first day; -1 if unknown); `follower_count`, `following_count` and `post_count`, as shown on the profile (-1 if unknown); `score` (-1 if unknown); `verified`, `is_pro`, `is_donor`, `is_investor`, `is_premium`, `is_tippable`, `is_private` and `is_accessible`, the flags of the account (1 or 0; -1 if unknown); `collected`, the time when the profile was scraped (Unix time; -1 for users not scraped); `followers_complete` and `following_complete`, whether the lists of followers and followings of the user were scraped completely (1 or 0; -1 if unknown).
-- Edges: `in_followers` and `in_following`, whether the relation is in the list of followers of v, and in the list of followings of u."""
+- (Node) `user_id`: the id of the user on Gab.
+- (Node) `username`.
+- (Node) `display_name`: the name shown on the profile.
+- (Node) `bio`: the biography of the profile (empty if none).
+- (Node) `has_record`: whether the user's profile was scraped (otherwise the user only appears in the lists of others).
+- (Node) `created`: the month of creation of the account (Unix time of its first day; -1 if unknown).
+- (Node) `follower_count`, `following_count` and `post_count`: as shown on the profile (-1 if unknown).
+- (Node) `score` (-1 if unknown).
+- (Node) `verified`, `is_pro`, `is_donor`, `is_investor`, `is_premium`, `is_tippable`, `is_private` and `is_accessible`: the flags of the account (1 or 0; -1 if unknown).
+- (Node) `collected`: the time when the profile was scraped (Unix time; -1 for users not scraped).
+- (Node) `followers_complete` and `following_complete`: whether the lists of followers and followings of the user were scraped completely (1 or 0; -1 if unknown).
+- (Edge) `in_followers` and `in_following`: whether the relation is in the list of followers of v, and in the list of followings of u."""
tags = ['Social', 'Online', 'Political', 'Unweighted', 'Metadata']
url = 'https://doi.org/10.5281/zenodo.2541324'
citation = [('G. Fair, R. Wesslen, "Shouting into the Void: A Database of the Alternative Social Media Platform Gab." Proceedings of the International AAAI Conference on Web and Social Media 13, 608-610 (2019)',
[entries] Remove unneeded code from the entries 2c69f19
Unneeded "global files" statements, unused imports and variables, f-strings without placeholders, duplicate keys in foodweb_little_rock (keeping the order and the values used), and the second definition of get_lines() in twitter_higgs. Nothing changes in the networks or descriptions. Changed: none
diff --git a/entries/gab_users_fair/__init__.py b/entries/gab_users_fair/__init__.py
index 8155206..d405949 100644
--- a/entries/gab_users_fair/__init__.py
+++ b/entries/gab_users_fair/__init__.py
@@ -79,7 +79,6 @@ def profile(u):
@coerce_props()
@annotate()
def parse(alts=None):
- global files
name = __name__.split(".")[-1]
idx = {}
attrs = {} # index -> profile list
[entries] New entry: gab_users_fair 4b73eea
diff --git a/entries/gab_users_fair/__init__.py b/entries/gab_users_fair/__init__.py
new file mode 100644
index 0000000..8155206
--- /dev/null
+++ b/entries/gab_users_fair/__init__.py
@@ -0,0 +1,174 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2026 Tiago de Paula Peixoto <tiago@skewed.de>
+#
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option) any
+# later version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along
+# with this program. If not, see <http://www.gnu.org/licenses/>.
+
+from .. import *
+try:
+ import orjson as fastjson
+except ImportError:
+ fastjson = json
+
+title = "Gab follower network (2019)"
+description = """The follower network of the users of Gab, a microblogging platform popular with the alt-right, from the database of Fair and Wesslen, who scraped the profiles of the users of Gab, with their lists of followers and followings, from January to February 2019 (the profiles of the users created from August 2016 to December 2018). A directed edge from user u to user v means that u follows v; the edges are the union of the lists of followers and followings of the 757,363 scraped users (each relation once). For some users, the lists were not scraped completely (`followers_complete` or `following_complete` false); the follower lists of 135,630 users and the following lists of 121,331 users are incomplete. The users that appear only in the lists of others are included, with the profile attributes given in the lists (including their display names and biographies). The data contain no phone numbers or e-mail addresses.
+
+The posts described by Fair and Wesslen (35.9 million) are not in the published data. Left out are the profile pictures and cover images of the users, and the separate list of usernames. Another crawl of Gab, of 2018, with posts, is the entry `gab_followers`.
+
+The network has the following properties:
+
+- Nodes: `user_id`, the id of the user on Gab; `username`; `display_name`, the name shown on the profile; `bio`, the biography of the profile (empty if none); `has_record`, whether the user's profile was scraped (otherwise the user only appears in the lists of others); `created`, the month of creation of the account (Unix time of its first day; -1 if unknown); `follower_count`, `following_count` and `post_count`, as shown on the profile (-1 if unknown); `score` (-1 if unknown); `verified`, `is_pro`, `is_donor`, `is_investor`, `is_premium`, `is_tippable`, `is_private` and `is_accessible`, the flags of the account (1 or 0; -1 if unknown); `collected`, the time when the profile was scraped (Unix time; -1 for users not scraped); `followers_complete` and `following_complete`, whether the lists of followers and followings of the user were scraped completely (1 or 0; -1 if unknown).
+- Edges: `in_followers` and `in_following`, whether the relation is in the list of followers of v, and in the list of followings of u."""
+tags = ['Social', 'Online', 'Political', 'Unweighted', 'Metadata']
+url = 'https://doi.org/10.5281/zenodo.2541324'
+citation = [('G. Fair, R. Wesslen, "Shouting into the Void: A Database of the Alternative Social Media Platform Gab." Proceedings of the International AAAI Conference on Web and Social Media 13, 608-610 (2019)',
+ 'https://doi.org/10.1609/icwsm.v13i01.3258')]
+upstream_license = ('Creative Commons Attribution 4.0 International (CC BY 4.0)',
+ 'https://creativecommons.org/licenses/by/4.0/')
+icon_hash = None
+
+# Zenodo record; a JSON array of the scraped users (one per line), 1.7 GB
+# compressed
+upstream_prefix = 'https://zenodo.org/records/2541324/files'
+files = [('gab_ai_user_data.xz', None, None)]
+
+# Number of lines read (None for all; set only for checks)
+_check_rows = None
+
+def fetch_upstream(force=False):
+ return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
+ force)
+
+FLAGS = ["verified", "is_pro", "is_donor", "is_investor", "is_premium",
+ "is_tippable", "is_private", "is_accessible"]
+COUNTS = ["follower_count", "following_count", "post_count", "score"]
+MONTHS = {m: i + 1 for i, m in enumerate(
+ ["January", "February", "March", "April", "May", "June", "July", "August",
+ "September", "October", "November", "December"])}
+
+def month(label):
+ try:
+ m, y = label.split()
+ return int(numpy.datetime64(f"{int(y):04d}-{MONTHS[m]:02d}-01", "s").astype("int64"))
+ except (AttributeError, ValueError, KeyError):
+ return -1
+
+def flag(x):
+ return -1 if x is None else int(bool(x))
+
+def profile(u):
+ "The attributes of a user (as in the profiles and the lists)."
+ return ([u.get("username") or "", u.get("name") or "", u.get("bio") or "",
+ month(u.get("created_at_month_label"))] +
+ [-1 if u.get(c) is None else int(u[c]) for c in COUNTS] +
+ [flag(u.get(f)) for f in FLAGS])
+
+@cache_network()
+@coerce_props()
+@annotate()
+def parse(alts=None):
+ global files
+ name = __name__.split(".")[-1]
+ idx = {}
+ attrs = {} # index -> profile list
+ records = {} # index -> (collected, followers_complete, following_complete)
+ chunks = []
+ edges = []
+ def vertex(u):
+ i = idx.get(u["id"])
+ if i is None:
+ i = idx[u["id"]] = len(idx)
+ return i
+ with open_upstream_file(name, files[0][0], "rb") as f:
+ for n, line in enumerate(f):
+ if _check_rows is not None and n >= _check_rows:
+ break
+ line = line.strip().rstrip(b",")
+ if not line.startswith(b"{"):
+ continue
+ d = fastjson.loads(line)
+ if d.get("type") != "users":
+ continue
+ data = d["data"]
+ u = data.get("user", data)
+ v = vertex(u)
+ attrs[v] = profile(u)
+ fl = data.get("followers") or {}
+ fg = data.get("following") or {}
+ records[v] = (int(numpy.datetime64(d["collected_on"].replace(" ", "T"), "s").astype("int64")),
+ flag(fl.get("no-more")) if "followers" in data else -1,
+ flag(fg.get("no-more")) if "following" in data else -1)
+ for w in fl.get("data") or []:
+ x = vertex(w)
+ if x not in attrs or x not in records:
+ attrs.setdefault(x, profile(w))
+ edges.append((x, v, 1))
+ for w in fg.get("data") or []:
+ x = vertex(w)
+ if x not in records:
+ attrs.setdefault(x, profile(w))
+ edges.append((v, x, 2))
+ if len(edges) > 1 << 22:
+ chunks.append(numpy.array(edges, dtype="int64"))
+ edges = []
+ chunks.append(numpy.array(edges, dtype="int64").reshape(-1, 3))
+ del edges
+ e = numpy.concatenate(chunks)
+ del chunks
+ # one edge per relation, with the lists in which it appears (bit mask)
+ order = numpy.lexsort((e[:, 1], e[:, 0]))
+ e = e[order]
+ del order
+ new = numpy.ones(len(e), dtype="bool")
+ new[1:] = (e[1:, 0] != e[:-1, 0]) | (e[1:, 1] != e[:-1, 1])
+ start = numpy.flatnonzero(new)
+ lists = numpy.bitwise_or.reduceat(e[:, 2], start) if len(e) else e[:, 2]
+ e = e[start, :2]
+
+ g = Graph(directed=True)
+ g.add_vertex(len(idx))
+ g.vp.user_id = g.new_vp("int64_t")
+ for k, i in idx.items():
+ g.vp.user_id[i] = k
+ del idx
+ g.vp.username = g.new_vp("string")
+ g.vp.display_name = g.new_vp("string")
+ g.vp.bio = g.new_vp("string")
+ g.vp.has_record = g.new_vp("bool")
+ names = ["created"] + COUNTS + FLAGS
+ props = [g.new_vp("int64_t", val=-1) for c in names]
+ for i, a in attrs.items():
+ g.vp.username[i] = a[0]
+ g.vp.display_name[i] = a[1]
+ g.vp.bio[i] = a[2]
+ for p, x in zip(props, a[3:]):
+ p[i] = x
+ del attrs
+ for c, p in zip(names, props):
+ g.vp[c] = p
+ g.vp.collected = g.new_vp("int64_t", val=-1)
+ g.vp.followers_complete = g.new_vp("int16_t", val=-1)
+ g.vp.following_complete = g.new_vp("int16_t", val=-1)
+ for i, (t, a, b) in records.items():
+ g.vp.has_record[i] = True
+ g.vp.collected[i] = t
+ g.vp.followers_complete[i] = a
+ g.vp.following_complete[i] = b
+ del records
+ g.ep.in_followers = g.new_ep("bool")
+ g.ep.in_following = g.new_ep("bool")
+ g.add_edge_list(numpy.column_stack((e, lists & 1, lists >> 1)),
+ eprops=[g.ep.in_followers, g.ep.in_following])
+ yield None, g