logo Netzschleuder network catalog, repository, and centrifuge

History of telegram_forwards

The changes to the source code of this entry in the repository, newest first, each with the differences it made to the entry's files. They are also listed on GitLab.

[entries] Add the date when each entry was added, and remove icon_hash by Tiago de Paula Peixoto beff7c9
The dates are those of the commits that first added the entries' files
(2020-08-11 for the entries of the initial commit); renamed entries keep
the date of their first name.

Changed: none
diff --git a/entries/telegram_forwards/__init__.py b/entries/telegram_forwards/__init__.py
index bfa7db6..05b9341 100644
--- a/entries/telegram_forwards/__init__.py
+++ b/entries/telegram_forwards/__init__.py
@@ -54,7 +54,7 @@ citation = [('J. Baumgartner, S. Zannettou, M. Squire, J. Blackburn, "The Pushsh
              'https://doi.org/10.1609/icwsm.v14i1.7348')]
 upstream_license = ('Creative Commons Attribution 4.0 International (CC BY 4.0)',
                     'https://creativecommons.org/licenses/by/4.0/')
-icon_hash = None
+date_added = '2026-10-31'
 
 # Zenodo record; messages.ndjson.zst is 52 GB (317 million messages), read
 # as a stream
[entries] Prefix the properties in the descriptions with (Node), (Edge) or (Graph) by Tiago de Paula Peixoto bc3dbbd
The lists of properties at the end of the descriptions marked edge
properties with "Edge" but left node properties unmarked, or grouped
them in "Nodes: `a`, ...; `b`, ..." items, so the kind of a property
was often unclear. Every item now starts with (Node), (Edge) or (Graph),
followed by the property names, a colon and the explanation, one item
per property or group of properties; qualifications (kinds of nodes,
networks) follow the names in parentheses. Properties that were only
named got a short explanation, and a few items that named the wrong
kind of property were corrected.
diff --git a/entries/telegram_forwards/__init__.py b/entries/telegram_forwards/__init__.py
index e5f1fbb..bfa7db6 100644
--- a/entries/telegram_forwards/__init__.py
+++ b/entries/telegram_forwards/__init__.py
@@ -32,9 +32,22 @@ In `posts`, the users have the information of their profiles (names, usernames,
 
 The networks have the following properties:
 
-- Nodes (channels): `channel_id`, the id of the channel on Telegram; `crawled`, whether the channel is one of the crawled ones (otherwise only a source of forwarded messages); `title` and `username` (its address is `https://t.me/<username>`), and `created`, its time of creation (Unix time), known for the crawled channels and for the channels linked to them (empty or -1 otherwise); `broadcast`, whether it is a broadcast channel (1), a group (0) or unknown (-1); `participants_count`, the number of members when it was crawled (-1 if unknown); `about`, its description (crawled channels only); `retrieved`, the time when its information was retrieved (Unix time; -1 for channels not crawled).
-- In `posts`, nodes also have `is_user`, whether the node is a user (otherwise a channel); `user_id`, the id of the user on Telegram (-1 for channels); `has_profile`, whether the user is in the file of accounts; `first_name` and `last_name`, the display name of the user; `phone`, the phone number of the user, if visible (international format without "+"); `bot`, `deleted`, `verified`, `scam` and `restricted`, the flags of the account (false for channels); for users, `username` is the username of the account (its address is `https://t.me/<username>`) and `retrieved` the time when the profile was retrieved (Unix time). The other channel properties are empty or -1 for users.
-- Edges: `count`, the number of forwarded messages (`forwards`) or of posted messages (`posts`); `first_time` and `last_time`, the times of the first and of the last of these messages (Unix time; in `forwards`, the times when they were forwarded)."""
+- (Node) `channel_id` (channels): the id of the channel on Telegram.
+- (Node) `crawled` (channels): whether the channel is one of the crawled ones (otherwise only a source of forwarded messages).
+- (Node) `title`, `username`, `created` (channels): its title, its username (its address is `https://t.me/<username>`) and its time of creation (Unix time), known for the crawled channels and for the channels linked to them (empty or -1 otherwise).
+- (Node) `broadcast` (channels): whether it is a broadcast channel (1), a group (0) or unknown (-1).
+- (Node) `participants_count` (channels): the number of members when it was crawled (-1 if unknown).
+- (Node) `about` (channels): its description (crawled channels only).
+- (Node) `retrieved` (channels): the time when its information was retrieved (Unix time; -1 for channels not crawled).
+- (Node) `is_user` (`posts`): whether the node is a user (otherwise a channel).
+- (Node) `user_id` (`posts`): the id of the user on Telegram (-1 for channels).
+- (Node) `has_profile` (`posts`): whether the user is in the file of accounts.
+- (Node) `first_name`, `last_name` (`posts`): the display name of the user.
+- (Node) `phone` (`posts`): the phone number of the user, if visible (international format without "+").
+- (Node) `bot`, `deleted`, `verified`, `scam`, `restricted` (`posts`): the flags of the account (false for channels).
+- (Node) `username`, `retrieved` (users, in `posts`): the username of the account (its address is `https://t.me/<username>`) and the time when the profile was retrieved (Unix time). The other channel properties are empty or -1 for users.
+- (Edge) `count`: the number of forwarded messages (`forwards`) or of posted messages (`posts`).
+- (Edge) `first_time` and `last_time`: the times of the first and of the last of these messages (Unix time; in `forwards`, the times when they were forwarded)."""
 tags = ['Social', 'Online', 'Communication', 'Political', 'Bipartite', 'Weighted', 'Timestamps', 'Metadata']
 url = 'https://doi.org/10.5281/zenodo.3607497'
 citation = [('J. Baumgartner, S. Zannettou, M. Squire, J. Blackburn, "The Pushshift Telegram Dataset." Proceedings of the International AAAI Conference on Web and Social Media 14, 840-847 (2020)',
[entries] Remove unneeded code from the entries by Tiago de Paula Peixoto 2c69f19
Unneeded "global files" statements, unused imports and variables,
f-strings without placeholders, duplicate keys in foodweb_little_rock
(keeping the order and the values used), and the second definition of
get_lines() in twitter_higgs. Nothing changes in the networks or
descriptions.

Changed: none
diff --git a/entries/telegram_forwards/__init__.py b/entries/telegram_forwards/__init__.py
index 6d85504..e5f1fbb 100644
--- a/entries/telegram_forwards/__init__.py
+++ b/entries/telegram_forwards/__init__.py
@@ -120,7 +120,6 @@ def read_messages(name, fname):
 @coerce_props()
 @annotate()
 def parse(alts=None):
-    global files
     name = __name__.split(".")[-1]
     for fnames, alt, fmt in files:
         if alts is not None and alt not in alts:
[entries] New entry: telegram_forwards by Tiago de Paula Peixoto b66b750
diff --git a/entries/telegram_forwards/__init__.py b/entries/telegram_forwards/__init__.py
new file mode 100644
index 0000000..6d85504
--- /dev/null
+++ b/entries/telegram_forwards/__init__.py
@@ -0,0 +1,224 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2026 Tiago de Paula Peixoto <tiago@skewed.de>
+#
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option) any
+# later version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from .. import *
+try:
+    import orjson as fastjson
+except ImportError:
+    fastjson = json
+
+title = "Pushshift Telegram channels and users (2015-2019)"
+description = """Networks of Telegram channels and groups from the Pushshift Telegram dataset of Baumgartner, Zannettou, Squire and Blackburn, who collected all messages (317 million, from 2015 to 2019) of 27,801 public channels and groups (supergroups), which they found by a snowball crawl starting from about 250 channels on politics, cryptocurrencies and other topics, through the channels referenced in their messages (mostly right-wing and cryptocurrency channels). The data are as returned by the Telegram API (Telethon). The entry contains two networks:
+
+- `forwards`: a directed edge from channel u to channel v means that messages of u were forwarded to v (v republished messages first posted in u), weighted by the number of forwarded messages. Besides the crawled channels, the nodes include all channels whose messages were forwarded to them. Forwards of messages of individual users (and of hidden ones, given only by name) are left out.
+- `posts`: a bipartite network of users and channels, where an edge connects a user to a channel (or group) in which they posted messages, weighted by the number of messages. Messages posted by the channels themselves (in broadcast channels, without user) are left out, and so are service messages (e.g. users joining). All crawled channels are included, also those without messages of users.
+
+In `posts`, the users have the information of their profiles (names, usernames, phone numbers and flags) from the file of the 2.2 million accounts of the dataset, retrieved in 2019; Telegram users have no biographies in these data, and most phone numbers are hidden (about 13,000 accounts show one). Left out are the profile photos and the online status of the users, and the contents of the messages, replies and mentions.
+
+The networks have the following properties:
+
+- Nodes (channels): `channel_id`, the id of the channel on Telegram; `crawled`, whether the channel is one of the crawled ones (otherwise only a source of forwarded messages); `title` and `username` (its address is `https://t.me/<username>`), and `created`, its time of creation (Unix time), known for the crawled channels and for the channels linked to them (empty or -1 otherwise); `broadcast`, whether it is a broadcast channel (1), a group (0) or unknown (-1); `participants_count`, the number of members when it was crawled (-1 if unknown); `about`, its description (crawled channels only); `retrieved`, the time when its information was retrieved (Unix time; -1 for channels not crawled).
+- In `posts`, nodes also have `is_user`, whether the node is a user (otherwise a channel); `user_id`, the id of the user on Telegram (-1 for channels); `has_profile`, whether the user is in the file of accounts; `first_name` and `last_name`, the display name of the user; `phone`, the phone number of the user, if visible (international format without "+"); `bot`, `deleted`, `verified`, `scam` and `restricted`, the flags of the account (false for channels); for users, `username` is the username of the account (its address is `https://t.me/<username>`) and `retrieved` the time when the profile was retrieved (Unix time). The other channel properties are empty or -1 for users.
+- Edges: `count`, the number of forwarded messages (`forwards`) or of posted messages (`posts`); `first_time` and `last_time`, the times of the first and of the last of these messages (Unix time; in `forwards`, the times when they were forwarded)."""
+tags = ['Social', 'Online', 'Communication', 'Political', 'Bipartite', 'Weighted', 'Timestamps', 'Metadata']
+url = 'https://doi.org/10.5281/zenodo.3607497'
+citation = [('J. Baumgartner, S. Zannettou, M. Squire, J. Blackburn, "The Pushshift Telegram Dataset." Proceedings of the International AAAI Conference on Web and Social Media 14, 840-847 (2020)',
+             'https://doi.org/10.1609/icwsm.v14i1.7348')]
+upstream_license = ('Creative Commons Attribution 4.0 International (CC BY 4.0)',
+                    'https://creativecommons.org/licenses/by/4.0/')
+icon_hash = None
+
+# Zenodo record; messages.ndjson.zst is 52 GB (317 million messages), read
+# as a stream
+upstream_prefix = 'https://zenodo.org/records/3607497/files'
+files = [(('channels.ndjson.zst', 'messages.ndjson.zst'), 'forwards', None),
+         (('channels.ndjson.zst', 'messages.ndjson.zst', 'accounts.ndjson.zst'),
+          'posts', None)]
+
+# Number of messages read (None for all; set only for checks on a sample)
+_check_rows = None
+
+def fetch_upstream(force=False):
+    return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
+                                force)
+
+def unix(s):
+    "Unix time of an ISO date string of the API (UTC), or -1."
+    if s is None:
+        return -1
+    return int(numpy.datetime64(s[:19], "s").astype("int64"))
+
+def read_channels(name, fname):
+    """The information on the channels: a dict mapping channel ids to tuples
+    (crawled, title, username, created, broadcast, participants_count,
+    about, retrieved). The crawled channels come with the channels linked to
+    them (e.g. their discussion groups), whose information is also kept."""
+    chans = {}
+    with open_upstream_file(name, fname, "rb") as f:
+        for line in f:
+            d = fastjson.loads(line)
+            fc = d["full_chat"]
+            for c in d["chats"]:
+                if c.get("_") not in ("Channel", "Chat"):
+                    continue
+                crawled = c["id"] == fc["id"]
+                if not crawled and chans.get(c["id"], (False,))[0]:
+                    continue
+                bc = c.get("broadcast")
+                chans[c["id"]] = (crawled, c.get("title") or "",
+                                  c.get("username") or "",
+                                  unix(c.get("date")),
+                                  -1 if bc is None else int(bc),
+                                  (fc.get("participants_count") if crawled else
+                                   c.get("participants_count")) or -1,
+                                  (fc.get("about") or "") if crawled else "",
+                                  d.get("retrieved_utc", -1) if crawled else -1)
+    return chans
+
+USER_FLAGS = ["bot", "deleted", "verified", "scam", "restricted"]
+
+def read_accounts(name, fname, uids):
+    """The profiles of the users with ids in uids: a dict mapping user ids to
+    tuples (first_name, last_name, username, phone, retrieved, *USER_FLAGS)."""
+    uids = set(int(u) for u in uids)
+    users = {}
+    with open_upstream_file(name, fname, "rb") as f:
+        for line in f:
+            d = fastjson.loads(line)
+            if d.get("id") not in uids:
+                continue
+            users[d["id"]] = ((d.get("first_name") or "", d.get("last_name") or "",
+                               d.get("username") or "", d.get("phone") or "",
+                               d.get("retrieved_utc") or -1) +
+                              tuple(bool(d.get(c)) for c in USER_FLAGS))
+    return users
+
+def read_messages(name, fname):
+    "Iterate over the messages, as dicts."
+    with open_upstream_file(name, fname, "rb") as f:
+        for i, line in enumerate(f):
+            if _check_rows is not None and i >= _check_rows:
+                break
+            yield fastjson.loads(line)
+
+@cache_network()
+@coerce_props()
+@annotate()
+def parse(alts=None):
+    global files
+    name = __name__.split(".")[-1]
+    for fnames, alt, fmt in files:
+        if alts is not None and alt not in alts:
+            continue
+        fchan, fmsg = fnames[:2]
+        chans = read_channels(name, fchan)
+        edges = {}   # (source, target) -> [count, first, last]
+        for m in read_messages(name, fmsg):
+            if m.get("_") != "Message":
+                continue
+            to = m.get("to_id") or {}
+            target = to.get("channel_id")
+            if target is None:
+                continue
+            if alt == "forwards":
+                fw = m.get("fwd_from")
+                if not fw or fw.get("channel_id") is None:
+                    continue
+                key = (fw["channel_id"], target)
+            else:
+                if m.get("from_id") is None:
+                    continue
+                key = (m["from_id"], target)
+            t = unix(m.get("date"))
+            e = edges.get(key)
+            if e is None:
+                edges[key] = [1, t, t]
+            else:
+                e[0] += 1
+                if t < e[1]:
+                    e[1] = t
+                if t > e[2]:
+                    e[2] = t
+
+        keys = numpy.array(list(edges.keys()), dtype="int64").reshape(-1, 2)
+        vals = numpy.array(list(edges.values()), dtype="int64").reshape(-1, 3)
+        del edges
+        crawled = numpy.array(sorted(k for k, v in chans.items() if v[0]), dtype="int64")
+        if alt == "forwards":
+            cids = numpy.union1d(crawled, keys.ravel())
+            src = numpy.searchsorted(cids, keys[:, 0])
+            tgt = numpy.searchsorted(cids, keys[:, 1])
+            uids = numpy.zeros(0, dtype="int64")
+        else:
+            cids = numpy.union1d(crawled, keys[:, 1])
+            uids = numpy.unique(keys[:, 0])
+            src = numpy.searchsorted(uids, keys[:, 0])
+            tgt = numpy.searchsorted(cids, keys[:, 1]) + len(uids)
+        nu = len(uids)
+        g = Graph(directed=(alt == "forwards"))
+        g.add_vertex(nu + len(cids))
+        if alt == "posts":
+            g.vp.is_user = g.new_vp("bool")
+            g.vp.is_user.a[:nu] = True
+            g.vp.user_id = g.new_vp("int64_t", val=-1)
+            g.vp.user_id.a[:nu] = uids
+        g.vp.channel_id = g.new_vp("int64_t", val=-1)
+        g.vp.channel_id.a[nu:] = cids
+        g.vp.crawled = g.new_vp("bool")
+        g.vp.title = g.new_vp("string")
+        g.vp.username = g.new_vp("string")
+        g.vp.created = g.new_vp("int64_t", val=-1)
+        g.vp.broadcast = g.new_vp("int16_t", val=-1)
+        g.vp.participants_count = g.new_vp("int64_t", val=-1)
+        g.vp.about = g.new_vp("string")
+        g.vp.retrieved = g.new_vp("int64_t", val=-1)
+        if alt == "posts":
+            g.vp.has_profile = g.new_vp("bool")
+            g.vp.first_name = g.new_vp("string")
+            g.vp.last_name = g.new_vp("string")
+            g.vp.phone = g.new_vp("string")
+            flags = [g.new_vp("bool") for c in USER_FLAGS]
+            for c, p in zip(USER_FLAGS, flags):
+                g.vp[c] = p
+            users = read_accounts(name, fnames[2], uids)
+            for i, u in enumerate(uids):
+                info = users.get(int(u))
+                if info is None:
+                    continue
+                v = g.vertex(i)
+                g.vp.has_profile[v] = True
+                (g.vp.first_name[v], g.vp.last_name[v], g.vp.username[v],
+                 g.vp.phone[v], g.vp.retrieved[v]) = info[:5]
+                for p, x in zip(flags, info[5:]):
+                    p[v] = x
+            del users
+        for i, c in enumerate(cids):
+            info = chans.get(int(c))
+            if info is None:
+                continue
+            v = g.vertex(nu + i)
+            (g.vp.crawled[v], g.vp.title[v], g.vp.username[v], g.vp.created[v],
+             g.vp.broadcast[v], g.vp.participants_count[v], g.vp.about[v],
+             g.vp.retrieved[v]) = info
+        g.ep.count = g.new_ep("int64_t")
+        g.ep.first_time = g.new_ep("int64_t")
+        g.ep.last_time = g.new_ep("int64_t")
+        g.add_edge_list(numpy.column_stack((src, tgt, vals)),
+                        eprops=[g.ep.count, g.ep.first_time, g.ep.last_time])
+        del keys, vals, src, tgt
+        yield alt, g