History of wikipedia_discussions_de
The changes to the source code of this entry in the repository, newest first, each with the differences it made to the entry's files. They are also listed on GitLab.
[entries] Add the date when each entry was added, and remove icon_hash beff7c9
The dates are those of the commits that first added the entries' files (2020-08-11 for the entries of the initial commit); renamed entries keep the date of their first name. Changed: none
diff --git a/entries/wikipedia_discussions_de/__init__.py b/entries/wikipedia_discussions_de/__init__.py
index 59e82cd..1e51940 100644
--- a/entries/wikipedia_discussions_de/__init__.py
+++ b/entries/wikipedia_discussions_de/__init__.py
@@ -48,7 +48,7 @@ url = ['https://archive.org/details/dewiki-20150211',
'https://dumps.wikimedia.org/other/mediawiki_content_current/',
'https://github.com/kunegis/konect-extr/tree/master/extr/wikipedia-discussions']
citation = [('"Wikimedia Foundation. Wikimedia downloads."', 'https://dumps.wikimedia.org/')]
-icon_hash = None
+date_added = '2026-10-27'
upstream_license = ("Creative Commons Attribution-ShareAlike 4.0 (CC BY-SA 4.0) and GFDL",
"https://dumps.wikimedia.org/legal.html")
# Too large to process on an ordinary machine (see --skip-huge): the two
[entries] Prefix the properties in the descriptions with (Node), (Edge) or (Graph) bc3dbbd
The lists of properties at the end of the descriptions marked edge properties with "Edge" but left node properties unmarked, or grouped them in "Nodes: `a`, ...; `b`, ..." items, so the kind of a property was often unclear. Every item now starts with (Node), (Edge) or (Graph), followed by the property names, a colon and the explanation, one item per property or group of properties; qualifications (kinds of nodes, networks) follow the names in parentheses. Properties that were only named got a short explanation, and a few items that named the wrong kind of property were corrected.
diff --git a/entries/wikipedia_discussions_de/__init__.py b/entries/wikipedia_discussions_de/__init__.py
index fc14928..59e82cd 100644
--- a/entries/wikipedia_discussions_de/__init__.py
+++ b/entries/wikipedia_discussions_de/__init__.py
@@ -38,11 +38,11 @@ User names are normalized as MediaWiki page titles (underscores replaced by spac
The networks have the following properties:
-- `name`: the user name.
-- Edge `time`: the time of the reply, from its signature (Unix time; the signatures give it to the minute, in German time, which is converted to UTC).
-- Edge `inferred_time`: whether the comment had no date of its own, so that its time is that of the closest preceding comment of lower indentation.
-- Edge `page`: the title of the talk page.
-- Edge `thread`: the title of the section (empty for comments before the first section heading of the page)."""
+- (Node) `name`: the user name.
+- (Edge) `time`: the time of the reply, from its signature (Unix time; the signatures give it to the minute, in German time, which is converted to UTC).
+- (Edge) `inferred_time`: whether the comment had no date of its own, so that its time is that of the closest preceding comment of lower indentation.
+- (Edge) `page`: the title of the talk page.
+- (Edge) `thread`: the title of the section (empty for comments before the first section heading of the page)."""
tags = ['Social', 'Online', 'Communication', 'Unweighted', 'Multigraph', 'Timestamps', 'Metadata']
url = ['https://archive.org/details/dewiki-20150211',
'https://dumps.wikimedia.org/other/mediawiki_content_current/',
[entries] Move contact, convote_mentions, douban, email_company, epinions, flixster, football_league_results, hyves, lastfm, libimseti, livemocha, reuters, sa_companies, sister_cities, tripadvisor_ratings, wikilens_ratings and wikipedia_discussions_de to their original sources 8f097de
These entries downloaded KONECT's copies of the data. They now download the original data (live, from pinned Wayback Machine snapshots, or by re-extracting from public dumps), gaining the metadata KONECT dropped (names, identifiers, timestamps, attributes), with descriptions and citations rewritten from the original sources and factual errors fixed (years and periods of contact, email_company, douban, flixster, libimseti, sa_companies; the rated objects of epinions; the sign of convote_mentions; songs merged by title alone in lastfm). New networks: the trust network of epinions, the links of wikilens_ratings, the JSON crawl of tripadvisor_ratings, and current versions of sister_cities and wikipedia_discussions_de next to the ones KONECT used. asu_social_networks only loses its mention of KONECT. Changed: upstream contact convote_mentions douban email_company epinions flixster football_league_results hyves lastfm libimseti livemocha reuters sa_companies sister_cities tripadvisor_ratings wikilens_ratings wikipedia_discussions_de
diff --git a/entries/wikipedia_discussions_de/__init__.py b/entries/wikipedia_discussions_de/__init__.py
index 9de3d15..fc14928 100644
--- a/entries/wikipedia_discussions_de/__init__.py
+++ b/entries/wikipedia_discussions_de/__init__.py
@@ -16,44 +16,313 @@
# You should have received a copy of the GNU Affero General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
+import array
+import bz2
+import calendar
+from datetime import datetime, timezone
+import html
+import re
+import zoneinfo
+import numpy
from .. import *
-title = 'German Wikipedia discussion replies (2001-2015)'
-description = """A directed network of replies between the users of the German Wikipedia in the discussion threads of the talk pages, as extracted by KONECT. Each node is a user, and each timestamped directed edge from user A to user B is a comment of A in reply to a comment of B. Users may reply to themselves. The data identify the users only by indices.
+title = 'German Wikipedia discussion replies (2001-2026)'
+description = """Directed networks of replies between the users of the German Wikipedia in the discussion threads of the talk pages of its articles, extracted from two dumps of the current versions of all pages: `2015`, from the dump of 11 February 2015, and `2026`, from the dump ("MediaWiki Content File Export") of 1 October 2026, which reflects the state of the talk pages at that time. Each node is a registered user, and each directed edge from user A to user B is a comment of A in reply to a comment of B, so that users who replied to each other several times are connected by multiple edges; users may reply to themselves.
-The network has the following properties:
+Only the talk pages of articles (namespace "Diskussion", including their archive subpages) are considered, not those of users, projects, templates, etc. The replies are inferred from the wikitext of the current version of each page with the method of N. Geilen and J. Kunegis (University of Koblenz–Landau), reimplemented here: each section heading starts a new thread; within a thread, a comment is a line that starts with colons (its indentation level is the number of colons), that starts with "--" (taking the indentation level of the previous line), or that contains a link to a user page (`[[Benutzer:...]]`, `[[Benutzerin:...]]` or `[[User:...]]`). The author of a comment is the user of the last such link of its line (the signature), and its date the last date in the format of the signatures of the German Wikipedia ("12:34, 5. Okt. 2010") of that line; a comment without a date takes that of the closest preceding comment of lower indentation that has one. Comments without an author (including those signed by unregistered users, identified by their IP addresses, and those marked as unsigned with the templates `{{unsigned}}` and `{{unsigniert}}`), without a date, or dated before 15 January 2001 or after the date of the dump are left out. Each remaining comment of a thread, except the first, is a reply to the closest preceding remaining comment of the thread with a lower indentation level, or with indentation level zero. Since only the current versions of the pages are used, discussions removed from the talk pages (and not archived) are not included, and since comments are recognized from their signatures, some replies are missed or attributed approximately.
-- Edge `time`: the time of the reply, in Unix time."""
-tags = ['Social', 'Online', 'Communication', 'Unweighted', 'Multigraph', 'Timestamps']
-url = 'http://konect.cc/networks/wikipedia-discussions-de'
-citation = [('J. Kunegis, "Wikipedia threads (de) network dataset." KONECT, the Koblenz Network Collection (2017)', 'http://konect.cc/networks/wikipedia-discussions-de'),
- ('J. Kunegis, "KONECT -- The Koblenz Network Collection." Proceedings of the 22nd International Conference on World Wide Web Companion, 1343-1350 (2013)', 'https://doi.org/10.1145/2487788.2488173')]
+User names are normalized as MediaWiki page titles (underscores replaced by spaces, first letter capitalized), so that different spellings of the link to the same user page give the same node.
+
+The networks have the following properties:
+
+- `name`: the user name.
+- Edge `time`: the time of the reply, from its signature (Unix time; the signatures give it to the minute, in German time, which is converted to UTC).
+- Edge `inferred_time`: whether the comment had no date of its own, so that its time is that of the closest preceding comment of lower indentation.
+- Edge `page`: the title of the talk page.
+- Edge `thread`: the title of the section (empty for comments before the first section heading of the page)."""
+tags = ['Social', 'Online', 'Communication', 'Unweighted', 'Multigraph', 'Timestamps', 'Metadata']
+url = ['https://archive.org/details/dewiki-20150211',
+ 'https://dumps.wikimedia.org/other/mediawiki_content_current/',
+ 'https://github.com/kunegis/konect-extr/tree/master/extr/wikipedia-discussions']
+citation = [('"Wikimedia Foundation. Wikimedia downloads."', 'https://dumps.wikimedia.org/')]
icon_hash = None
-upstream_license = None
-upstream_prefix = 'http://konect.cc/files'
-files = [('download.tsv.wikipedia-discussions-de.tar.bz2', None, 'konect')]
+upstream_license = ("Creative Commons Attribution-ShareAlike 4.0 (CC BY-SA 4.0) and GFDL",
+ "https://dumps.wikimedia.org/legal.html")
+# Too large to process on an ordinary machine (see --skip-huge): the two
+# dumps are compressed 5.8 GB and 11.6 GB
+huge = True
-def postprocess(alt, g):
- assert (g.ep.weight.a == 1).all()
- del g.ep["weight"]
+# The dump of 2015 matches the time span of the replies first extracted with
+# this method (up to 11 February 2015, a cutoff fixed in the original
+# scripts); it is no longer on dumps.wikimedia.org, but in the Internet
+# Archive.
+archive_prefix = 'https://archive.org/download/dewiki-20150211'
+files_2015 = [('dewiki-20150211-pages-meta-current.xml.bz2', '2015', None)]
+# Fixed dump date. The legacy "pages-meta-current" XML dumps stop being
+# published in January 2027 (and are unavailable from the middle of each
+# month until then), so the current network is extracted from their
+# replacement, the "MediaWiki content current" exports, which are split in
+# files by page ranges (not predictable, hence listed), and kept only for the
+# last three months: they must be fetched before the end of 2026.
+DUMP_DATE = "2026-10-01"
+upstream_prefix = 'https://dumps.wikimedia.org/other/mediawiki_content_current'
+files_2026 = [(tuple(f"dewiki/{DUMP_DATE}/xml/bzip2/dewiki-{DUMP_DATE}-{r}.xml.bz2"
+ for r in ["p1p3436160", "p3436161p7812035",
+ "p7812042p11358706", "p11358707p13992174"]),
+ '2026', None)]
+files = files_2015 + files_2026
+# Comments are left out if dated before Wikipedia existed, or after the dump
+EARLIEST = calendar.timegm((2001, 1, 15, 0, 0, 0))
+LATEST = {"2015": calendar.timegm((2015, 2, 12, 0, 0, 0)),
+ "2026": calendar.timegm((2026, 10, 2, 0, 0, 0))}
def fetch_upstream(force=False):
- return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
- force)
+ name = __name__.split(".")[-1]
+ retry_download(fetch_upstream_files, name, archive_prefix, files_2015,
+ force)
+ retry_download(fetch_upstream_files, name, upstream_prefix, files_2026,
+ force)
+
+# For checks only: if set, only (about) this many replies are read per network
+_check_rows = None
+
+MONTHS = {
+ b"Mai": 5, b"May": 5, b"Jan": 1, b"J\xc3\xa4n": 1, b"Feb": 2, b"Febr": 2,
+ b"Mar": 3, b"M\xc3\xa4r": 3, b"Apr": 4, b"Jun": 6, b"Jul": 7, b"Aug": 8,
+ b"Sep": 9, b"Sept": 9, b"Okt": 10, b"Oct": 10, b"Nov": 11, b"Dez": 12,
+ b"Dec": 12, b"Januar": 1, b"J\xc3\xa4nner": 1, b"Februar": 2, b"Marz": 3,
+ b"M\xc3\xa4rz": 3, b"Maerz": 3, b"Mrz": 3, b"April": 4, b"Juni": 6,
+ b"June": 6, b"Juli": 7, b"August": 8, b"September": 9, b"Oktober": 10,
+ b"November": 11, b"Dezember": 12, b"December": 12}
+
+# The patterns of the original extraction (Perl, on bytes); the leading ".*"
+# selects the last match of the line
+re_date = re.compile(rb".*\d\d?:\d\d, (\d\d?)\. ([A-Z]\S+[a-z])\.? (\d\d\d?\d?)")
+re_time = re.compile(rb"(\d\d?):(\d\d)$")
+re_tz = re.compile(rb"\s*\(([A-Za-z]+)\)")
+re_unsigned = re.compile(rb"\{\{[Uu]nsign(?:ed|iert)\|[^|}]*[|}]")
+re_user = re.compile(rb".*\[\[:?(?:w:)?(?:..:)?(?:[Bb]enutzer(?:in)?|[Uu]ser):([^\]\|:\/]*)(?:\]\]|\||\/)")
+re_ip = re.compile(rb"\d\d?\d?\.\d\d?\d?\.\d\d?\d?\.\d\d?\d?")
+re_heading = re.compile(rb"^=+\s*|\s*=+$")
+re_text = re.compile(rb"^<text[^>]*>(.*)")
+re_space = re.compile(r"[\s_]+")
+
+TZ_OFFSET = {b"CET": 1, b"MEZ": 1, b"CEST": 2, b"MESZ": 2, b"UTC": 0, b"GMT": 0}
+BERLIN = zoneinfo.ZoneInfo("Europe/Berlin")
+
+def comment_date(s, dump_year):
+ """The day of the last signature date of line `s` (as Unix time, at
+ midnight UTC, as in the original extraction), and the time of that
+ signature (Unix time), or None if there is none; False if the date is
+ invalid (the comment is then dropped, as in the original)."""
+ m = re_date.match(s)
+ if m is None:
+ return None
+ month = MONTHS.get(m.group(2))
+ if month is None:
+ return None
+ d, y = int(m.group(1)), int(m.group(3))
+ # Years as interpreted by Perl's timegm()
+ if y < 100:
+ y += 2000 if 2000 + y <= dump_year + 50 else 1900
+ elif y < 1000:
+ y += 1900
+ try:
+ day = datetime(y, month, d, tzinfo=timezone.utc)
+ except ValueError:
+ return False
+ t0 = int(day.timestamp())
+ t = t0
+ hm = re_time.search(s, 0, m.start(1) - 2)
+ if hm is not None:
+ H, M = int(hm.group(1)), int(hm.group(2))
+ if H < 24 and M < 60:
+ tz = re_tz.match(s, m.end())
+ off = TZ_OFFSET.get(tz.group(1)) if tz is not None else None
+ if off is not None:
+ t = t0 + (H - off) * 3600 + M * 60
+ else:
+ local = datetime(y, month, d, H, M, tzinfo=BERLIN)
+ t = int(local.timestamp())
+ return t0, t
+
+def user_name(s):
+ "Normalized user name, as a MediaWiki page title."
+ s = html.unescape(s.decode("utf8", errors="replace"))
+ s = re_space.sub(" ", s.split("#")[0]).strip()
+ return s[:1].upper() + s[1:]
+
+def parse_discussions(f, latest, dump_year, max_edges=None):
+ """Reply network from the decompressed pages-meta-current XML dump `f` of
+ the German Wikipedia, following the extraction of N. Geilen and
+ J. Kunegis (konect-extr, extr/wikipedia-discussions: filter_discussion,
+ extract_meta, time_to_unix, approx_date, mk_answer_table), streamed line
+ by line."""
+ users = {}
+ names = []
+ pages = []
+ threads = {"": 0}
+ thread_names = [""]
+ sig_names = {} # signature -> normalized user name
+ e_s = array.array("i")
+ e_t = array.array("i")
+ e_time = array.array("q")
+ e_inferred = array.array("b")
+ e_page = array.array("i")
+ e_thread = array.array("i")
+
+ run = [] # current run of comments: (indent, user, day, time)
+
+ def flush():
+ # approx_date: comments without date take the last date of the lower
+ # indentation levels, up to the first gap
+ if not run:
+ return
+ STOP = "stop"
+ dates = []
+ kept = []
+ for indent, user, day, t in run:
+ while len(dates) < indent + 2:
+ dates.append(None)
+ dates[indent] = (day, t) if day else None
+ dates[indent + 1] = STOP
+ inferred = False
+ if not day:
+ for x in dates:
+ if x is STOP:
+ break
+ if x:
+ day, t = x
+ inferred = True
+ if user and day and EARLIEST < day < latest:
+ kept.append((indent, user, t, inferred))
+ run.clear()
+ # mk_answer_table: a comment replies to the closest preceding one
+ # with lower indentation, or with indentation zero
+ for i in range(1, len(kept)):
+ indent, user, t, inferred = kept[i]
+ for j in range(i - 1, -1, -1):
+ jindent, juser = kept[j][:2]
+ if jindent < indent or jindent == 0:
+ for u in (user, juser):
+ if u not in users:
+ users[u] = len(names)
+ names.append(u)
+ e_s.append(users[user])
+ e_t.append(users[juser])
+ e_time.append(t)
+ e_inferred.append(inferred)
+ e_page.append(len(pages) - 1)
+ e_thread.append(thread)
+ break
+
+ in_page = False
+ is_text = False
+ indentation_last = 0
+ thread = 0
+ for line in f:
+ if not in_page:
+ if b"<title>Diskussion:" not in line:
+ continue
+ in_page = True
+ flush()
+ s = line.strip()
+ if not is_text:
+ if s.startswith(b"<title>"):
+ pages.append(html.unescape(s[7:-8].decode("utf8", errors="replace")))
+ thread = 0 # no section yet
+ elif s == b"</page>":
+ flush()
+ in_page = False
+ if max_edges is not None and len(e_s) >= max_edges:
+ break
+ continue
+ m = re_text.match(s)
+ if m is None:
+ continue
+ flush()
+ if s.endswith(b"/>"): # empty page
+ continue
+ is_text = True
+ s = m.group(1)
+ close_text = False
+ if s.endswith(b"</text>"):
+ close_text = True
+ s = s[:-7]
+
+ if len(s) >= 4 and s.startswith(b"==") and s.endswith(b"=="):
+ flush()
+ tname = html.unescape(re_heading.sub(b"", s).decode("utf8", errors="replace"))
+ thread = threads.get(tname)
+ if thread is None:
+ thread = threads[tname] = len(thread_names)
+ thread_names.append(tname)
+
+ is_comment = False
+ indent = 0
+ if s.startswith(b":"):
+ indent = len(s) - len(s.lstrip(b":"))
+ is_comment = True
+ if s.startswith(b"--"):
+ indent = indentation_last
+ is_comment = True
+ indentation_last = indent
+
+ date = comment_date(s, dump_year)
+
+ user = None
+ if re_unsigned.search(s) is None:
+ m = re_user.match(s)
+ if m is not None:
+ is_comment = True
+ sig = m.group(1)
+ if sig[:1].isspace():
+ sig = sig[1:]
+ if sig[-1:].isspace():
+ sig = sig[:-1]
+ if re_ip.search(sig) is None:
+ user = sig_names.get(sig)
+ if user is None:
+ user = sig_names[sig] = user_name(sig)
+
+ if is_comment and date is not False:
+ day, t = date if date else (None, None)
+ run.append((indent, user, day, t))
+
+ if close_text:
+ flush()
+ is_text = False
+ flush()
+
+ g = Graph(directed=True)
+ g.add_vertex(len(names))
+ g.add_edge_list(numpy.column_stack((numpy.frombuffer(e_s, dtype="int32"),
+ numpy.frombuffer(e_t, dtype="int32"))))
+ g.vp.name = g.new_vp("string", vals=names)
+ g.ep.time = g.new_ep("int64_t")
+ g.ep.time.a = numpy.frombuffer(e_time, dtype="int64")
+ g.ep.inferred_time = g.new_ep("bool")
+ g.ep.inferred_time.a = numpy.frombuffer(e_inferred, dtype="int8")
+ g.ep.page = g.new_ep("string", vals=(pages[i] for i in e_page))
+ g.ep.thread = g.new_ep("string", vals=(thread_names[i] for i in e_thread))
+ return g
@cache_network()
@coerce_props()
@annotate()
def parse(alts=None):
- global files
name = __name__.split(".")[-1]
- for fname, alt, fmt in files:
+ for fnames, alt, fmt in files:
if alts is not None and alt not in alts:
continue
- with open_upstream_file(name, fname, "rb") as f:
- g = parse_graph([f], fmt, directed=None)
- postprocess(alt, g)
+ if isinstance(fnames, str):
+ fnames = [fnames]
+ def lines():
+ for fn in fnames:
+ with open_upstream_file(name, fn, "rb") as fs, bz2.open(fs) as f:
+ yield from f
+ g = parse_discussions(lines(), LATEST[alt], int(alt), _check_rows)
yield alt, g
[entries] New entry: wikipedia_discussions_de 2fb4913
diff --git a/entries/wikipedia_discussions_de/__init__.py b/entries/wikipedia_discussions_de/__init__.py
new file mode 100644
index 0000000..9de3d15
--- /dev/null
+++ b/entries/wikipedia_discussions_de/__init__.py
@@ -0,0 +1,59 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2026 Tiago de Paula Peixoto <tiago@skewed.de>
+#
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option) any
+# later version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along
+# with this program. If not, see <http://www.gnu.org/licenses/>.
+
+
+
+from .. import *
+
+
+title = 'German Wikipedia discussion replies (2001-2015)'
+description = """A directed network of replies between the users of the German Wikipedia in the discussion threads of the talk pages, as extracted by KONECT. Each node is a user, and each timestamped directed edge from user A to user B is a comment of A in reply to a comment of B. Users may reply to themselves. The data identify the users only by indices.
+
+The network has the following properties:
+
+- Edge `time`: the time of the reply, in Unix time."""
+tags = ['Social', 'Online', 'Communication', 'Unweighted', 'Multigraph', 'Timestamps']
+url = 'http://konect.cc/networks/wikipedia-discussions-de'
+citation = [('J. Kunegis, "Wikipedia threads (de) network dataset." KONECT, the Koblenz Network Collection (2017)', 'http://konect.cc/networks/wikipedia-discussions-de'),
+ ('J. Kunegis, "KONECT -- The Koblenz Network Collection." Proceedings of the 22nd International Conference on World Wide Web Companion, 1343-1350 (2013)', 'https://doi.org/10.1145/2487788.2488173')]
+icon_hash = None
+upstream_license = None
+upstream_prefix = 'http://konect.cc/files'
+files = [('download.tsv.wikipedia-discussions-de.tar.bz2', None, 'konect')]
+
+def postprocess(alt, g):
+ assert (g.ep.weight.a == 1).all()
+ del g.ep["weight"]
+
+def fetch_upstream(force=False):
+ return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
+ force)
+
+@cache_network()
+@coerce_props()
+@annotate()
+def parse(alts=None):
+ global files
+ name = __name__.split(".")[-1]
+ for fname, alt, fmt in files:
+ if alts is not None and alt not in alts:
+ continue
+ with open_upstream_file(name, fname, "rb") as f:
+ g = parse_graph([f], fmt, directed=None)
+ postprocess(alt, g)
+ yield alt, g