History of dbpedia_link

The changes to the source code of this entry in the repository, newest first, each with the differences it made to the entry's files. They are also listed on GitLab.

2026-10-25 [entries] Merge duplicate entries and add cross-references between related ones by Tiago de Paula Peixoto 9bc6033

- Removed duplicates, merged into the entries holding the same data:
  dblp_coauthor_snap (soc_net_comms), cattle and bison (dom; bison's
  breeding data move there, as do the wolves' age and sex),
  football_tsevans (football, with both conference assignments),
  eu_procurements_alt (eu_procurements), foursquare_friendships
  (foursquare_global).
- software_dependencies: colt and scolt are one network; ambassador: the
  duplicate 2001 network is dropped and the three same-edge networks
  combined; dom: the dataset count corrected (436).
- macaque_neural: Young's 71-area network added (young71; the 47-area one
  is now cocomac47), both from archived copies.
- collins_yeast, celegans_interactomes: from archived copies of the dead
  CCSB site (byte-identical files).
- Cross-references between entries holding versions of the same data
  (food webs, Wikipedia, Slashdot, physicians, SNAP/Mislove, CCSB).
- ICON ids of duplicate records added; macaque_neural's corrected.
- prosper, dbpedia_all: string properties given in edge-index order (needs
  graph-tool with the fixed vals ordering).

diff --git a/entries/dbpedia_link/__init__.py b/entries/dbpedia_link/__init__.py
index 83654d6..196be36 100644
--- a/entries/dbpedia_link/__init__.py
+++ b/entries/dbpedia_link/__init__.py
@@ -25,7 +25,7 @@ import urllib.parse
 title = "English Wikipedia hyperlinks (DBpedia 3.9, 2013)"
 description = """The network of hyperlinks among the articles of the English Wikipedia, as extracted by DBpedia, a project that extracts structured information from Wikipedia. The data are the "Wikipedia Pagelinks" dataset of version 3.9 of DBpedia, which was extracted from the English Wikipedia dumps of late March / early April 2013 and published in September 2013; it contains the internal links (wikilinks) found in the text of the articles. Nodes are Wikipedia pages, and a directed edge $(i,j)$ means that page $i$ contains at least one link to page $j$, weighted by the number of links from $i$ to $j$ in the dataset (the dataset has one triple per link occurrence, i.e. 172,308,906 triples for about 136.6 million distinct pairs of pages). Links of pages to themselves are kept as self-loops.
 
-The pages include redirect pages, pages that do not exist (links to missing articles), and pages in other namespaces: in particular, the category memberships of the articles appear as links to the category pages (`Category:...`, about 16.6 million triples), and embedded images as links to file pages (`File:...` or `Image:...`, about 2.7 million triples).
+The pages include redirect pages, pages that do not exist (links to missing articles), and pages in other namespaces: in particular, the category memberships of the articles appear as links to the category pages (`Category:...`, about 16.6 million triples), and embedded images as links to file pages (`File:...` or `Image:...`, about 2.7 million triples). The network `en` of the entry `wikipedia_links_2013` was extracted from a slightly earlier dump (February 2013) with different processing: redirects are resolved to their targets, and redirect pages, pages in other namespaces and missing pages are excluded.
 
 The network has the following properties:

2026-10-25 [entries] Move more KONECT-derived entries to their original sources by Tiago de Paula Peixoto bf9dc0e

The entries now download the data from where KONECT obtained them (live,
or via the Wayback Machine), using the same release as KONECT where it is
still available, and gain the metadata KONECT dropped; descriptions and
citations are rewritten from the original sources:

- Linton Freeman's UCINET data page: bible_nouns, bison, cattle, hens,
  highschool, kangaroo, macaques, moreno_sheep, physician_trust (the three
  relations and the physicians' attributes), rhesus_monkey,
  train_terrorists, windsurfers (now the networks `interaction` and
  `closeness`), college_freshmen, crime, interactome_yeast (from the
  primary source), moreno_taro (undirected), residence_hall, zebras
- other pages, archived: contiguous_usa, advogato, google, github,
  digg_reply, twitter, bookcrossing, dbtropes_feature (without KONECT's
  spurious "ItemName" tropes)
- movielens_100k (in fact the MovieLens 10M tag assignments)
- dblp release 2017-05-02: dblp_author_paper, dblp_cite, dblp_coauthor
- DBpedia 3.9: dbpedia_country, dbpedia_genre, dbpedia_location,
  dbpedia_occupation, dbpedia_producer, dbpedia_recordlabel,
  dbpedia_starring, dbpedia_team, dbpedia_writer, dbpedia_all, dbpedia_link
- Discogs releases dump of October 2015: discogs_affiliation,
  discogs_genre and discogs_label (now with genre and style networks)

diff --git a/entries/dbpedia_link/__init__.py b/entries/dbpedia_link/__init__.py
index b2a937c..83654d6 100644
--- a/entries/dbpedia_link/__init__.py
+++ b/entries/dbpedia_link/__init__.py
@@ -17,33 +17,98 @@
 # with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 from .. import *
+import array
+import bz2
+import re
+import urllib.parse
 
-title = "en-Wikipedia (DBpedia) (2008)"
-description = """The network of hyperlinks among English wikipedia pages, as extracted in DBpedia in 2008.[^icon]
-[^icon]: Description obtained from the [ICON](https://icon.colorado.edu) project."""
-tags = ['Informational', 'Web graph', 'Unweighted']
-url = 'http://konect.cc/networks/dbpedia-link'
-citation = [('S. Auer et al. "DBpedia: A nucleus for a web of open data." In Proc. Int. Semantic Web Conf. (ISWC), 722-735 (2008).', 'https://doi.org/10.1007/978-3-540-76298-0_52')]
+title = "English Wikipedia hyperlinks (DBpedia 3.9, 2013)"
+description = """The network of hyperlinks among the articles of the English Wikipedia, as extracted by DBpedia, a project that extracts structured information from Wikipedia. The data are the "Wikipedia Pagelinks" dataset of version 3.9 of DBpedia, which was extracted from the English Wikipedia dumps of late March / early April 2013 and published in September 2013; it contains the internal links (wikilinks) found in the text of the articles. Nodes are Wikipedia pages, and a directed edge $(i,j)$ means that page $i$ contains at least one link to page $j$, weighted by the number of links from $i$ to $j$ in the dataset (the dataset has one triple per link occurrence, i.e. 172,308,906 triples for about 136.6 million distinct pairs of pages). Links of pages to themselves are kept as self-loops.
+
+The pages include redirect pages, pages that do not exist (links to missing articles), and pages in other namespaces: in particular, the category memberships of the articles appear as links to the category pages (`Category:...`, about 16.6 million triples), and embedded images as links to file pages (`File:...` or `Image:...`, about 2.7 million triples).
+
+The network has the following properties:
+
+- `name`: the name of the DBpedia resource, i.e. the title of the corresponding English Wikipedia page (with underscores instead of spaces), with the percent-encoding of the URI decoded.
+- `type`: the most specific class of the resource in the DBpedia ontology, from the "Mapping-based Types" dataset of the same release (empty if the resource has no type, e.g. because it has no infobox mapped to the ontology).
+- Edges: `count`, the number of links from the source page to the target page."""
+tags = ['Informational', 'Web graph', 'Weighted', 'Metadata']
+url = 'https://downloads.dbpedia.org/3.9/en/'
+citation = [('S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, Z. Ives, "DBpedia: A nucleus for a web of open data." In The Semantic Web (ISWC 2007 + ASWC 2007), Lecture Notes in Computer Science 4825, 722-735 (2007)', 'https://doi.org/10.1007/978-3-540-76298-0_52'),
+            ('J. Lehmann et al., "DBpedia - A large-scale, multilingual knowledge base extracted from Wikipedia." Semantic Web 6(2), 167-195 (2015)', 'https://doi.org/10.3233/SW-140134')]
 icon_hash = '57ec18278cf714f319c38ebd'
-upstream_prefix = 'http://konect.cc/files'
-files = [('download.tsv.dbpedia-link.tar.bz2', None, 'konect')]
+upstream_license = ("Creative Commons Attribution-ShareAlike 3.0 (CC BY-SA 3.0)",
+                    "https://creativecommons.org/licenses/by-sa/3.0/")
+# Too large to process on an ordinary machine (see --skip-huge)
+huge = True
+upstream_prefix = 'https://downloads.dbpedia.org/3.9/en'
+files = [(('page_links_en.nt.bz2', 'instance_types_en.nt.bz2'), None, None)]
 
 def fetch_upstream(force=False):
     return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
                                 force)
 
+_res = b"<http://dbpedia.org/resource/"
+_ntesc = re.compile(r"\\u([0-9A-Fa-f]{4})|\\U([0-9A-Fa-f]{8})|\\(.)")
+_ntchr = {"t": "\t", "n": "\n", "r": "\r", "b": "\b", "f": "\f"}
+
+def _unescape(s):
+    """Decode the escapes of N-Triples strings and URIs."""
+    return _ntesc.sub(lambda m: (chr(int(m.group(1) or m.group(2), 16))
+                                 if m.group(3) is None
+                                 else _ntchr.get(m.group(3), m.group(3))), s)
+
+def _resource_name(uri):
+    """Name of a DBpedia resource, from its N-Triples URI token."""
+    return urllib.parse.unquote(_unescape(uri[len(_res):-1].decode("utf-8")))
+
+def read_types(f, uris):
+    """Most specific DBpedia ontology class of each resource in uris (the
+    first one listed in the file, which is the most specific one)."""
+    rdftype = b" <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://dbpedia.org/ontology/"
+    types = {}
+    for line in f:
+        i = line.find(rdftype)
+        if i < 0:
+            continue
+        s = line[:i]
+        if s not in uris or s in types:
+            continue
+        types[s] = line[i + len(rdftype):].split(b">", 1)[0].decode()
+    return types
+
 @cache_network()
-@coerce_props()
+@coerce_props(vforce={"name": "string", "type": "string"})
 @annotate()
 def parse(alts=None):
-    global files
     name = __name__.split(".")[-1]
-    for fnames, alt, fmt in files:
-        if alts is not None and alt not in alts:
-            continue
-        if isinstance(fnames, str):
-            fnames = [fnames]
-        with ExitStack() as stack:
-            fs = [stack.enter_context(open_upstream_file(name, fn, "rb")) for fn in fnames]
-            g = parse_graph(fs, fmt, directed=True)
-        yield alt, g
+    (flinks, ftypes), alt, fmt = files[0]
+    pred = b" <http://dbpedia.org/ontology/wikiPageWikiLink> "
+    nodes = {}
+    edges = array.array("l")
+    with open_upstream_file(name, flinks, "rb") as f:
+        with bz2.open(f) as bf:
+            for line in bf:
+                i = line.find(pred)
+                if i < 0:
+                    continue
+                s = line[:i]
+                o = line[i + len(pred):].rstrip()[:-1].rstrip()
+                edges.append(nodes.setdefault(s, len(nodes)))
+                edges.append(nodes.setdefault(o, len(nodes)))
+    with open_upstream_file(name, ftypes, "rb") as f:
+        with bz2.open(f) as bf:
+            types = read_types(bf, nodes)
+    # Repeated links are merged, and counted
+    N = len(nodes)
+    edges = numpy.frombuffer(edges, dtype="int64").reshape(-1, 2)
+    key, count = numpy.unique(edges[:, 0] * N + edges[:, 1], return_counts=True)
+    del edges
+    g = Graph(directed=True)
+    g.add_vertex(N)
+    g.add_edge_list(numpy.column_stack((key // N, key % N)))
+    del key
+    g.ep.count = g.new_ep("int32_t", vals=count)
+    g.vp.name = g.new_vp("string", vals=[_resource_name(u) for u in nodes])
+    g.vp.type = g.new_vp("string", vals=[types.get(u, "") for u in nodes])
+    yield None, g

2026-10-24 Update copyright information by Tiago de Paula Peixoto 73a4f0d

diff --git a/entries/dbpedia_link/__init__.py b/entries/dbpedia_link/__init__.py
index d17a32b..b2a937c 100644
--- a/entries/dbpedia_link/__init__.py
+++ b/entries/dbpedia_link/__init__.py
@@ -1,7 +1,7 @@
 #! /usr/bin/env python
 # -*- coding: utf-8 -*-
 #
-# Copyright (C) 2020-2024 Tiago de Paula Peixoto <tiago@skewed.de>
+# Copyright (C) 2020-2026 Tiago de Paula Peixoto <tiago@skewed.de>
 #
 # This program is free software: you can redistribute it and/or modify it under
 # the terms of the GNU Affero General Public License as published by the Free

2024-05-27 Update copyright information by Tiago de Paula Peixoto 3d288ea

diff --git a/entries/dbpedia_link/__init__.py b/entries/dbpedia_link/__init__.py
index 19536e9..d17a32b 100644
--- a/entries/dbpedia_link/__init__.py
+++ b/entries/dbpedia_link/__init__.py
@@ -1,7 +1,7 @@
 #! /usr/bin/env python
 # -*- coding: utf-8 -*-
 #
-# Copyright (C) 2020-2023 Tiago de Paula Peixoto <tiago@skewed.de>
+# Copyright (C) 2020-2024 Tiago de Paula Peixoto <tiago@skewed.de>
 #
 # This program is free software: you can redistribute it and/or modify it under
 # the terms of the GNU Affero General Public License as published by the Free

2023-02-19 Update copyright information by Tiago de Paula Peixoto 8d005ae

diff --git a/entries/dbpedia_link/__init__.py b/entries/dbpedia_link/__init__.py
index 9f9e58e..19536e9 100644
--- a/entries/dbpedia_link/__init__.py
+++ b/entries/dbpedia_link/__init__.py
@@ -1,7 +1,7 @@
 #! /usr/bin/env python
 # -*- coding: utf-8 -*-
 #
-# Copyright (C) 2021 Tiago de Paula Peixoto <tiago@skewed.de>
+# Copyright (C) 2020-2023 Tiago de Paula Peixoto <tiago@skewed.de>
 #
 # This program is free software: you can redistribute it and/or modify it under
 # the terms of the GNU Affero General Public License as published by the Free

2021-06-22 Update copyright info by Tiago de Paula Peixoto db2efcc

diff --git a/entries/dbpedia_link/__init__.py b/entries/dbpedia_link/__init__.py
index 69f537c..9f9e58e 100644
--- a/entries/dbpedia_link/__init__.py
+++ b/entries/dbpedia_link/__init__.py
@@ -1,7 +1,7 @@
 #! /usr/bin/env python
 # -*- coding: utf-8 -*-
 #
-# Copyright (C) 2020 Tiago de Paula Peixoto <tiago@skewed.de>
+# Copyright (C) 2021 Tiago de Paula Peixoto <tiago@skewed.de>
 #
 # This program is free software: you can redistribute it and/or modify it under
 # the terms of the GNU Affero General Public License as published by the Free

2021-02-01 Un-orphan KONECT datasets by Tiago de Paula Peixoto 894eba5

KONECT has been brought back from the dead for the time being.

diff --git a/entries/dbpedia_link/__init__.py b/entries/dbpedia_link/__init__.py
index 2fd02ca..69f537c 100644
--- a/entries/dbpedia_link/__init__.py
+++ b/entries/dbpedia_link/__init__.py
@@ -26,7 +26,6 @@ url = 'http://konect.cc/networks/dbpedia-link'
 citation = [('S. Auer et al. "DBpedia: A nucleus for a web of open data." In Proc. Int. Semantic Web Conf. (ISWC), 722-735 (2008).', 'https://doi.org/10.1007/978-3-540-76298-0_52')]
 icon_hash = '57ec18278cf714f319c38ebd'
 upstream_prefix = 'http://konect.cc/files'
-orphan = True
 files = [('download.tsv.dbpedia-link.tar.bz2', None, 'konect')]
 
 def fetch_upstream(force=False):

2020-11-28 Update citations to use DOIs by Tiago de Paula Peixoto e878874

diff --git a/entries/dbpedia_link/__init__.py b/entries/dbpedia_link/__init__.py
index c1e921c..2fd02ca 100644
--- a/entries/dbpedia_link/__init__.py
+++ b/entries/dbpedia_link/__init__.py
@@ -23,7 +23,7 @@ description = """The network of hyperlinks among English wikipedia pages, as ext
 [^icon]: Description obtained from the [ICON](https://icon.colorado.edu) project."""
 tags = ['Informational', 'Web graph', 'Unweighted']
 url = 'http://konect.cc/networks/dbpedia-link'
-citation = [('S. Auer et al. "DBpedia: A nucleus for a web of open data." In Proc. Int. Semantic Web Conf. (ISWC), 722-735 (2008).', 'http://www.informatik.uni-leipzig.de/~auer/publication/dbpedia.pdf')]
+citation = [('S. Auer et al. "DBpedia: A nucleus for a web of open data." In Proc. Int. Semantic Web Conf. (ISWC), 722-735 (2008).', 'https://doi.org/10.1007/978-3-540-76298-0_52')]
 icon_hash = '57ec18278cf714f319c38ebd'
 upstream_prefix = 'http://konect.cc/files'
 orphan = True

2020-11-02 Add support for 'orphan' datasets and tag konect entries by Tiago de Paula Peixoto e330b80

diff --git a/entries/dbpedia_link/__init__.py b/entries/dbpedia_link/__init__.py
index 3712e1d..c1e921c 100644
--- a/entries/dbpedia_link/__init__.py
+++ b/entries/dbpedia_link/__init__.py
@@ -26,6 +26,7 @@ url = 'http://konect.cc/networks/dbpedia-link'
 citation = [('S. Auer et al. "DBpedia: A nucleus for a web of open data." In Proc. Int. Semantic Web Conf. (ISWC), 722-735 (2008).', 'http://www.informatik.uni-leipzig.de/~auer/publication/dbpedia.pdf')]
 icon_hash = '57ec18278cf714f319c38ebd'
 upstream_prefix = 'http://konect.cc/files'
+orphan = True
 files = [('download.tsv.dbpedia-link.tar.bz2', None, 'konect')]
 
 def fetch_upstream(force=False):

2020-09-15 Change license from GPLv3 to AGPLv3 by Tiago de Paula Peixoto d754be5

diff --git a/entries/dbpedia_link/__init__.py b/entries/dbpedia_link/__init__.py
index 334d9ef..3712e1d 100644
--- a/entries/dbpedia_link/__init__.py
+++ b/entries/dbpedia_link/__init__.py
@@ -3,18 +3,18 @@
 #
 # Copyright (C) 2020 Tiago de Paula Peixoto <tiago@skewed.de>
 #
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option) any
+# later version.
 #
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public License for more
+# details.
 #
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+# You should have received a copy of the GNU Affero General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 from .. import *

2020-09-13 Add explicit attribution to descriptions obtained from ICON by Tiago de Paula Peixoto 1645314

diff --git a/entries/dbpedia_link/__init__.py b/entries/dbpedia_link/__init__.py
index 2504bcd..334d9ef 100644
--- a/entries/dbpedia_link/__init__.py
+++ b/entries/dbpedia_link/__init__.py
@@ -19,7 +19,8 @@
 from .. import *
 
 title = "en-Wikipedia (DBpedia) (2008)"
-description = "The network of hyperlinks among English wikipedia pages, as extracted in DBpedia in 2008."
+description = """The network of hyperlinks among English wikipedia pages, as extracted in DBpedia in 2008.[^icon]
+[^icon]: Description obtained from the [ICON](https://icon.colorado.edu) project."""
 tags = ['Informational', 'Web graph', 'Unweighted']
 url = 'http://konect.cc/networks/dbpedia-link'
 citation = [('S. Auer et al. "DBpedia: A nucleus for a web of open data." In Proc. Int. Semantic Web Conf. (ISWC), 722-735 (2008).', 'http://www.informatik.uni-leipzig.de/~auer/publication/dbpedia.pdf')]

2020-08-11 Initial commit by Tiago de Paula Peixoto cc85872

diff --git a/entries/dbpedia_link/__init__.py b/entries/dbpedia_link/__init__.py
new file mode 100644
index 0000000..2504bcd
--- /dev/null
+++ b/entries/dbpedia_link/__init__.py
@@ -0,0 +1,48 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2020 Tiago de Paula Peixoto <tiago@skewed.de>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from .. import *
+
+title = "en-Wikipedia (DBpedia) (2008)"
+description = "The network of hyperlinks among English wikipedia pages, as extracted in DBpedia in 2008."
+tags = ['Informational', 'Web graph', 'Unweighted']
+url = 'http://konect.cc/networks/dbpedia-link'
+citation = [('S. Auer et al. "DBpedia: A nucleus for a web of open data." In Proc. Int. Semantic Web Conf. (ISWC), 722-735 (2008).', 'http://www.informatik.uni-leipzig.de/~auer/publication/dbpedia.pdf')]
+icon_hash = '57ec18278cf714f319c38ebd'
+upstream_prefix = 'http://konect.cc/files'
+files = [('download.tsv.dbpedia-link.tar.bz2', None, 'konect')]
+
+def fetch_upstream(force=False):
+    return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
+                                force)
+
+@cache_network()
+@coerce_props()
+@annotate()
+def parse(alts=None):
+    global files
+    name = __name__.split(".")[-1]
+    for fnames, alt, fmt in files:
+        if alts is not None and alt not in alts:
+            continue
+        if isinstance(fnames, str):
+            fnames = [fnames]
+        with ExitStack() as stack:
+            fs = [stack.enter_context(open_upstream_file(name, fn, "rb")) for fn in fnames]
+            g = parse_graph(fs, fmt, directed=True)
+        yield alt, g