logo Netzschleuder network catalog, repository, and centrifuge

History of cisco_hosts

The changes to the source code of this entry in the repository, newest first, each with the differences it made to the entry's files. They are also listed on GitLab.

[entries] New entry: cisco_hosts by Tiago de Paula Peixoto f8ddff0
diff --git a/entries/cisco_hosts/__init__.py b/entries/cisco_hosts/__init__.py
new file mode 100644
index 0000000..405e5c3
--- /dev/null
+++ b/entries/cisco_hosts/__init__.py
@@ -0,0 +1,175 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2020-2026 Tiago de Paula Peixoto <tiago@skewed.de>
+#
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option) any
+# later version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+from .. import *
+
+import zipfile
+
+title = "Communication networks of computing hosts (Cisco Secure Workload, 2019-2022)"
+description = r"""Networks of communications among computing hosts running distributed applications in various enterprises, provided by the Cisco Secure Workload group (O. Madani, S. A. Averineni and S. Gandham) and distributed by SNAP, for research on the clustering of hosts by their function. Software sensors on the hosts recorded, in samples of one hour, the connections between hosts with the (service) port and protocol used and the number of packets exchanged. Nodes are hosts (IP addresses), and an edge points from the client (consumer) of a connection to its server (provider). The hosts and ports are anonymized as integers (separately for each network); the protocols are given by their numbers (6 for TCP, 17 for UDP). There are 27 networks:
+
+- `g1` to `g20`: 20 disjoint networks of different applications (from 86 to 278,739 hosts), recorded in 27 hourly samples on four days, one in January and three in March 2019 (6 or 7 samples per day).
+- `g21`: a small network (317 hosts), recorded in 96 hourly samples from 10 to 13 February 2022. For 59 of the hosts, which had a sensor and a host name, the data give a ground-truth grouping of the hosts by function (based on their host names, which are not given).
+- `g21_4days`, `g21_2days`, `g21_12hrs`: the connections of `g21` among the hosts with a ground-truth group, aggregated over 4 days, 2 days and 12 hours from 10 February 2022 (without the numbers of packets).
+- `g21_few_sensors_6days`, `g21_few_sensors_1day`: the same application observed with fewer than 10 sensors (instead of about 50), aggregated over 6 days and 1 day until 9 February 2022 (without the numbers of packets), so that only part of the network is observed.
+- `e1` (also called `g22`): a network of machines doing software builds and regression tests (33,241 hosts), recorded in 99 hourly samples between January 2021 and October 2022, with a candidate ground-truth grouping of 486 hosts (based on their host names).
+
+In the networks recorded in hourly samples (`g1` to `g21` and `e1`), each edge is a connection observed in one sample, so that repeated connections between the same hosts in different samples are separate edges (a multigraph). In the aggregated networks, each pair of hosts appears once. The dates and hours of the samples are not given exactly: the samples are identified by the names of their files (see `snapshots` below).
+
+The networks have the following properties:
+
+- `host_id`: the anonymized identifier of the host.
+- `g21` networks and `e1`: `group`, the index of the host's ground-truth group (starting from 1; -1 if none); `prefix_code`, the prefix code of the host name, which encodes the position of the host name in the prefix tree of all host names (e.g. "b1l8c2_b1l1c1": first branch, prefix of length 8, 2 hosts below, and so on; empty if none). `g21` networks: `group_prefix5`, the index of the host's group according to the grouping of the host names with a common prefix of at least 5 characters (-1 if none).
+- Edge `ports`: the ports and protocols of the connection, with the numbers of packets, as given, e.g. "1p6-22,1p17-4" (port 1 on protocol 6 with 22 packets, and port 1 on protocol 17 with 4 packets); for the aggregated networks, the numbers of packets are omitted.
+- Edge `packets` (sampled networks): the total number of packets of the connection.
+- Edge `snapshot` (sampled networks): the index (starting from 0) of the hourly sample in the graph property `snapshots`, which lists the names of the sample files (e.g. "out1_4" for the 4th sample of day 1, "202202130300" for 13 February 2022 at 03:00, "out2022_7_5" for the 5th sample of July 2022)."""
+tags = ['Technological', 'Communication', 'Multigraph', 'Timestamps', 'Unweighted', 'Metadata']
+url = ['https://snap.stanford.edu/data/cisco-networks.html']
+citation = [('O. Madani, S. A. Averineni, and S. Gandham, "A Dataset of Networks of Computing Hosts." Proceedings of the 2022 ACM International Workshop on Security and Privacy Analytics (IWSPA), 100-104 (2022).', 'https://doi.org/10.1145/3510548.3519368')]
+icon_hash = None
+upstream_license = None
+upstream_prefix = 'https://snap.stanford.edu/data'
+_zip = 'CiscoSecureWorkload_22_networks.zip'
+_d = 'Cisco_22_networks'
+_g21 = f'{_d}/dir_g21_small_workload_with_gt'
+_g21_gt = (f'{_zip}:{_g21}/groupings.gt.txt', f'{_zip}:{_g21}/groupings.gt.viaPrefix5.txt',
+           f'{_zip}:{_g21}/prefix_codes.txt')
+_e1_gt = (f'{_zip}:{_d}/dir_g22_extra_graph_with_gt/candidate_gt.minPrefix5.txt',
+          f'{_zip}:{_d}/dir_g22_extra_graph_with_gt/id_gt_path_code.txt')
+files = [((f'{_zip}:{_d}/README',), f'g{i}', None) for i in range(1, 21)]
+files += [((f'{_zip}:{_d}/README',) + _g21_gt, 'g21', None)]
+for _alt, _f in [('g21_4days', 'edges_4days_feb10thruFeb13_all_49sensors.csv.txt.gz'),
+                 ('g21_2days', 'edges_2days_feb10thruFeb11_all_49sensors.csv.txt.gz'),
+                 ('g21_12hrs', 'edges_12hrs_feb10_all_49sensors.csv.txt.gz'),
+                 ('g21_few_sensors_6days', 'dir_few_sensors/edges_6days_tillFeb9_partial_sensors.csv.txt.gz'),
+                 ('g21_few_sensors_1day', 'dir_few_sensors/edges_1days_tillFeb9_partial_sensors.csv.txt.gz')]:
+    files += [((f'{_zip}:{_g21}/dir_no_packets_etc/{_f}',) + _g21_gt, _alt, None)]
+files += [((f'{_zip}:{_d}/README',) + _e1_gt, 'e1', None)]
+
+# directories of the hourly samples of the networks
+_sample_dirs = {"g21": f"{_g21}/dir_includes_packets_and_other_nodes/",
+                "e1": f"{_d}/dir_g22_extra_graph_with_gt/dir_edges/"}
+
+def fetch_upstream(force=False):
+    return fetch_upstream_files(__name__.split(".")[-1], upstream_prefix, files,
+                                force)
+
+def _sample_files(name, alt):
+    "The members of the archive with the hourly samples of network `alt`."
+    with open_upstream_file(name, _zip, "rb", seekable=True) as f:
+        members = zipfile.ZipFile(f).namelist()
+    d = _sample_dirs.get(alt, f"{_d}/dir_20_graphs/")
+    fs = [m for m in members if m.startswith(d) and m.endswith(".txt.gz")]
+    def label(m):
+        b = m.split("/")[-1].removesuffix(".txt.gz").removesuffix(".anon")
+        return b.removeprefix("edges_to_ports_")
+    def key(m):
+        # order the samples numerically (e.g. "out2022_10_2" after "out2022_9_13")
+        return [int(x) for x in label(m).removeprefix("out").split("_")]
+    fs.sort(key=key)
+    return [(m, label(m)) for m in fs]
+
+def _read_lines(name, member, alt):
+    import gzip
+    with open_upstream_file(name, f"{_zip}:{member}", "rb") as f:
+        with gzip.open(f, "rt") as gf:
+            for line in gf:
+                if line.startswith("#"):
+                    continue
+                parts = line.split()
+                if len(parts) < 4 or parts[0] != alt:
+                    continue
+                yield int(parts[1]), int(parts[2]), parts[3]
+
+def _read_groups(name, fname):
+    groups = {}
+    i = 0
+    with open_upstream_file(name, fname, "r") as f:
+        for line in f:
+            line = line.strip()
+            if line == "" or line.startswith("#"):
+                continue
+            i += 1
+            for x in line.split(","):
+                groups[int(x)] = i
+    return groups
+
+def _read_codes(name, fname):
+    codes = {}
+    with open_upstream_file(name, fname, "r") as f:
+        for line in f:
+            if line.startswith("#") or not line.strip():
+                continue
+            x, code = line.split()
+            codes[int(x)] = code
+    return codes
+
+@cache_network()
+@coerce_props()
+@annotate()
+def parse(alts=None):
+    name = __name__.split(".")[-1]
+    for fnames, alt, fmt in files:
+        if alts is not None and alt not in alts:
+            continue
+        g = Graph(directed=True)
+        g.ep.ports = g.new_ep("string")
+        if fnames[0].endswith("README"):
+            g.ep.packets = g.new_ep("int64_t")
+            g.ep.snapshot = g.new_ep("int32_t")
+            samples = _sample_files(name, alt)
+            def edges():
+                for i, (m, label) in enumerate(samples):
+                    for u, v, ports in _read_lines(name, m, alt):
+                        packets = sum(int(p.split("-")[1]) for p in ports.split(",")
+                                      if "-" in p)
+                        yield u, v, ports, packets, i
+            g.vp.host_id = g.add_edge_list(edges(), hashed=True, hash_type="int",
+                                           eprops=[g.ep.ports, g.ep.packets,
+                                                   g.ep.snapshot])
+            g.gp.snapshots = g.new_gp("vector<string>",
+                                      val=[label for m, label in samples])
+        else:
+            member = fnames[0].split(":", 1)[1]
+            def edges():
+                for u, v, ports in _read_lines(name, member, "g21"):
+                    # no packet counts in these files (given as 1)
+                    yield u, v, ",".join(p.split("-")[0] for p in ports.split(","))
+            g.vp.host_id = g.add_edge_list(edges(), hashed=True, hash_type="int",
+                                           eprops=[g.ep.ports])
+        if alt.startswith("g21") or alt == "e1":
+            gt = fnames[1:]
+            g.vp.group = g.new_vp("int32_t", val=-1)
+            g.vp.prefix_code = g.new_vp("string")
+            ids = {g.vp.host_id[v]: v for v in g.vertices()}
+            if alt.startswith("g21"):
+                groups = _read_groups(name, gt[0])
+                groups5 = _read_groups(name, gt[1])
+                codes = _read_codes(name, gt[2])
+                g.vp.group_prefix5 = g.new_vp("int32_t", val=-1)
+            else:
+                groups = _read_groups(name, gt[0])
+                groups5 = {}
+                codes = _read_codes(name, gt[1])
+            for x, v in ids.items():
+                g.vp.group[v] = groups.get(x, -1)
+                g.vp.prefix_code[v] = codes.get(x, "")
+                if alt.startswith("g21"):
+                    g.vp.group_prefix5[v] = groups5.get(x, -1)
+        yield alt, g