SENSEI: Scraper for ENhanced AnalySis to Evaluate Illicit Trends

Abstract

Over the last years, we faced an exponential growth of illegal online market services in the Dark Web, making it easier than ever before of acquiring illicit goods online via a simple service interaction. To study and understand this emerging illegal services economy, we developed a trend analysis and (dark-)web services monitoring tool: SENSEI, which stands for ‘Scraper for ENhanced analySis to Evaluate Illicit trends’. SENSEI extracts specific service transaction trends and analyses the human behaviours behind, to produce symmetric insights on specific service transaction habits from both customers and vendors on the Dark Web. Moreover, a trend analysis tool is provided to discover and typify relationships among different criminal activities and hence provide evidence and support investigation activities and Law Enforcement Agencies (LEAs) detecting criminal operations.

Appendices

Requirement

In the following, we provide installation information and the interaction process between the SENSEI platform and the SENSEI SOA architecture.

1.1 Installation

The SENSEI architecture lays on Docker to build and run its components. Docker-compose is used to create an interconnected infrastructure, where the database can communicate with the SENSEI SOA tool and then with the SENSEI platform. The usage of a docker-compose eases the configuration settings, the building process, and the execution process. Indeed, to build and execute the entire framework, it is sufficient to run, the first time, from the main project folder, the following commands:

to create the network where the SENSEI framework operates and this command:

to build and execute the framework. The platform is available in the GitHub repository: https://github.com/danieldp92/SENSEI.

Services

1.1 Load Services

Table 1. Original dataset. The attribute name is an identifier. Instead age, gender and postcode are quasi-identifiers.

Load services manage the loading of datasets and the extraction of data from HTML pages passed as input. Table 1 shows the endpoint list of all LOAD services developed.

The upload dump endpoint reads the HTML page received and extract information to store into the database. As shown in Fig. 1, the tool has as many scrapers as the number of dark markets taken into account. This work takes into account three dark markets: Agartha, Cannazon and Dark Market.

The delete dumps is used to delete one or more markets scraped in the past. It is possible to set parameters as ‘timestamp’ and ‘market name’.

1.2 Get Services

Table 2. Get services.

Get services allow the end-user to effectively retrieve information stored during the loading process. The information retrieved by the GET services includes information like vendors and the product list and data selection to build the graph analysis by vendor’s name or PGP key. The services provided in this macro category can be grouped into four main categories:

• Market services: services used to get information regarding marketplaces, such as details of a specific marketplace stored, timestamp, size of the dump, and the number of pages.

• Vendor services: services used to retrieve information about vendors.

• Product services: services used to retrieve information about products.

• Graph analysis services: services used to show the connections of a vendor among marketplaces, leveraging the vendor’s username and the vendor’s PGP key.

In Table 2 we provided a detailed list of all the services under the GET category. In addition, we listed all the service names and a short description for each of them to explain their purpose.

Trend Analysis Platform

1.1 Home Page

Fig. 2.

SENSEI platform screen: home. It contains general info about the top vendors, a trend analysis of the last month recorder, general info for each country and a map containing the number of products sold in every country.

Table 3. Platform home services.

1.2 Trend Analysis Page

Fig. 3.

SENSEI platform screen: trend analysis. Comparison between two countries based on the amount of drug, counted as the total price, in all the markets.

Table 4. Platform trend analysis services.

Table 5. Platform trend analysis comparison services.

1.3 Vendor’s Tree-Map Page

In Table 6 we provide the list of services used to build the tree-map analysis. In the first row of the table, we have GET /vendor/treemap/n-products, the service is in charge of retrieving the number of products on sale per each vendor. Next, the service GET /market/list/ is used to show the market list as a treemap’s filtering option. The last service, GET /vendor/treemap/vendor-name extracts further details about the number of products on sale per each drug of a specific vendor.

Table 6. Platform treemap services.

1.4 Vendor’s Search Page

Fig. 4.

SENSEI platform screen: vendors general info with detailed information regarding a specific vendor.

In Table 7 are shown the two services used in the vendor search page. The first service is in charge of retrieving the general data of all the vendors. While, the second one GET /vendor/info/vendor-name retrieves the information of a specific vendor.

Table 7. Platform search vendor services.

1.5 Vendor-Market Graph Analysis

Table 8 shows the services used to build the interactive graph. First, GET /market/n-products/ is used to estimate the impact of the vendor in a market, retrieving the number of products sold. Next, GET /market/graph/ service provides the list of all the vendors from a specific market. Last, the service GET /market/graph/vendor/ provides additional information, like the number of markets connected to a vendor and the number of products for each market.

Table 8. Platform interactive graph services.