One of the main problems with regular port scans on a large network is the overhead: Depending on the number of servers, the list of open ports can quickly scale to a point that is impossible to track manually. Watching the whole network requires careful, and thus very time consuming, scrutiny of the scan results, and in many cases, you will not have the staff to expend the necessary attention. Additionally, many organizations lack a precisely defined, centrally available target state with which to compare the results, especially if individual departments are allowed to set up their own servers and services.

If you use different scanner locations to analyze your networks (e.g., one location within the organization and one across the Internet), you also face the problem of interpreting different lists of results. Inconsistencies arise because of connections blocked by firewalls and scans occurring at the different times. Reconciling the results requires manual evaluation.

An open source tool known as Dr. Portscan can help you automate distributed port scans and their evaluation. Dr. Portscan is a delta reporting tool that can assist almost any port scanning tool in analyzing parallel or staggered scans of arbitrarily overlapping networks. The results of these scans are automatically aggregated and compared. Changes discovered in the process can be added to reports for various target groups or passed into scripts as parameters. This approach opens up advanced diagnostic options, including the ability to compare the results with a defined target state.

In this article, we examine the general structure of distributed port scan infrastructures, the architecture and functionality of Dr. Portscan, and the basic steps for commissioning Dr Portscan to keep watch over your network.

Distributed Port Scans

If you are looking for authoritative results from the port scanning process, you'll need to ask yourself the following questions:

• What port scanning tools will you deploy? In addition to the classic Nmap tool, you will find numerous other free and commercial scanners. Security and network management tools can deliver information about open ports either explicitly or implicitly and thus serve as data sources.
• How many systems will run port scans? Multiple port scanners at different locations let you differentiate between internal and external views of the networks you analyze.
• How often will you launch scans from these systems? You need to take into consideration that some Internet-based port scanning services only envisage fixed intervals.
• What is the scope of the scan? External port scans are typically slower than internal LAN scans. For this reason, you might want to restrict scans to individual subnets, or some other subset of all possible TCP/UDP ports.
• How can you validate the ports detected as being open; that is, how can you compare this with a target state? Besides an initial scan with subsequent manual checks, you might be able to retrieve a target state documented in an asset database or a configuration management system.

Figure 1 shows a somewhat simplified view of a typical scenario for using Dr. Portscan.

Figure 1: Sample scenario for distributed port scans with Dr. Portscan.

The scripts on which Dr. Portscan is based let you compare an arbitrary number of port scan result lists created by different computers. (The popular Ndiff tool lets you compare exactly two Nmap scan results.)

Figure 2 shows the system architecture of Dr. Portscan with its most important components. Data sources (1) can be any port scanning tools. Agents (2) are responsible for converting the product-specific scan results into a uniform format. The input agent for Nmap, for example, contains parsers for XML and grepable Nmap output as of version 5. The overhead for implementing additional input agents depends on the port scanning tool you need to integrate, but it is typically reasonable because the code segments you need to modify simply consist of a list of the detected IP addresses and open ports. The delta reporter (3) transfers the results delivered by the input agents to a database (4), which stores and evaluates the results of all port scans. The information generated in the process indicates changes compared with recent scans.

Figure 2: Dr. Portscan system architecture.

Freely definable output agents (5) can provide an overview, create reports about changes on the individual subnets, mail information to the administrator in charge, or trigger arbitrary scripts to retrieve more information about the service hiding behind the newly opened port.

Dr. Portscan is designed to serve as a centralized relational database, initially to collect the results of the decentralized port scans in a uniform format. Because the input agents, the delta reporter, and the output agents are all implemented in Perl, you can use a wide spectrum of open source and commercial database products. The minimal sample configuration relies on SQLite. This sample configuration makes it easy to try out Dr. Portscan without having to set up a database server and a new database. For larger installations, it makes sense to use MariaDB, PostgreSQL, or some other relational database management system for which the Perl DBI modules are available.

The database stores all the information for every scanned machine and every port that has drawn attention to itself. The most important thing to watch is the changes compared with the last port scan run for the same entry. Table 1 shows the supported change types.

Data Sources and Input Agents

The port scanning tools (or data sources, as they are known in Dr. Portscan) are initially configured independently of the input agents. For example, you could automatically run nmap as a cronjob once a day on a Linux machine to scan all the ports in a network range equivalent to a legacy Class B network:

/usr/bin/nmap -v -v -oG - -p 1-65535 10.1.0.0/16 > /tmp/nmap-results.txt

Each data source is assigned to an input agent; its main task is to generate a list of all IP address and port pairs detected as open by the data source. The agent reads and parses the output from the scan tool. The preceding call to Nmap would return the following text output on a web server:

Host: 10.1.123.123 (www.example.com) \
Ports: 80/open/tcp//http///, 443/open/tcp//https///    Ignored State: filtered

The results from the input agent – a simple text file with an IP address, a port, the protocol details, and an optional timestamp in each line – are then either deposited for the delta reporter to pick up or pushed to another system (e.g., using scp). This intermediate step is often necessary because the SQLite database used by Dr. Portscan in the default configuration (without add-on software such as cubeSQL) is not accessible on the network and because other databases are often hardened against Internet access by a firewall, thus preventing direct entry of the results in a central database.

Delta Reporter, Output Agents, and Reports

The delta reporter first transfers the port scan results returned by the input agents to the central database. The reporter checks for each IP address and port pair, taking into consideration the Layer 4 protocol, whether an entry for the current scanner already exists, or whether a new entry needs to be created. When existing entries are updated, the appropriate change type is set.

After all the port scan results have been imported, the output agent can evaluate the central data repository. Because changes compared with the status quo are of primary interest, output agents typically only look for database entries with specific change types. To prevent an output agent from processing the same entry multiple times, the agent enters its own identifier in the corresponding column of the database table.

To suit your own needs, you should modify the number and type of output agents. Dr. Portscan offers a preconfigured output agent that gives you an overview of all changes since the last report, but you can restrict evaluation by the output agents in a targeted way. For example, you might just want to see the results for specific subnets or specific events (e.g., newly opened ports) to compare these results with a defined target state. The box titled "Cleanup Tips" describes some steps for ensuring a readable result.