Putting It All Together: Host Discovery Strategies

Effective scanning requires more than knowing all of the options described in this and previous sections. Users must understand how and when to use them to suit the target network topology and scanning goals.

In May 2009 we did some research to find the most effective ping probes. We tested dozens of different probes and host discovery options against thousands of Internet hosts. The results of this research are shown in Table 3.2, which ranks the different probes by effectiveness. Percentages are out of the number of hosts that responded to any of the probes. We tested 90 different probes; this is a subset.

Table 3.2. Best host discovery probes
Hosts foundProbe
31.17%-PU40125 --source-port 53 --data-length 24
29.96%-PU31338 --source-port 53 --data-length 24
29.05%-PU631 --source-port 53 --data-length 24
24.15%-PU53 --source-port 53 --data-length 24

We can pick out some interesting features from Table 3.2. The best single probe by far is -PE, ICMP echo. Following that are various SYN and ACK probes (-PS and -PA) to common ports. ICMP timestamp, -PP, does quite well. The best UDP probes (-PU) are those to random high ports, and they are more effective when combined with --source-port 53 --data-length 24. SYN and ACK to port 25 (SMTP) are notably worse than other TCP probes, perhaps because that port is commonly filtered. Various IP protocol pings (-PO) are not very effective, and -PM, the ICMP address mask request, comes in last.

The best multi-probe combinations are not found simply by picking entries from the top of Table 3.2. That's because some probes find many of the same hosts as another, so using both of them is little better than using just one. We had a computer program find the optimum combination for different numbers of probes, by examining how many hosts each probe found and how much they overlapped with other probes. The best combinations are in Table 3.3. Nmap's default four-probe combination was selected from this list as a compromise between comprehensiveness and speed.

Table 3.3. Best host discovery probe combinations
 Hosts foundProbe combination
1 probe62.47%-PE
2 probes77.61%-PE -PA80
3 probes83.83%-PE -PA80 -PS443
4 probes88.64%-PE -PA80 -PS443 -PP
5 probes91.12%-PE -PA80 -PS443 -PP -PU40125 --source-port 53
6 probes92.42%-PE -PS80 -PS443 -PP -PU40125 -PA3389 --source-port 53
7 probes93.10%-PE -PS80 -PS443 -PP -PU40125 -PS3389 -PA21 --source-port 53
8 probes93.69%-PE -PS80 -PS443 -PP -PU40125 -PS3389 -PA21 -PU161 --source-port 53

TCP probe and port selection

The TCP ping options are some of the most powerful discovery techniques in Nmap. An administrator may be able to get away with blocking ICMP echo request packets without affecting most users, but a server absolutely must respond to SYN packets sent to the public services it provides. Meanwhile, ACK packets often get through non-stateful firewalls. I would recommend using both of SYN and ACK probes, using lists of ports based on any knowledge you might have of the target networks as well as more generally popular ports. Table 3.4 lists the ports that are most likely to be responsive (open or closed) based on empirical testing (see the section called “What Are the Most Popular Ports?”). On hosts with a default-drop filter (the hardest type to reach), these are the ports most likely to elicit a response.

In addition to popular ports such as the ones in the list above, choosing at least one high-numbered port is recommended. Many poorly configured firewalls only have default-deny for the privileged ports, meaning those below 1,024. I usually pick a high numbered port out of the air, such as 40,000 or 10,042, to catch machines behind this sort of firewall.

In choosing the ports to probe, remember to emphasize platform diversity. If you are limiting your ping scan to two ports, HTTP (80) and SSH (22) are probably better than HTTP (80) and HTTPS (443) because the latter two are related web services, and many machines that have HTTPS will often have HTTP available anyway. Finding two accessible ports on the same machine is no better for ping scanning purposes than finding one. The goal is to choose ports so that a broad set of hosts will match at least one of them.

Note that the valuable port table does not include many client-oriented ports such as the ubiquitous Windows SMB port 135. The primary reason is that this table only looked at hosts behind default-deny firewalls, where the vast majority of ports are filtered. In those situations, Windows ports such as 135–139 and 445 are usually blocked. When these machines are not behind a firewall, the open ports are unimportant for ping scanning because the thousands of closed ports work just as well.

UDP port selection

In selecting UDP ports, remember that an open port is unlikely to respond to the probes. Unfiltered ports are desired. To avoid open ports, you might consider excluding common UDP services like DNS (port 53) and SNMP (161). On the other hand, firewall rules are often so broad that those probes (particularly to port 53) might get through and hit a closed port. So I would recommend choosing at least port 53 and an arbitrarily selected high-numbered port such as 37,452.

ICMP probe selection

For ICMP, the standard ping (echo request) is usually worth trying. Many administrators specifically allow this because it is useful for debugging or because RFC 1122 requires it. I would also use at least one of the address mask or timestamp requests. These are valuable for networks where administrators intentionally block echo request packets, but forget about other ICMP queries.

Designing the ideal combinations of probes

How all of these ping types are combined into a ping scan strategy depends on characteristics of the target network and on the scan goals. For internal networks, the default ping type usually works well. The default is also fine for most casual scanning, where missing an occasional host is no big deal. Adding more probes can help catch those occasional stealthy machines, at the expense of making the ping scan take a bit longer. Time taken is roughly proportional to the number of probes sent to each machine. For security scans of target networks over the Internet, adding more probes is usually advisable. Try to include a diverse set of the techniques discussed previously. Here is a set of ping options that should catch the vast majority of hosts: -PE -PP -PS21,22,23,25,80,113,443,31339 -PA80,113,443,10042. Adding in --source-port 53 might be worthwhile as well. How much better will the results be, and how much longer will it take? That depends on the target network, of course, but the Nmap random target selection option (-iR) makes it easy to perform a quick test. Example 3.13 shows Nmap generating 50,000 random IP addresses and then performing a default ping scan. You should remember that the default is four probes: an ICMP echo request, a TCP SYN to port 443, a TCP ACK to port 80, and an ICMP timestamp request.

Example 3.13. Generating 50,000 IP addresses, then ping scanning with default options
# nmap -n -sL -iR 50000 | awk '/^Host / {print $2}' | sort -n > 50K_IPs
# head -5 50K_IPs
# nmap -n -sn -T4 -iL 50K_IPs -oA 50KHosts_DefaultPing

Starting Nmap ( https://nmap.org )
Host is up (0.27s latency).
Host is up (0.20s latency).
Host is up (0.089s latency).
[thousands of lines cut]
Host is up (0.20s latency).
Nmap done: 50000 IP addresses (3927 hosts up) scanned in 2532.05 seconds

Scanning the 50,000 addresses took just over 42 minutes, and 3,927 hosts were detected. To determine the effects of using a wider range of ping techniques, the same 50K hosts were rescanned with 14 probes per port rather than the default of four. As shown in Example 3.14, Nmap was able to detect 785 (20%) more hosts. It took about 147 minutes, which is almost 3.5 times as long. Given all the new hosts detected, that extra time was well spent. Note that not all of the new hosts will be legitimate. Increasing the number of ping probes increases the chances that Nmap will hit network artifacts that make a non-existent host appear to be active. Firewalls that return a RST for SYN or ACK packets to port 113 are one example of this.

Example 3.14. Repeating ping scan with extra probes
# nmap -n -sn -PE -PP -PS21,22,23,25,80,113,443,31339 -PA80,113,443,10042 \
  -T4 --source-port 53 -iL 50K_IPs -oA 50KHosts_ExtendedPing
Starting Nmap ( https://nmap.org )
Host is up (0.44s latency).
Host is up (0.13s latency).
Host is up (0.092s latency).
[thousands of hosts cut]
Host is up (0.23s latency).
Nmap done: 50000 IP addresses (4712 hosts up) scanned in 8842.31 seconds

When performing security audits for clients, I normally start TCP analysis with a port scan against the most common 1,000 ports (the default) with comprehensive ping scan options like those shown in Example 3.14, “Repeating ping scan with extra probes”. Such a scan does not take particularly long, allowing me to quickly start working. I also launch -Pn (ping disabled) scans against all 65K TCP ports in the background while I work. When they finish, which may be days later, I compare them to my initial quick scan and investigate any new ports or machines found.