Nmap Explained: Port Scanning, Service Detection, and OS Fingerprinting thumbnail

Nmap Explained: Port Scanning, Service Detection, and OS Fingerprinting

⏱ approx. 20 min views 332 likes 0 LOG_DATE:2026-05-17
TOC

Nmap (Network Mapper) #

Nmap (Network Mapper) is an open-source scanner for discovering hosts and services on a network. Since Gordon Lyon (handle: Fyodor) released it in 1997, it has become the de-facto standard tool in penetration testing, vulnerability assessment, and network management. By sending packets to a given IP range and analyzing the responses, Nmap infers which hosts are alive, which ports are open, what service and version are running on each port, and which OS the target is most likely running.

1. What Nmap Can Do (4 Core Functions) #

What Nmap Can Do (4 Core Functions)

Nmap's capabilities can be grouped into four main functions.

  1. Host Discovery — Combine ICMP / ARP / TCP probes to enumerate live hosts in the given range. Use -sn to perform discovery only, skipping the port scan.
  2. Port Scanning — Send probe packets to TCP / UDP ports and classify each as open / closed / filtered based on the responses.
  3. Service / Version Detection — For open ports, send tailored probes from the built-in nmap-service-probes database and match the responses to identify the service name and version (-sV).
  4. OS Detection — Fingerprint the TCP/IP stack (TCP option ordering, default TTL, Window Size, IP ID behavior, etc.) and compare against nmap-os-db to infer the closest matching OS with a confidence percentage (-O).

On top of these, the NSE (Nmap Scripting Engine) lets you run Lua scripts for vulnerability checks, brute-force attacks, and additional information gathering. Categories like --script vuln let you run many scripts in one shot.

2. Legal and Ethical Notice #

Nmap Legal and Ethical Notice

Because Nmap is such a powerful tool, misusing it can lead to criminal charges. In Japan, the Act on the Prohibition of Unauthorized Computer Access and similar laws in other countries do not directly criminalize port scanning itself, but scanning networks you do not have permission to test is often interpreted as a "preparation for an attack" — and that crosses into illegal territory. Fines, imprisonment, and a permanent criminal record are all possible outcomes.

Only use Nmap against targets that fall into one of these categories.

  • Networks you own or manage — your home LAN, your own VPS, an isolated lab built for learning.
  • Targets covered by explicit written permission — a penetration testing engagement, a vulnerability assessment contract, a red-team exercise. The scope and the time window must be documented.
  • Authorized learning platforms — Hack The Box, TryHackMe, VulnHub, OverTheWire, and other environments where the operator explicitly allows scanning.

"Just testing a little out of curiosity" can become one of the most expensive mistakes of your life. There are publicly documented arrests and lawsuits in many countries — even with the best of intentions, never scan someone else's network without permission.

3. The History of Nmap #

3.1 Phrack issue 51 (1997) #

In September 1997, Gordon Lyon (Fyodor) published "The Art of Port Scanning" in Phrack Magazine issue 51 and released the first version of Nmap at the same time. Scanners such as strobe, netcat, and queso already existed, but Nmap stood out by integrating multiple scan techniques into a single tool with fast, flexible output formatting.

3.2 OS Fingerprinting (1998) #

In 1998, Nmap 2.0 introduced TCP/IP stack fingerprinting for OS detection. It compares subtle behavioral differences between each vendor's TCP/IP implementation against a fingerprint database. With this, Nmap evolved from a plain port scanner into a general-purpose network asset discovery tool.

3.3 NSE arrives (2006) #

In 2006, Nmap 4.21 embedded the Lua interpreter and introduced the NSE (Nmap Scripting Engine), letting users extend Nmap with their own scripts. This opened up an ecosystem for vulnerability checks, service-specific information gathering, and brute-force tasks — things common enough to need, but too specialized to bake into the core binary.

3.4 Companion Tools #

The Nmap project ships several other tools beyond the main binary: the GUI Zenmap, the general-purpose network I/O tool Ncat (a netcat successor), the port-testing/packet-crafting tool Nping, and the result-comparison tool Ndiff. They share Npcap / WinPcap (Windows) or libpcap (Unix) as the underlying capture layer and are usually installed together.

4. Host Discovery #

Before any port scan, Nmap first decides "is this host alive?" for each target (IP / CIDR / hostname). By default, it sends a combination of ICMP Echo, TCP SYN to 443, TCP ACK to 80, and ICMP Timestamp.

Key options:

  • -sn — Discovery only, no port scan (formerly -sP, "Ping Scan"). Use this when you only want a list of live hosts.
  • -Pn — Skip discovery entirely and treat every target as alive. Use this in environments where ICMP is blocked, so you don't miss hosts that are up but silent.
  • -PS<port> / -PA<port> — Send a TCP SYN or ACK to the given port and watch for a response. These often get through firewalls.
  • -PE / -PP / -PM — Send ICMP Echo / Timestamp / Address Mask requests.

On a local LAN, Nmap automatically uses ARP (faster and more reliable than ICMP). You can force this with -PR.

5. Port-Scanning Techniques #

Nmap offers many scan methods, each with a different combination of TCP flags or protocols. You pick between them based on stealth, required privileges, speed, and your investigation's goal.

Nmap Scan Techniques Comparison

5.1 TCP SYN Scan (-sS) #

Also known as the half-open scan, and the default when Nmap has root privileges. Nmap sends a SYN packet — SYN/ACK in reply means open, RST means closed, no response (or ICMP Unreachable) means filtered. Because it never completes the 3-way handshake, application-level logs on the server are unlikely to record a connection — it's comparatively quiet.

It requires raw sockets, so root is needed on Linux / macOS and Npcap on Windows.

5.2 TCP Connect Scan (-sT) #

This scan uses the OS's connect() syscall: it completes the full 3-way handshake and then closes the connection with RST. No root required, so any user can run it.

The trade-offs are that (1) application-level logs on the server do record the connection, and (2) the handshake overhead makes it slower. It's the fallback when -sS is unavailable.

5.3 UDP Scan (-sU) #

Send protocol-specific payloads (DNS / SNMP / NTP, etc.) to UDP ports and decide the state from the response.

Because UDP is connectionless, classification is harder. A protocol response means open, ICMP Port Unreachable means closed, and no response means open|filtered. It's very slow (each port requires retransmits and timeouts), so use it selectively against a narrow scope.

It's essential when you want to find services that only exist on UDP — DNS (53/udp), SNMP (161/udp), NTP (123/udp), TFTP (69/udp), and so on.

5.4 ACK Scan (-sA) #

Send ACK packets instead of SYN. The idea is that a stateful firewall may misclassify an ACK packet as "part of an existing session" and let it through.

If you get a RST back, the port is unfiltered (the firewall is not blocking it). No response or ICMP Unreachable means filtered. It cannot tell whether a port is open or closed — it's a specialized scan for inferring firewall rule sets.

5.5 NULL / FIN / Xmas Scans (-sN / -sF / -sX) #

Send packets with no flags / FIN only / FIN+PSH+URG, respectively. According to RFC 793, an open port should ignore these "invalid combinations" and a closed port should reply with RST.

  • No response → open|filtered
  • RST → closed
  • ICMP Unreachable → filtered

Windows, some Cisco, and some BSD implementations don't follow the RFC, so results are unreliable against them. It works well on Linux / Solaris / UNIX-family systems. Historically used to bypass older stateless firewalls.

5.6 Idle / Zombie Scan (-sI) #

An advanced scan that infers the target's port state by watching IP ID changes on a third-party "zombie" host. Your own IP address never appears in packets sent to the target, so IDS and firewall logs hold no trace of you.

Requirement: a zombie host that increments IP IDs globally and predictably — an older implementation that modern operating systems no longer use. Finding a suitable zombie is non-trivial today, but the technique is worth knowing for CTF and learning purposes.

5.7 How to Choose a Scan Type #

How to Choose an Nmap Scan Type

A practical rule of thumb:

  • Default to -sS — when root is available, it offers the best balance of speed, stealth, and compatibility.
  • Fall back to -sT when you don't have root — leaves traces in logs but works without elevated privileges.
  • Add -sU when you need UDP-sS -sU runs TCP and UDP scans at the same time.
  • Reach for -sA / -sN / -sF / -sX / -sI for special purposes — investigating firewall behavior, validating IDS coverage, or running a deeper probe without leaving traces.

6. Port State Classification #

Nmap classifies each port into one of six states based on the response.

State Meaning Typical response
open A service is actively listening SYN/ACK is returned
closed Reachable, but no service is listening RST is returned
filtered Packets are filtered; state can't be determined No response / ICMP Unreachable
unfiltered Reachable, but open vs. closed is unknown RST returned (result from an ACK scan)
open|filtered Could be open or filtered No response (during UDP / NULL / FIN / Xmas scans)
closed|filtered Could be closed or filtered Specific to Idle scans

filtered in particular doesn't mean the host doesn't exist — it means a firewall or router is dropping the probe. From an attacker's perspective, filtered behaves like closed: you can't make further progress. From a defender's perspective, that's the most desirable state.

7. Service / Version Detection (-sV) #

Once a port is identified as open, adding -sV tells Nmap to figure out "which service is this, and what version is it running?"

How it works:

  1. Note whether the port is TCP or UDP.
  2. Send the prioritized probes from nmap-service-probes for that port number.
  3. Match the responses (banners, HTTP headers, custom protocol replies) against regular-expression signatures.
  4. Extract the service name, version, and additional fields (product / version / extrainfo / ostype / hostname) from the matched pattern.

You can control thoroughness with --version-intensity 0..9 (default: 7). The -A switch turns on -sV -O --traceroute --script=default all at once.

8. OS Detection (-O) #

-O makes Nmap probe the target's TCP/IP stack from multiple angles to infer the operating system.

The main signals:

  • TCP ISN (Initial Sequence Number) generation pattern (randomness, periodicity)
  • TCP option order and values (MSS / SACK / Timestamp / Window Scale / NOP ordering varies by OS)
  • TCP Window Size initial value
  • IP ID generation method (sequential / random / always zero)
  • ICMP message structure
  • TCP/IP edge-case behavior (response to malformed packets)

Nmap collects this data with 16 probe packets and compares the result against nmap-os-db. The output looks like "Linux 5.x (confidence 92%)" — a best guess with a confidence rating.

9. NSE (Nmap Scripting Engine) #

NSE embeds a Lua interpreter into Nmap so users can extend its functionality with scripts.

Script categories (selected with --script):

  • default (same as -sC) — Safe, informative scripts that run by default
  • discovery — Additional host / service information gathering
  • version — Augments -sV
  • vuln — Known vulnerability checks (many CVE-specific scripts)
  • exploit — Actually tries to exploit (be careful)
  • auth — Authentication-related (anonymous FTP, default credentials, etc.)
  • brute — Brute-force attacks
  • intrusive — May impact the target
  • safe — Explicitly marked as having no impact
  • malware — Malware infection checks
  • dos — DoS testing

Common examples:

  • --script vuln — Sweep for known CVEs
  • --script http-enum — Enumerate directories on an HTTP server
  • --script smb-os-discovery — Get OS info via SMB
  • --script ssl-enum-ciphers — List SSL/TLS cipher suites
  • --script ssh-hostkey — Retrieve the SSH host key

NSE scripts live in /usr/share/nmap/scripts/ on Linux, and users can write their own.

10. Evasion and the Defender's View #

Nmap has many options for evading IDS / IPS / firewalls. Only use them within the bounds of a legal engagement.

Key options:

  • -T0 through -T5 — Timing templates. -T0 (paranoid) is ultra-slow to stay under IDS thresholds. -T4 (aggressive) is the practical default.
  • -D <decoy1>,<decoy2>,ME — Mix in decoy IPs so your own address blends in.
  • -f / --mtu — Fragment packets to slip past signature-based inspection.
  • --data-length — Add padding to change packet size.
  • --source-port <port> — Set the source port (using 53 for DNS or 20 for FTP, etc., often passes through firewalls that whitelist those).
  • --randomize-hosts — Shuffle the order of multi-target scans.

From a defender's perspective, Nmap scans leave the following traces in logs.

  • Many port-connection attempts from one IP in a short time (hundreds to thousands of ports)
  • TCP SYN with no follow-up ACK (-sS) → doesn't show up in application logs but does trip stateful firewall and IDS SYN-flood detection
  • Abnormal TCP flag combinations (-sN / -sF / -sX)
  • Bursts of ICMP / ARP traffic (host discovery)

IDS tools like Snort, Suricata, and Zeek ship with built-in Nmap signatures.

11. Practical Examples #

Commands you'll commonly use in learning environments like HTB and TryHackMe.

Initial reconnaissance (you know nothing about the host):

sudo nmap -sS -sV -O -A -T4 -p- <target>
  • -sS — SYN scan (the default, but explicit)
  • -sV — Service / version detection
  • -O — OS detection
  • -A — Shorthand for -sV -O --traceroute --script=default
  • -T4 — Speed priority
  • -p- — All 65535 ports

Quick overview:

sudo nmap -sS --top-ports 1000 -T4 <target>

Specific vulnerability check:

sudo nmap --script vuln -p 80,443 <target>

Include UDP:

sudo nmap -sS -sU -p T:80,443,U:53,161 <target>

Save results as XML for downstream tools:

sudo nmap -sS -sV -oA scan_results <target>

-oA saves output simultaneously in normal, XML, and grepable formats. The XML file can be imported into Metasploit with db_import, among other things.

Don't try to finish in a single Nmap command — the standard practice is to add options incrementally based on what you've learned. Start conservatively and layer on detailed scans as needed: that's how you gather information efficiently without disturbing the target.