Nmap Scripting Engine

While Nmap offers many capabilities as a network discovery tool, adding new functionality can be a difficult process. The normal approach is to parse Nmap output and pass the relevant results to another tool or custom script. This works, but it can be slow and merging the output of both tools is a pain.

The object of this project is to enhance Nmap with scripting capabilities so users can easily customize Nmap to meet their particular needs. This can be done with an Nmap Attack Scripting Language. We won't really call this NASL, but the goal is similar to the way Nessus is extended with NASL scripts.

This doesn't mean Nmap is going to become a full-fledged comprehensive vulnerability scanner like Nessus. While some vulnerability detection scripts may be valuable, there are also many discovery scripts we would like to have which are more general-purpose than detecting a flaw in a certain application. Examples might be "attempt to fetch SNMP information by trying a few dozen popular community strings", or "check if this service is an open proxy", or "read the /robots.txt and main page title for the detected web server", or "look up the AS number for this IP", or "Look up the whois contact for this IP".

REQUIRED FEATURES

If you think of other important features/requirements for this document, or feel it should be changed in other ways, please post your suggestions to nmap-dev. Labels such as "should" or "would be nice" denote less-critical features that can be worked on last, while "musts" have to be met unless we agree to make changes. Here are the proposed features and infrastructure requirements for the new Nmap scripting engine:

• Everything in the General Requirements document.
• Must be tested to work on at least Linux, Windows, Mac OS X, Solaris, FreeBSD, and OpenBSD. You must have at least Linux and Windows available for your own testing, and can recruit volunteers from nmap-dev for testing platforms you have no access to.
• The system must either be based on an existing procedural/OO language, such as Perl, Python, Lua, TCL, or Ruby, or a custom language of your own design. Scheme, LISP, and Java proposals are unlikely to be accepted. C/C++ is a possibility, even though it lacks some of the ease-of-use and capabilities of true scripting languages.
• A dependency system must allow scripts to run (or not) based on Nmap results such as port state, port number, version detection, OS detection, and host enumeration. You should be able to write constraints which mean "if OS.family is Solaris and the ports was identified as rpc.statd by version detection, run RPCStatDVulnTest". Or "if service is tcp and was determined to be httpd or the port number is 8080 or 3128, run HTTPProxyTest"
• A script should be able to run against a certain host, or just a service on that host. Scripts should be able to make TCP connections and send/receive TCP or UDP data. We may or may not want to limit the communication to just the target hosts -- I can imagine scripts wanting to contact name servers or databases (such as whois).
• Scripts don't need to be sandboxed the way they are in Nessus. Users should examine scripts before running them and never run scripts from people they don't trust. Or just stick with scripts from the base distribution and/or those you write yourself.
• There probably should be a shared registry (like Nessus has) for scripts to store various data. This may be as simple as a C hash table with access bindings for the scripting language used. Perhaps script results could be returned here in a standard way so that Nmap knows how and whether to print them.
• Scripts should be able to generate no results, results specific to a service, or results which should be printed for the target IP as a whole. For example, you may want to print proxy check results in the port information table, while displaying whois results underneath the table for the machine in general.
• Performance is critical, so the new system must be able to execute many scripts in parallel. Idealy, Nmap can mange multiple scripts per thread/process by letting them block for network I/O and then scheduling them for execution when the I/O completes. This is similar to the way Nmap does parallel version detection (which is a very simple form of scripting). Here are some ideas from an email discussion with one of the SoC Applicants:

  > Host parallel scans would work with Lua ( http://lua-users.org/wiki/
  > ThreadsTutorial ) but
  > they would require some extra planning.
  
  Nice.  I think that is worth doing.  The coroutines look particularly
  appealing (though I have only researched them briefly).  Most of these
  scripts will follow the format of:
  
  1) Connect to the target host/port (block until connection made)
  2) Format a probe string
  3) Send probe string
  4) Wait for response (block until response or network error like disconnect)
  5) Interpret response, then either return result or continue further
     communcation (goto step 2)
  
  So in an ideal case, perhaps, Nmap could have a pool of contexts
  (whatever info is required) for each "executing" script.  A script can
  run unimpeded until it does a connect/read/write.  When Nmap learns of
  a successful connection or reads new data, it resumes the
  corresponding script.  Nmap has a callback-based library named Nsock
  which handles network connections and I/O in a portable, efficient
  manner.  This is what version detection (among other parts of Nmap
  uses).
  

• Output must be clean and consistant, supporting both normal/interactive and XML output modes. Don't worry about grepable output for now.
• A variety (at least 3-5) of useful sample modules must be written to demonstrate the system capabilities. These should be written early in the process (obviously they may need to change as the system evolves) so we are sure that the capabilities are sufficient before all of the implementation work is done.
• Maybe scripts should support dependencies on each other, so that script B will always run after A, and perhaps only when a certain value (written by A) exists in the registry.
• Must be elegant and powerful, but still easy for those who know the language to use. Try to reduce the amount of "standard cruft" that each module/plug-in/script (whatever you want to call it) must contain.
• Try to make the system at least somewhat extensible, as the goals and needs of the system may change over time.
• Must come up with an initial timeline showing each week of SoC and what you plan to implement during that week. The weeks should be numberd 1-12, with week one ending on May 29 and week 12 ending August 21. That way it will correspond with the status report dates detailed in the general requirements. The focus is one the milestones you will complete by a given date. Obviously you won't be able to estimate perfectly, so this is subject to change.
• Be careful of dependencies. Very few scripting languages are supported out of the box on the systems supported by Nmap. So we will likely need to include an interpreter with the Nmap package on at least some platforms. So the smaller and/or more commonly installed the interpreter is, the better.
• Users must be able to disable the scripting system at build time. This may be useful for low-resource embedded systems, or when users are having trouble compiling/obtaining the scripting language interpreter.
• There shoud be a command-line option to specify a list of scripts to execute (rather than the default list of available scripts).
• The system must be capable of efficiently scaling to support tens of thousands of scripts. This is not as excessive as it sounds -- Nessus has more than 10,000 plugins.
• The system must be well documented. We would strongly suggest DocBook XML, as that is easy to convert to HTML, PDF, or NROFF. Let Fyodor know if you have DocBook questions or want some examples. One example is Nmap man page XML.
• Here is an example timeline for the project:
  1. Come up with an initial proposal, deciding on the language and the basics of how you will implement all of these requirements. If you can send an example script or two, that would really help people understand the system.
  2. Write your initial timeline/milestones
  3. Incorporate feedback from nmap-dev and fyodor. If changes are dramatic, repeat the review cycle until you and Fyodor are happy with it, and everyone has had a chance to contribute ideas.
  4. Actually implement the scripting language :). Test, re-test, fix bugs, etc. You may want to create a very limited, but usable version first. Then start adding all of the features. Beta releases should be sent to the nmap-dev list every couple weeks or so once the system is (somewhat) functional.
  5. Profit! And end up with a useful system ready to be incorporated into the base Nmap distribution so millions of users can benefit.