<indexterm class="startofrange" id="nse-indexterm"><primary>Nmap Scripting Engine (NSE)</primary></indexterm>
<indexterm><primary>scripting</primary><see>Nmap Scripting Engine</see></indexterm>
<indexterm><primary>NSE</primary><see>Nmap Scripting Engine</see></indexterm>

<sect1 id="nse-intro">
    <title>Introduction</title>

    <para>The Nmap Scripting Engine (NSE) is one of Nmap's most
    powerful and flexible features.  It allows users to write (and
    share) simple scripts to automate a wide variety of networking
    tasks.  Those scripts are then executed in parallel with the speed
    and efficiency you expect from Nmap.  Users can rely on the
    growing and diverse set of scripts distributed with Nmap, or write
    their own to meet custom needs.</para>

    <para>We designed NSE to be versatile, with the following tasks in mind:</para>    

    <variablelist>
      <varlistentry>
        <term>Network discovery</term>
        <listitem>
          <para>This is Nmap's bread and butter. Examples include
           looking up whois data based on the target domain,
           querying ARIN, RIPE, or APNIC for the target IP to determine ownership,
           performing identd lookups on open ports, SNMP queries, and
           listing available NFS/SMB/RPC shares and services.</para>
        </listitem>

      </varlistentry>
      <varlistentry>
        <term>
        <indexterm><primary>version detection</primary><secondary>using NSE</secondary></indexterm>
        More sophisticated version detection</term>
        <listitem>
          <para>The Nmap version detection system (<xref linkend="vscan"/>)
       is able to recognize thousands of different services through
       its probe and regular expression based matching system, but it
       cannot recognize everything.  For example, identifying the Skype v2 service requires two
       independent probes, which version detection isn't flexible enough to handle.  Nmap could also recognize more SNMP services
       if it tried a few hundred different community names by brute
       force.  Neither of these tasks are well suited to traditional
       Nmap version detection, but both are easily accomplished with
       NSE.  For these reasons, version detection now calls NSE by
       default to handle some tricky services.  This is described in
       <xref linkend="nse-vscan"/>.</para>

        </listitem>
      </varlistentry>
      <varlistentry>
        <term>
        <indexterm><primary>vulnerability detection</primary></indexterm>
        Vulnerability detection</term>
        <listitem>

          <para>When a new vulnerability is discovered, you often want
       to scan your networks quickly to identify vulnerable systems
       before the bad guys do.  While Nmap isn't a
       comprehensive <ulink role="hidepdf" url="http://sectools.org/vuln-scanners.html">vulnerability scanner</ulink>, 
       NSE is powerful enough to handle even demanding vulnerability
       checks.  Many vulnerability detection scripts have already been
       written and we plan to distribute more as they are written.
       </para>

        </listitem>
      </varlistentry>
      <varlistentry>
        <term>Backdoor detection</term>
        <listitem>
          <para>
	   Many attackers and some automated worms leave
	   backdoors to enable later reentry.  Some of these can be
	   detected by Nmap's regular expression based version detection.
	   For example, within hours of the MyDoom worm hitting the
	   Internet,
	   Jay Moran<indexterm><primary>Moran, Jay</primary></indexterm>
           posted an Nmap version detection probe and
	   signature so that others could quickly scan their networks.
	   For more complex worms and backdoors, NSE is needed
	   for reliable detection.
       </para>
        </listitem>
      </varlistentry>
      <varlistentry>
        <term>Vulnerability exploitation</term>
        <listitem>
          <para>
	   As a general scripting language, NSE can even
	   be used to exploit vulnerabilities rather than just find them.
	   The capability to add custom exploit scripts may be valuable
	   for some people (particularly
	   penetration testers),<indexterm><primary>penetration testing</primary></indexterm>
           though we aren't
	   planning to turn Nmap into an exploitation framework like
	   <ulink url="http://www.metasploit.com">Metasploit</ulink>.<indexterm><primary><application>Metasploit</application></primary></indexterm>
       </para>
        </listitem>
      </varlistentry>
    </variablelist>
    <para>
      The listed items were our initial goals, but we expect that Nmap
      users will come up with inventive uses for NSE.
    </para>

    <print><note><para>The Nmap Scripting Engine is a new Nmap feature
    which already works well, but is under active development. To
    provide the latest NSE news and updates, this chapter has been
    updated and posted for free online at
    <ulink url="http://nmap.org/book/nse.html"/>.</para></note></print>

    <para>
      Scripts are written in the
      embedded
      <ulink url="http://www.lua.org/">Lua programming language</ulink>.<indexterm><primary>Lua programming language</primary><seealso>Nmap Scripting Engine</seealso></indexterm>
      The language itself is well documented in the books 
<web>
      <citetitle><ulink url="http://www.amazon.com/exec/obidos/ASIN/8590379825/secbks-20">Programming
	in Lua, Second Edition</ulink></citetitle> and
      <citetitle><ulink url="http://www.amazon.com/exec/obidos/ASIN/8590379825/secbks-20">Lua
	5.1 Reference Manual</ulink></citetitle>.  
</web>
<print>
      <citetitle>Programming in Lua, Second Edition</citetitle> and
      <citetitle>Lua 5.1 Reference Manual</citetitle>.  
</print>

The reference manual is also 
      <ulink url="http://www.lua.org/manual/5.1/">freely available
	online</ulink>, as is the 
      <ulink url="http://www.lua.org/pil/">first edition of <citetitle>Programming in
	Lua</citetitle></ulink>. Given the availability of these excellent general
      Lua programming references, this document only covers aspects and
      extensions specific to Nmap's scripting engine.
    </para>
    <para>

      NSE is activated with the <option>-sC</option> option (or
      <option>--script</option> if you wish to specify a custom set of
      scripts) and results are integrated into Nmap
      normal<indexterm><primary>normal output</primary></indexterm>
      and XML output.<indexterm><primary>XML output</primary></indexterm>
      Two types of scripts are supported: service and host
      scripts.  Service scripts relate to a certain open port
      (service) on the target host, and any results they produce are included
      next to that port in the Nmap output port table.  Host scripts,
      on the other hand, run no more than once against each target IP
      and produce results below the port table.  <xref
      linkend="nse-ex1" xrefstyle="select: label nopage"/> shows a typical script scan.  Examples of
      service scripts producing output are: <literal>Stealth SSH
      version</literal>, which tricks some SSH servers into divulging
      version information without logging the attempt as they normally
      would; <literal>Service Owner</literal>, which connects to open
      ports, then performs a reverse-identd query to determine what
      username each is running under; and <literal>HTML Title</literal>,
      which simply grabs the title of the root path of any web servers
      found.  A sample host script is <literal>RIPE Query</literal>,
      which looks up and reports target IP ownership
      information.<indexterm><primary>script names, examples of</primary></indexterm>

    </para>
    <example id="nse-ex1">
      <title>Typical NSE output</title>
<indexterm><primary><option>-sC</option></primary><secondary>example of</secondary></indexterm>
<screen>
$ nmap -sC localhost -p 22,23,80,113

Starting Nmap ( http://nmap.org )
Interesting ports on localhost (127.0.0.1):
PORT    STATE  SERVICE
22/tcp  open   ssh
|_ Stealth SSH version: SSH-1.99-OpenSSH_4.2
|_ SSH protocol version 1: Server supports SSHv1
23/tcp  closed telnet
80/tcp  open   http
|_ HTML title:Test Page for Apache Installation
113/tcp closed auth

Host script results:
|_ RIPE Query: IP belongs to:          Internet Assigned Numbers Authority

Nmap finished: 1 IP address (1 host up) scanned in 0.907 seconds
</screen>
    </example>

  </sect1>
  <sect1 id="nse-usage">
    <title>Usage and Examples</title>
    <para>
      While NSE has a complex implementation for efficiency, it is
      strikingly easy to use.  Simply specify
      <option>-sC</option><indexterm><primary><option>-sC</option></primary></indexterm>
      to enable the most common scripts.  Or specify the
      <option>--script</option><indexterm><primary><option>--script</option></primary></indexterm>
      option to choose your own scripts to
      execute by providing categories, script file names, or the name of
      directories full of scripts you wish to execute.  You can customize 
	  some scripts by providing arguments to them via the
	  <option>--script-args</option><indexterm><primary><option>--script-args</option></primary></indexterm>
          option. The two remaining options,
	  <option>--script-trace</option><indexterm><primary><option>--script-trace</option></primary></indexterm>
          and <option>--script-updatedb</option>,<indexterm><primary><option>--script-updatedb</option></primary></indexterm>
          are generally only used for script debugging and development.
    </para>

    <sect2 id="nse-categories"><title>Script Categories</title>
    <indexterm><primary>script categories</primary></indexterm>

    <para>NSE scripts define a list of categories they belong to.
          Currently defined categories are <literal>safe</literal>,
          <literal>intrusive</literal>, <literal>malware</literal>,
          <literal>version</literal>, <literal>discovery</literal>,
          <literal>vuln</literal>, <literal>auth</literal> and
          <literal>default</literal>.
          Category names are not case
          sensitive.  The following list describes each category.</para>

     <variablelist>
        <varlistentry>
          <term>
            <indexterm><primary sortas="safe script category">&ldquo;<literal>safe</literal>&rdquo; script category</primary></indexterm>
            <option>safe</option>
          </term>
          <listitem>
            <para>Scripts
	      which weren't designed to crash services, use large
	      amounts of network bandwidth or other resources, or
	      exploit security holes. These are less likely to offend
	      remote sysadmins.  Of course (as with all other Nmap
	      features) we cannot guarantee that they won't ever cause
	      adverse reactions.  Most of these perform general
	      network discovery. Examples are echoTest (sends a string
	      to the UDP echo service) and showHTMLTitle (grabs the
	      title from a web page).</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>
            <indexterm><primary sortas="intrusive script category">&ldquo;<literal>intrusive</literal>&rdquo; script category</primary></indexterm>
            <option>intrusive</option>
          </term>
          <listitem>

            <para>These are scripts that cannot be classified in the
            <literal>safe</literal> category because the risks are too high that they
            will crash the target system, use up significant resources
            on the target host (such as bandwidth or CPU time), or 
            otherwise be perceived as malicious by the target's
            system administrators.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>
            <indexterm><primary sortas="malware script category">&ldquo;<literal>malware</literal>&rdquo; script category</primary></indexterm>
            <option>malware</option>
          </term>
          <listitem>
            <para>These scripts test whether the target platform is
            infected by malware or backdoors.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>
            <indexterm><primary sortas="version script category">&ldquo;<literal>version</literal>&rdquo; script category</primary></indexterm>
            <indexterm><primary>version detection</primary><seealso>&ldquo;<literal>version</literal>&rdquo; script category</seealso></indexterm>
            <option>version</option>
          </term>
          <listitem>

            <para>The scripts in this category are an extension to the
            version detection option and cannot be selected
            explicitly. They are selected to run only if version
            detection (<option>-sV</option>) was requested.  Their
            output cannot be distinguished from version detection
            output and they do not produce service or host script
            results.</para>

          </listitem>
        </varlistentry>

        <varlistentry>
          <term>
            <indexterm><primary sortas="discovery script category">&ldquo;<literal>discovery</literal>&rdquo; script category</primary></indexterm>
            <option>discovery</option>
          </term>
          <listitem>
            <para>These scripts try to actively learn more about the
            network by querying public registries, SNMP-enabled
            devices, directory services, and the like.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>
            <indexterm><primary sortas="vuln script category">&ldquo;<literal>vuln</literal>&rdquo; script category</primary></indexterm>
            <option>vuln</option>
          </term>
          <listitem>
            <para>These scripts check for specific known vulnerabilities and
            generally only report results if they are found.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>
            <indexterm><primary sortas="auth script category">&ldquo;<literal>auth</literal>&rdquo; script category</primary></indexterm>
            <option>auth</option>
          </term>
          <listitem>
            <para>These scripts try to determine authentication credentials
            on the target system, often through a brute-force attack.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>
            <indexterm><primary sortas="default script category">&ldquo;<literal>default</literal>&rdquo; script category</primary></indexterm>
            <option>default</option>
          </term>
          <listitem>
            <para>These scripts are the default set and are run when
            using <option>-sC</option>, <option>-A</option>
            or <option>--script</option> without any arguments. This
            category can also be specified explicitly like any other
            using <option>--script=default</option>.</para>
          </listitem>
        </varlistentry>
    </variablelist>

    </sect2>

    <sect2 id="nse-cmd-line-args">
      <title>Command-line Arguments</title>
      <para>
	These are the five command line arguments specific to script-scanning:
      </para>
      <variablelist>
        <varlistentry>
          <term>
            <indexterm><primary><option>-sC</option></primary></indexterm>
            <option>-sC</option>
          </term>
          <listitem>
          <para>Performs a script scan using the default set of scripts. It is 
		  equivalent to <option>--script=default</option>. Some of the
		  scripts in this category are considered intrusive and should
		  not be run against a target network without permission. </para>
          </listitem>
        </varlistentry>

     <varlistentry>
        <term>
          <indexterm><primary><option>--script</option></primary></indexterm>
          <option>--script <replaceable>script-categories</replaceable>|<replaceable>directory</replaceable>|<replaceable>filename</replaceable>|all</option></term>

<listitem>

<para>Runs a script scan (like <option>-sC</option>) using the comma-separated list of
script categories, individual scripts, or directories containing
scripts, rather than the default set. Nmap first tries to interpret the
arguments as categories, then (if that fails) as files or
directories. A script or directory of scripts may be specified as an
absolute or relative path. Absolute paths are used as
supplied. Relative paths are searched for in the following places
until found:<indexterm><primary>data files</primary><secondary>directory search order</secondary></indexterm><indexterm><primary>scripts, location of</primary></indexterm>
<filename>--datadir/</filename>; 
<filename>$NMAPDIR/</filename>;<indexterm><primary><envar>NMAPDIR</envar> environment variable</primary></indexterm>
<filename>~/.nmap/</filename> (not searched on Windows);<indexterm><primary sortas="nmap directory"><filename>.nmap</filename> directory</primary></indexterm>
NMAPDATADIR/ or<indexterm><primary>NMAPDATADIR</primary></indexterm>
<filename>./</filename>.  A <filename>scripts/</filename> subdirectory
is also tried in each of these.</para>

<para>If a directory is specified and found, Nmap loads all NSE
scripts (any filenames ending with <literal>.nse</literal>) from that
directory. Filenames without the <literal>nse</literal> extension are
ignored. Nmap does not search recursively into subdirectories to find
scripts.  If individual file names are specified, the file extension
does not have to be <literal>nse</literal>.</para>

<para>Nmap scripts are stored in a <filename>scripts</filename>
subdirectory of the Nmap data directory by default (see
<xref linkend="data-files"/>). For efficiency, scripts are indexed in
a database stored
in <filename>scripts/script.db</filename>.<indexterm><primary><filename>script.db</filename></primary></indexterm>
which lists the category or categories in which each script belongs.
Give the argument <literal>all</literal> to execute all scripts in the
Nmap script database.</para>

<para>Malicious scripts are not run in a sandbox and thus could damage your system or invade your privacy.  Never run scripts from third parties unless you trust the authors or have carefully audited the scripts yourself.</para>


          </listitem>
        </varlistentry>

        <varlistentry>
          <term>
            <indexterm><primary><option>--script-args</option></primary></indexterm>
            <option>--script-args</option>
          </term>
          <listitem>
            <para>provides arguments to the scripts. See <xref
			linkend="nse-args"/> for a detailed explanation.</para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term>
            <indexterm><primary><option>--script-trace</option></primary></indexterm>
            <option>--script-trace</option>
          </term>
          <listitem>
            <para>

	      This option is similar to
              <option>--packet-trace</option>, but works at the
              application level rather than packet by packet. If this
              option is specified, all incoming and outgoing
              communication performed by scripts is printed. The
              displayed information includes the communication
              protocol, source and target addresses, and the
              transmitted data. If more than 5% of transmitted data is
              unprintable, hex dumps are given instead.

	    </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term>
            <indexterm><primary><option>--script-updatedb</option></primary></indexterm>
            <option>--script-updatedb</option>
          </term>
          <listitem>

            <para>This option updates the script database found
            in <filename>scripts/script.db</filename> which is used by
            Nmap to determine the available default scripts and
            categories. It is only necessary to update the database if
            you have added or removed NSE scripts from the
            default <filename>scripts</filename> directory or if you
            have changed the categories of any script.  This option is
            generally used by
            itself: <command>nmap --script-updatedb</command>.</para>

          </listitem>
        </varlistentry>
      </variablelist>

      <para>
        Some of the Nmap options have effects on script scans. The most
        prominent of these is
	<option>-sV</option>.<indexterm><primary><option>-sV</option></primary></indexterm>
        A version scan executes
        the scripts in the
	<literal>version</literal> category.<indexterm><primary sortas="version script category">&ldquo;<literal>version</literal>&rdquo; script category</primary></indexterm>
        The scripts
        in this category are slightly different than other scripts. Their
        output blends in with the version scan and they do not produce any
        script scan output.
      </para>
      <para>
        Another option which has effect on the scripting engine is 
	<option>-A</option>.<indexterm><primary><option>-A</option></primary><secondary>features enabled by</secondary></indexterm>
        The advanced/aggressive mode of Nmap implies
	the option <option>-sC</option>.
      </para>

      <para>
      </para>
    </sect2>

	<sect2 id="nse-args">
      <title>Arguments to Scripts</title>
      <indexterm><primary>script arguments</primary></indexterm>
      <para>
      You can pass arguments to NSE scripts via the
	  <option>--script-args</option> option. The script-arguments generally are
	  name-value pairs, which are provided to the script as a Lua table called
	  <literal>args</literal> inside the <literal><link
	  linkend="nse-api-registry">nmap.registry</link></literal> with
	  the names as keys for the corresponding values. The values can either be
	  strings or tables. Subtables can be used to pass arguments to 
	  scripts with a finer granularity (e.g. pass different usernames for
	  different scripts). A typical nmap invocation with script arguments may
	  look like:
	  </para>
	  <para>
<indexterm><primary><option>-sC</option></primary><secondary>example of</secondary></indexterm>
<indexterm><primary><option>--script-args</option></primary><secondary>example of</secondary></indexterm>
<userinput>
$ nmap -sC --script-args user=foo,pass=bar,anonFTP={pass=ftp@foobar.com}
</userinput>
	  </para>
	  <para>
	  which would result in the Lua table: 
          </para>
<programlisting>
{user="foo",pass="bar",anonFTP={pass="nobody@foobar.com"}}
</programlisting>
 
	  <para>You could therefore access the username (<literal>"foo"</literal>)
	  inside your script as
          <literal>local username= nmap.registry.args.user</literal>.
          As a general rule the subtables used to override
	  options for scripts should be named as the script's
	  <literal>id</literal>, otherwise scripts won't know where to
	  retrieve their arguments.
	  </para>
	</sect2>


    <sect2 id="nse-usage-examples">
      <title>Usage Examples</title>
      <para> 
	A simple script scan using the default set of scripts
      </para>
      <para>
        <indexterm><primary><option>-sC</option></primary><secondary>example of</secondary></indexterm>
        <userinput> 
	  $ nmap -sC example.com
      </userinput>
      </para>
      <para> 
	Tracing a specific script.
      </para>
      <para>
        <indexterm><primary><option>--script</option></primary><secondary>example of</secondary></indexterm>
        <indexterm><primary><option>--script-trace</option></primary><secondary>example of</secondary></indexterm>
        <userinput> 
	  $ nmap --script=./showSSHVersion.nse --script-trace example.com
      </userinput>
      </para>

      <para> 
	All scripts in a subdirectory named <filename>mycustomscripts</filename> in addition to all of Nmap's included scripts which are in the <literal>safe</literal> category.
      </para>
      <para>
        <userinput> 
	  $ nmap --script=mycustomscripts,safe example.com
      </userinput>
      </para>
    </sect2>
  </sect1>
  <sect1 id="nse-scripts">
    <title>Script Format</title>
    <para>NSE scripts consist of six descriptive fields along with either a port or host rule defining when the script should be executed and an action block containing the actual script instructions. Values can be assigned to these fields just as you would assign any other Lua variables.  Their names must be lowercase as shown here.</para>

    <sect2 id="nse-format-id">
      <title><literal>id</literal> Field</title>
      <indexterm><primary sortas="id script variable">&ldquo;<varname>id</varname>&rdquo; script variable</primary></indexterm>
      <para>
	The script's <literal>id</literal> field is displayed in the Nmap output 
	table if the script produces any output.  It should be unique so users
	can identify exactly which script file produced a message. IDs
	should be kept short to conserve space in Nmap output, while
	still being meaningful enough for users to recognize.  Some
	good examples are <literal>RIPE query</literal>, <literal>HTML
	title</literal>, and <literal>Kibuv worm</literal>.<indexterm><primary>script names, examples of</primary></indexterm>
      </para>
    </sect2>
    <sect2 id="nse-format-description">
      <title><literal>description</literal> Field</title>
      <indexterm><primary sortas="description script variable">&ldquo;<varname>description</varname>&rdquo; script variable</primary></indexterm>
      <para>The <literal>description</literal> field describes what the script is testing
      for and any critical notes the user must be aware of. A good
      example is this description from a user-contributed recursive
      DNS script: <quote>Checks whether a nameserver on UDP port 53
      allows queries for third party names. It is expected that
      recursion will be enabled on your own internal
      nameserver.</quote></para>
    </sect2>

    <sect2 id="nse-format-categories">
      <title><literal>categories</literal> Field</title>
      <indexterm><primary sortas="category script variable">&ldquo;<varname>category</varname>&rdquo; script variable</primary></indexterm>

      <para>The <literal>categories</literal> field defines one or
      more categories to which a script belongs (see
      <xref linkend="nse-categories"/>).  The categories are case-insensitive and may be specified in any order.  They are listed in an array-like Lua table as in this example:</para>

<programlisting>
categories = {"default", "discovery", "safe"}
</programlisting>

    </sect2>

    <sect2 id="nse-format-author">
      <title><literal>author</literal> Field </title>
      <indexterm><primary sortas="author script variable">&ldquo;<varname>author</varname>&rdquo; script variable</primary></indexterm>
      <para>
	The <literal>author</literal> field contains the script authors name and contact information. If you are worried about spam, you might want to omit or obscure your email address, or give your home page URL instead.  This optional field is not used by NSE, but is important for giving script authors due credit or blame. 
      </para>
    </sect2>

    <sect2 id="nse-format-license">
      <title><literal>license</literal> Field </title>
      <indexterm><primary sortas="license script variable">&ldquo;<varname>license</varname>&rdquo; script variable</primary></indexterm>
      <indexterm><primary>copyright</primary><secondary>of scripts</secondary></indexterm>

      <para>Nmap is a community project and we welcome all sorts of
      code contributions, including NSE scripts.  So if you write a
      valuable script, don't keep it to yourself!
      The <literal>license</literal> field helps ensure that we have
      legal permission to distribute all the scripts which come with Nmap. All of those scripts
      currently use the standard Nmap license
      (described in <xref linkend="nmap-copyright"/>).  They include
      the following line:</para>

<programlisting>
license = "Same as Nmap--See http://nmap.org/book/man-legal.html"
</programlisting>

<para>The Nmap license is similar to the GNU GPL.  Script authors may
use a BSD-style license (no advertising clause) instead if they prefer
that.</para>

    </sect2>

    <sect2 id="nse-format-runlevel">
      <title><literal>runlevel</literal> Field</title>
      <indexterm><primary sortas="runlevel script variable">&ldquo;<varname>runlevel</varname>&rdquo; script variable</primary></indexterm>
      <indexterm><primary>run level of scripts</primary></indexterm>
      <para>
	This optional field determines script execution order.  When
	this section is absent the run level defaults to 1.0. A script
	with the run level 1.0 is run before any scripts with <literal>runlevel</literal> set to
	<literal>2.5</literal>, which in turn runs before any scripts
	with <literal>runlevel 2.55</literal>.  Scripts with the same run level are run
        concurrently. One
	application of run levels is allowing scripts to depend on
	each other. If <literal>script A</literal> relies on some
	information gathered by <literal>script B</literal>, give
	<literal>B</literal> a lower run level than
	<literal>A</literal>. <literal>Script B</literal> can store
	information in the NSE registry for <literal>A</literal> to
	retrieve later. For information on the NSE registry see to
	<xref linkend="nse-api-registry"/>.
      </para>
    </sect2>

    <sect2 id="nse-format-rules">
      <title>Port and Host Rules</title>
      <indexterm><primary sortas="portrule script variable">&ldquo;<varname>portrule</varname>&rdquo; script variable</primary></indexterm>
      <indexterm><primary sortas="hostrule script variable">&ldquo;<varname>hostrule</varname>&rdquo; script variable</primary></indexterm>
      <indexterm><primary>rules in NSE</primary><see>&ldquo;<varname>portrule</varname>&rdquo; and &ldquo;<varname>hostrule</varname>&rdquo;</see></indexterm>
      <para>

Nmap uses the script rules to determine whether a script should be run
against a target.  A script contains either a <emphasis>port
rule</emphasis>, which governs which ports of a target the scripts may
run against, or a <emphasis>host rule</emphasis>, which specifies that
the script should be run only once against a target IP and only if
certain conditions are met.  A rule is a Lua function that returns
either <literal>true</literal> or <literal>false</literal>. The
script <emphasis>action</emphasis> is only performed if the rule
evaluates to <literal>true</literal>.  The host rule accepts a host
table as an argument and may test, for example, the IP address or
hostname of the target.  A port rule accepts both host and port tables
as arguments for any TCP or UDP port in either the
<literal>open</literal><indexterm><primary><literal>open</literal> port state</primary></indexterm>, 
<literal>open|filtered</literal><indexterm><primary><literal>open|filtered</literal> port state</primary></indexterm>,
or <literal>unfiltered</literal><indexterm><primary><literal>unfiltered</literal> port state</primary></indexterm> port states.  Port rules generally test factors such as the port number, port state, or listening service name in deciding whether to run against a port. Example rules are shown in <xref linkend="nse-tutorial-rule"/>.</para> 
</sect2>

     <sect2 id="nse-format-action"><title>Action</title>
      <indexterm><primary sortas="action script variable">&ldquo;<varname>action</varname>&rdquo; script variable</primary></indexterm>

      <para>
The action is the heart of an NSE script. It contains all of the
instructions to be executed when the script's port or host rule
triggers. It is a Lua function which accepts the same arguments as the
rule and can return either <literal>nil</literal> or a string. If a string is returned by a service script, the string and script ID are printed in the Nmap port table output.  A string returned by a host script is printed below the port table.  No output is produced if the
script returns <literal>nil</literal>.  For an example of an NSE
action refer to <xref linkend="nse-tutorial-action"/>.
      </para>
    </sect2>

  </sect1>
  <sect1 id="nse-language">
    <title>Script Language</title>
    <indexterm><primary>Nmap Scripting Engine (NSE)</primary><secondary>parts of</secondary></indexterm>

    <para>
      The core of the Nmap Scripting Engine is an embeddable Lua
      interpreter. Lua is a lightweight language designed for
      extensibility.  It offers a powerful and well documented API for
      interfacing with other software such as Nmap.
    </para>
    <indexterm><primary>Nmap Scripting Engine (NSE)</primary><secondary>library</secondary></indexterm>
    <para>
      The second part of the Nmap Scripting Engine is the NSE Library, which
      connects Lua and Nmap. This layer
      handles issues such as initialization of the Lua interpreter, 
      scheduling of parallel script execution, script retrieval and
      more. It is also the heart of the NSE network I/O framework and the
	  exception handling mechanism.  It also includes utility libraries to make scripts more powerful and convenient.  The utility library modules and extensions are described in <xref linkend="nse-library"/>.</para>

    <sect2 id="nse-lua">
    <title>Lua Base Language</title>
      <indexterm><primary>Lua programming language</primary></indexterm>
      <para>

	The Nmap scripting language is an embedded <ulink
	url="http://www.lua.org/">Lua</ulink> interpreter which was
	extended with libraries for interfacing with Nmap.  The Nmap
	API is in the Lua namespace <literal>nmap</literal>.  This
	means that all calls to resources provided by Nmap have an
	<literal>nmap</literal> prefix.<indexterm><primary><varname>nmap</varname> NSE module</primary></indexterm>
	<literal>nmap.new_socket()</literal>, for example, returns a
	new socket wrapper object. The Nmap library layer also takes
	care of initializing the Lua context, scheduling parallel
	scripts and collecting the output produced by completed
	scripts.  
	</para>
      <para>

	During the planning stages, we considered several programming
	languages as the bases for Nmap scripting. One option was to
	implement a completely new programming language. The criteria
	imposed on the options were strict, NSE needed to be easy to
	use, small in size, compatible with the Nmap license,
	scalable, fast and parallelizable.  There have been several
	efforts to design a security auditing language from scratch
	which have resulted in well-known awkward solutions. It was
	clear from the beginning that we would not go down this
	road. For a while the Guile Scheme interpreter was considered
	but the preference drifted towards Elk in favor of its more
	liberal license. But parallelizing Elk scripts would have been
	difficult. In addition, the subset of Nmap users familiar with
	functional programming is regarded too small to consider
	Scheme as an option.  Larger interpreters like Perl, Python or
	Ruby are well-known and loved, but are difficult to embed
	efficiently.  In the end, Lua exceeded in all criteria for
	NSE. It is small, distributed under the MIT license, has
	coroutines for efficient parallel script
	execution, was designed with embeddability in mind, has
	excellent documentation, and is actively developed by a large
	and committed community.
	Lua is even embedded in popular applications including
	<application>Wireshark</application> and <application>Second Life</application>.

      </para>
    </sect2>
	</sect1>
    <sect1 id="nse-library">
      <indexterm class="startofrange" id="nse-library-indexterm"><primary>Nmap Scripting Engine (NSE)</primary><secondary>modules</secondary></indexterm>
      <title>Lua Extensions</title>
      <para>In addition to the significant built-in capabilities of
      Lua, we have written or integrated several extensions to make
      NSE scripts more powerful and convenient to write. These 
	  <emphasis>modules</emphasis> are compiled and installed along with 
	  Nmap. They have their own directory, <filename>nselib</filename>, which
	  is installed in the configured datadir. Scripts need only 
      <ulink url="http://www.lua.org/manual/5.1/manual.html#pdf-require">
      <literal>require</literal>
      </ulink> the default modules in order to use them.
      The default modules are described in the following sections.
	  </para>

      <sect2 id="nse-bitops">
        <title>Bitwise Logical Operations</title>
        <indexterm><primary><varname>bit</varname> NSE module</primary></indexterm>
        <para>
	  Lua does not provide
	  bitwise logical operations.<indexterm><primary>bitwise operations in NSE</primary></indexterm>
          Since they
	  are often useful for low-level network communication,
	  Reuben Thomas'<indexterm><primary>Thomas, Reuben</primary></indexterm>
         <ulink url="http://luaforge.net/projects/bitlib">bitwise operation library</ulink>
      for Lua has been
	  integrated into NSE. The arguments to the bitwise operation
	  functions should be integers.  The number of bits available
	  for logical operations depends on the data type used to
	  represent Lua numbers&mdash;this is typically 8-byte IEEE
	  floats (double), which give 53 bits (the size of the mantissa).

	  This implies that the bitwise operations won't work (as expected) 
	  for numbers larger than 10<superscript>14</superscript>. You
	  can use them with 32-bit wide numbers without any problems. Operations
	  involving 64-bit wide numbers, however, may not return the expected
	  result.
	  The logical operations start with <quote>b</quote> (for <literal>bit</literal>) to avoid
	  clashing with reserved words; although <literal>xor</literal> isn't a
	  reserved word, it seemed better to use <literal>bxor</literal> for
	  consistency. In NSE the bitwise functions are in the <literal>bit</literal>
	  namespace.

	  <variablelist>
	    <varlistentry>
	      <term><option>bit.bnot(a)</option>
              </term>
	      <listitem>
		<para>
		  Returns the one's complement of <literal>a</literal>.
		</para>
	    </listitem>
	    </varlistentry>
	    
	    <varlistentry>
	      <term><option>bit.band(w1,...)</option>
		</term>
	      <listitem>
		<para>
		  Returns the bitwise <literal>and</literal> of the
		  <literal>w</literal> variables.
		</para>
	      </listitem>
	    </varlistentry>
	    
	    <varlistentry>
	      <term><option>bit.bor(w1,...)</option>
		</term>
	      <listitem>
		<para>
		  Returns the bitwise <literal>or</literal> of the <literal>w</literal> variables.
		</para>
	      </listitem>
	    </varlistentry>
	    
	    <varlistentry>
	      <term><option>bit.bxor(w1,...)</option>
		</term>
	      <listitem>

		<para>
		  Returns the bitwise <literal>xor</literal> of the <literal>w</literal> variables.
		</para>
	      </listitem>
	    </varlistentry>
	    
	    <varlistentry>
	      <term><option>bit.lshift(a,b)</option>
		</term>
	      <listitem>
		<para>
		  Returns <literal>a</literal> shifted left <literal>b</literal> places&mdash;padded with zeros.
		</para>
	      </listitem>
	    </varlistentry>
	    
	    <varlistentry>
	      <term><option>bit.rshift(a,b)</option>
		</term>
	      <listitem>
		<para>
		  Returns <literal>a</literal> shifted logically right <literal>b</literal> places.
		</para>
	      </listitem>
	    </varlistentry>

	    <varlistentry>
	      <term><option>bit.arshift(a,b)</option>
		</term>
	      <listitem>
		<para>
		  Returns <literal>a</literal> shifted arithmetically right <literal>b</literal> places.
		</para>
	      </listitem>
	    </varlistentry>
	    
	    <varlistentry>
	      <term><option>bit.mod(a,b)</option>
		</term>
	      <listitem>
		<para>
		  Returns the integer remainder of <literal>a</literal> divided by <literal>b</literal>.
		</para>
	      </listitem>
	    </varlistentry>
	  </variablelist>
	</para>
      </sect2>

      <sect2 id="nse-binlib">
        <title>Binary Data Handling</title>
        <indexterm><primary><varname>bin</varname> NSE module</primary></indexterm>
        <para>
	  A problem script authors often face is the necessity of encoding values
	  into binary data. For example after analyzing a protocol the starting 
	  point to write a script could be a hex dump, which serves as a preamble
	  to every sent packet. Although it is possible work with the 
	  functionality Lua provides, it's not very convenient. Therefore the 
	  Binlib has been added to NSE, based on 
	  <ulink url="http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/">lpack</ulink> 
	  by Luiz Henrique de Figueiredo.<indexterm><primary>Henrique de Figueiredo, Luiz</primary></indexterm>

	  The Binlib functions take a format string to encode and decode binary
	  data. The operators of the format string are shown in <xref linkend="scripting-tbl-binlib" xrefstyle="select: label nopage"/>.</para>

	  <table id="scripting-tbl-binlib">
	   <title>Binlib format string operators</title>
	     <tgroup cols="2">
		<colspec colwidth="2*" />
		<colspec colwidth="5*" />
	    <thead><row>
		<entry>Operator</entry>
    		<entry>Description</entry>
   	    </row></thead>
   	    <tbody>
		<row><entry><literal>H</literal></entry><entry>hex string</entry></row>
		<row><entry><literal>B</literal></entry><entry>bit string</entry></row>
		<row><entry><literal>x</literal></entry><entry>null byte</entry></row>
		<row><entry><literal>z</literal></entry><entry>zero-terminated string</entry></row>
		<row><entry><literal>p</literal></entry><entry>string preceded by 1-byte integer length</entry></row>
		<row><entry><literal>P</literal></entry><entry>string preceded by 2-byte integer length</entry></row>
		<row><entry><literal>a</literal></entry><entry>string preceded by 4-byte integer length</entry></row>
		<row><entry><literal>A</literal></entry><entry>string</entry></row>
		<row><entry><literal>f</literal></entry><entry>float</entry></row>
		<row><entry><literal>d</literal></entry><entry>double</entry></row>
		<row><entry><literal>n</literal></entry><entry>Lua number</entry></row>
		<row><entry><literal>c</literal></entry><entry>char (1-byte integer)</entry></row>
		<row><entry><literal>C</literal></entry><entry>byte = unsigned char (1-byte unsigned integer)</entry></row>
		<row><entry><literal>s</literal></entry><entry>short (2-byte integer)</entry></row>
		<row><entry><literal>S</literal></entry><entry>unsigned short (2-byte unsigned integer)</entry></row>
		<row><entry><literal>i</literal></entry><entry>int (4-byte integer)</entry></row>
		<row><entry><literal>I</literal></entry><entry>unsigned int (4-byte unsigned integer)</entry></row>
		<row><entry><literal>l</literal></entry><entry>long (8-byte integer)</entry></row>
		<row><entry><literal>L</literal></entry><entry>unsigned long (8-byte unsigned integer)</entry></row>
		<row><entry><literal>&lt;</literal></entry><entry>little endian modifier</entry></row>
		<row><entry><literal>&gt;</literal></entry><entry>big endian modifier</entry></row>
		<row><entry><literal>=</literal></entry><entry>native endian modifier</entry></row>
	   </tbody></tgroup></table>

	   <para>Note that the endian operators work as modifiers to all the characters following them in the format string.</para>

	  <variablelist>
	    <varlistentry>
	      <term><option>bin.pack(fmt, p1, ...)</option>
              </term>
	      <listitem>
		<para>
		  Returns a binary packed string. The format string describes how 
		  the parameters (p1, ...) will be interpreted. Numerical values following 
		  operators stand for operator repetitions and need an according amount of 
		  parameters. Operators expect appropriate parameter types.
		  
		</para>
	    </listitem>
	    </varlistentry>
	    
	    <varlistentry>
	      <term><option>bin.unpack(fmt, data, [init])</option>
		</term>
	      <listitem>
		<para>
		  Returns values read from the binary data string.
		  First result is the position, at which unpack stopped. This can 
		  be used as init value for subsequent calls. The following results 
		  are the values according to the format string. Numerical values in 
		  the format string are interpreted as repetitions like in pack, 
		  except if used with A, B or H, in which cases the number tells unpack
		  how many bytes to read.

		  Unpack stops if either the format string or the binary data string
		  are exhausted.
		</para>
	      </listitem>
	    </varlistentry>
	  </variablelist>
      </sect2>


      <sect2 id="nse-pcre">
        <indexterm class="startofrange" id="nse-pcre-indexterm"><primary><varname>pcre</varname> NSE module</primary></indexterm>
        <indexterm><primary>Perl Compatible Regular Expressions (PCRE)</primary><secondary>in NSE</secondary></indexterm>
        <indexterm><primary>regular expressions</primary><secondary>in NSE</secondary></indexterm>
        <title>Perl Compatible Regular Expressions</title>

        <para>
	  One of Lua's quirks is its string patterns. While they have
	  great performance and are tightly integrated into the Lua
	  interpreter, they are very different in syntax and not as
	  powerful as standard regular expressions.  So we have
	  integrated Perl compatible regular expressions into Lua
	  using PCRE and a modified version of the Lua PCRE library
	  written by Reuben Thomas<indexterm><primary>Thomas, Reuben</primary></indexterm>
          and Shmuel Zeigerman.<indexterm><primary>Zeigerman, Shmuel</primary></indexterm>
          These are
	  the same sort of regular expressions used by Nmap version
	  detection.  The main modification to their library is that
	  the NSE version only supports PCRE expressions instead of both
	  PCRE and POSIX patterns. In order to maintain a high script
	  execution speed, the library interfacing with PCRE is
	  kept very thin. It is not integrated as seamlessly as the
	  Lua string pattern API. This allows script authors to decide
	  when to use PCRE expressions versus Lua patterns.  The use of PCRE
	  involves a separate pattern compilation step, which saves
	  execution time when patterns are reused.  Compiled patterns
	  can be cached in the NSE registry and reused by other
	  scripts. The PCRE functions reside inside the <literal>pcre</literal> 
	  namespace.
	  </para>

      <indexterm><primary>Perl Compatible Regular Expressions (PCRE)</primary><secondary>security vulnerabilities in</secondary></indexterm>
      <warning><para>PCRE has a history of security vulnerabilities
      allowing attackers who are able to compile arbitrary regular
      expressions to execute arbitrary code.  More such
      vulnerabilities may be discovered in the future.  These have
      never affected Nmap because it doesn't give attackers any
      control over the regular expressions it uses.  Similarly, NSE
      scripts should never build regular expressions with untrusted
      network input.  Matching hardcoded regular expressions
      <emphasis>against</emphasis> the untrusted input is
      fine.</para></warning>

      <para>The following documentation is derived from that supplied by
      the PCRE Lua lib.</para>

	<variablelist>

	    <varlistentry>
	      <term><option>pcre.new(pattern, flags, locale)</option>
		</term>
	      <listitem>
		<para>
		  Returns a compiled regular expression. The first
		  argument is a string describing the pattern, such as 
		  <literal>^foo$</literal>.  The second
		  argument is a number describing which compilation
		  flags are set. The compilation flags are set
		  bitwise. If you want to set the 3rd (corresponding to 
		  the number 4) and the 1st (corresponding to 1) bit
		  for example you would pass the number 5 as a second
		  argument. The compilation flags accepted are those 
		  of the PCRE C library. These include flags for case 
		  insensitive matching (1), matching line beginnings (^) 
		  and endings ($) even in multiline strings (i.e. strings 
		  containing <quote>\n</quote>) (2) and a flag for matching across 
		  line boundaries (4). No compilation flags yield a default 
		  value of 0. The third (optional) argument is a string 
		  describing the locale which should be used to compile the 
		  regular expression. The variable is a string which is 
		  passed to the C standard library function
		  <function>setlocale</function>.  For more
		  information on this argument refer to the
		  documentation of <function>setlocale</function>. The
		  resulting compiled regular expression is ready to be
		  matched against strings. Compiled regular
		  expressions are subject to Lua's garbage collection.
		  Generally speaking, <literal>my_regex = pcre.new("<replaceable>pcre-pattern</replaceable>",0,"C")</literal>
		  should do the job most of the time.
		</para>
	      </listitem>
	    </varlistentry>


	    <varlistentry>
	      <term><option>pcre.flags()</option>
		</term>
	      <listitem>
		<para>
		  Returns a table of the available PCRE option flags
		  (numbers) keyed by their names (strings). Possible
		  names of the available strings can be retrieved from
		  the documentation of the PCRE library used to link
		  against Nmap. The key is the option name in the
		  manual minus the <literal>PCRE</literal>
		  prefix. <literal>PCRE_CASELESS</literal> becomes
		  <literal>CASELESS</literal> for example.
		</para>
	      </listitem>
	    </varlistentry>

	    <varlistentry>
	      <term><option>pcre.version()</option>
		</term>
	      <listitem>
		<para>
		  Returns the version of the PCRE library in use as a
		  string.  For example <literal>6.4 05-Sep-2005</literal>.
		</para>
	      </listitem>
	    </varlistentry>

	    <varlistentry>
	      <term><option>pcre_obj:match(string, start, flags)</option>
		</term>
	      <listitem>
		<para>
		  Returns the start point and the end point of
		  the first match of the compiled regular expression
		  pcre_obj in the string.  A third
		  returned value is a table which contains
		  <literal>false</literal> in the positions where the
		  pattern did not match. If named sub-patterns were
		  used, the table also contains substring matches keyed
		  by their sub-pattern name. Should no match be found the 
		  function returns <literal>nil</literal>.
		  The second and third arguments are optional. The second 
		  argument is a number specifying where the engine should 
		  start trying to apply the pattern. The third argument
		  specifies execution flags for the pattern. 
		  If you want to see if a given string matches a certain expression
		  you could use:</para>

<programlisting>
s = pcre_obj:match("string to be searched", 0,0);
if(s) code_to_be_done_on_match end
</programlisting>

	      </listitem>
	    </varlistentry>

	    <varlistentry>
	      <term><option>pcre_obj:exec(string, start, flags)</option>
		</term>
	      <listitem>
		<para>
		  This function is like <literal>match()</literal> except that a table returned as
		  a third result contains offsets of substring matches rather
		  than substring matches themselves. That table will not
		  contain string keys, even if named sub-patterns are used.  For
		  example, if the whole match is at offsets <literal>10, 20</literal> and substring
		  matches are at offsets <literal>12, 14</literal> and <literal>16, 19</literal> then the function
		  returns the following: <literal>10, 20, {12,14,16,19}</literal>
		</para>
	      </listitem>
	    </varlistentry>

	    <varlistentry>
	      <term><option>pcre_obj:gmatch(string, func, n, ef)</option>
		</term>
	      <listitem>
		<para>
		  Tries to match the regular expression <replaceable>pcre_obj</replaceable> against <replaceable>string</replaceable>
		  up to <replaceable>n</replaceable> times (or as many as possible if <replaceable>n</replaceable> is either
		  not given or is not a positive number), subject to
		  execution flags ef. Each time there is a match, <replaceable>func</replaceable>
		  is called as <replaceable>func(m, t)</replaceable>, where <replaceable>m</replaceable> is the matched
		  string and <replaceable>t</replaceable> is a table of substring matches. This
		  table contains <literal>false</literal> in the
		  positions where the corresponding sub-pattern did
		  not match. If named sub-patterns are used then the
		  table also contains substring matches keyed by their
		  correspondent sub-pattern names (strings).  If <replaceable>func</replaceable>
		  returns a <literal>true</literal> value, then gmatch
		  immediately returns; gmatch returns the number of
		  matches made.
		</para>
	      </listitem>
	    </varlistentry>

	</variablelist>
        <indexterm class="endofrange" startref="nse-pcre-indexterm"/>
    </sect2>
	
	<sect2 id="nse-lib-ipOps">
	<title>IP Operations</title>
        <indexterm><primary><varname>ipOps</varname> NSE module</primary></indexterm>
	<para>
	The <literal>ipOps</literal> module provides some functions for 
	manipulating IPv4 addresses. The functions reside inside the 
	<literal>ipOps</literal> namespace.
	</para>
	<variablelist>
	    <varlistentry>
	      <term>
              <indexterm><primary>private addresses</primary><secondary>in NSE</secondary></indexterm>
              <option>bool = ipOps.isPrivate("ip-string")</option>
		  </term>
	      <listitem>
		  <para>
		  checks whether an IP address, provided as a string in 
		  dotted-quad notation, is part of the non-routed private IP address 
		  space, as described in <ulink role="hidepdf" url="http://www.rfc-editor.org/rfc/rfc1918.txt">RFC 1918</ulink>. These addresses are the well-known 
		  <literal>10.0.0.0/8</literal>, <literal>192.168.0.0/16</literal> and
		  <literal>172.16.0.0/12</literal> networks.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>DWORD = ipOps.todword("ip-string")</option>
		  </term>
	      <listitem>
		  <para>
		  returns the IP address as DWORD value (i.e. the IP <replaceable>a.b.c.d</replaceable> becomes
		  <literal>(((a*256+b)*256+c)*256+d)</literal> )
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>a,b,c,d = ipOps.get_parts_as_number("ip-string")</option>
		  </term>
	      <listitem>
		  <para>
		  returns 4 numbers corresponding to the fields in dotted-quad notation.
		  For example, <literal>ipOps.get_parts_as_number("192.168.1.1")
		  </literal> returns <literal>192,168,1,1</literal>.
		  </para>
	      </listitem>
	    </varlistentry>
	</variablelist>
	</sect2>
	<sect2 id="nse-lib-shortport">
	<title>Short Portrules</title>
        <indexterm><primary><varname>shortport</varname> NSE module</primary></indexterm>
	<para>
	Since portrules are mostly the same for many scripts, the 
        <literal>shortport</literal> module provides functions for the most common tests.
	The arguments in brackets (<literal>[]</literal>) are optional. If no 
        <literal>proto</literal> is provided, <literal>tcp</literal> is used. The default 
	<literal>state</literal> is <literal>open</literal>
	</para>
	<variablelist>
	    <varlistentry>
	      <term><option>shortport.portnumber(port,[proto],[state])</option>
		  </term>
	      <listitem>
		  <para>
		  The port argument is either a number or a table of numbers which are
		  interpreted as port numbers, against which the script should run.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>shortport.service(service,[proto],[state])</option>
		  </term>
	      <listitem>
		  <para>
		  The service argument is either a string or a table 
		  of strings which are interpreted as service names
		  (e.g. <literal>"http"</literal>, <literal>"https"</literal>, <literal>"smtp"</literal> or <literal>"ftp"</literal>) against which the
		  script should run. These service names are
		  determined by Nmap's version scan or (if no version
		  scan information is available) the service assigned
		  to the port in <filename>nmap-services</filename>
		  (e.g. <literal>"http"</literal> for TCP port 80).
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>shortport.port_or_service(port,service,[proto],[state])</option>
		  </term>
	      <listitem>
		  <para>
		  This is a combination of the above functions, since many scripts
		  explicitly try to run against the well-known ports, but want 
		  also to run against any other port which was discovered to run the 
		  named service. A typical example for this function is: 
		  <literal>portrule = shortport.port_or_service(22,"ssh")</literal>.
		  </para>
	      </listitem>
	    </varlistentry>
	</variablelist>
	</sect2>
	<sect2 id="nse-lib-listop">
	<title>Functional Programming Style List Operations</title>
        <indexterm><primary><varname>listop</varname> NSE module</primary></indexterm>
	<para>
	People used to programming in functional languages, such as Lisp or
        Haskell, appreciate their handling of lists very much. The <literal>listop</literal> module tries to bring much of the functionality from
	functional languages to Lua using Lua's central data structure, the table,
	as a base for its list operations. Highlights include a <literal>map</literal> 
	function applying a given function to each element of a list.
	</para>
	<variablelist>
	    <varlistentry>
	      <term><option>bool = listop.is_empty(list)</option>
		  </term>
	      <listitem>
		  <para>
		  Returns <literal>true</literal> if the given list is empty.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>bool = listop.is_list(value)</option>
		  </term>
	      <listitem>
		  <para>
		  Returns <literal>true</literal> if the given value is a list (or rather a table).
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>list = listop.map(function, list)</option>
		  </term>
	      <listitem>
		  <para>
		  The provided function is applied to each element of the list 
		  separately. The returned list contains the results of each 
		  function call. For example <literal>listop.map(tostring,{1,2,true})
		  </literal> returns <literal>{"1","2","true"}</literal>.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>value = listop.apply(function, list)</option>
		  </term>
	      <listitem>
		  <para>
		  All of the elements in the list are passed to a call of <literal>
		  function</literal>. The result is then returned. For example
		  <literal>listop.apply(math.max,{1,5,6,7,50000})</literal>
		  yields <literal>50000</literal>.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>list = listop.filter(predicate, list)</option>
		  </term>
	      <listitem>
		  <para>
		  Returns a list containing only those elements for which the predicate
		  returns true. The predicate has to be a function, which takes an 
		  element of the list as argument and the result of which
		  is interpreted as a Boolean value. If it returns true (or rather 
		  anything besides <literal>false</literal> and <literal>nil</literal>)
		  the argument is appended to the return value of <literal>filter</literal>. 
		  For example: <literal>listop.filter(isnumber,{1,2,3,"foo",4,"bar"})</literal> returns
		  <literal>{1,2,3,4}</literal>.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>list = listop.flatten(list)</option>
	      </term>
	      <listitem>
		  <para>
		  Since a list can itself contain lists as elements,
		  <literal>flatten</literal> returns a list which
		  only contains values that are not themselves
		  lists. For example:
		  <literal>listop.flatten({1,2,3,"foo",{4,5,{"bar"}}})</literal> returns
		  <literal>{1,2,3,"foo",4,5,"bar"}</literal>.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>list = listop.append(list1, list2)</option>
		  </term>
	      <listitem>
		  <para>
		  Returns a list containing all elements of list1 appended by all 
		  elements of <replaceable>list2</replaceable>.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>list = listop.cons(value1, value2)</option>
		  </term>
	      <listitem>
		  <para>
		  Returns a list containing <replaceable>value1</replaceable> appended by <replaceable>value2</replaceable>, which may be
		  of any type.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>list = listop.reverse(list)</option>
		  </term>
	      <listitem>
		  <para>
		  Returns a list containing all elements of the given list in inverted
		  order.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>value = listop.car(list)</option>
		  </term>
	      <listitem>
		  <para>
		  Returns the first element of the given list.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>value = listop.ncar(list,n)</option>
		  </term>
	      <listitem>
		  <para>
		  Returns the nth (or first if n is omitted) element of the given list.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>value = listop.cdr(list)</option>
		  </term>
	      <listitem>
		  <para>
		  Returns a list containing all elements but the first of the 
		  given list.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>value = listop.ncdr(list, n)</option>
		  </term>
	      <listitem>
		  <para>
		  Returns a list containing all elements but the first n of the 
		  given list, where n is 2 if it is omitted.
		  </para>
	      </listitem>
	    </varlistentry>
	</variablelist>
	</sect2>
	<sect2 id="nse-lib-strbuf">
	<title>String Buffer Operations</title>
        <indexterm><primary><varname>strbuf</varname> NSE module</primary></indexterm>
	<para>
	Lua's string operations are very flexible and offer an easy-to-use way
	to manipulate strings.  Concatenation using the <literal>..</literal>
	operator is such an operation. The drawback of the built-in API however is the way it handles 
	concatenation of many string values. Since strings in Lua are 
	immutable values, each time you concatenate two strings both get copied
	into the result string. The <literal>strbuf</literal> module offers a
	workaround for this problem, while maintaining the nice syntax. This
	is accomplished by overloading the concatenation operator (<literal>..</literal>) the equality operator (<literal>==</literal>) and the 
	tostring operator.  By overloading
        these operators, we reduce the overhead of using a string buffer instead
        of a plain string to wrap the first literal string assigned to a
        variable inside a <literal>strbuf.new()</literal> call.  Afterwards you can append to the string buffer, or compare
	two string buffers for equality just as you would do with normal strings.
	When looking at the details there are some more restrictions/oddities:
        The concatenation operator requires its left-hand value to be a
        string buffer. Therefore, if you want to prepend a string to a given
        string buffer you have to create a new string buffer out of the string
        you want to prepend.
	The string buffer's <literal>tostring</literal> operator concatenates the 
	strings inside the buffer using newlines by default, since this appears to 
	be the separator used most often.
	</para>
	<variablelist>
	    <varlistentry>
	      <term><option>buffer = strbuf.new(...)</option>
		  </term>
	      <listitem>
		  <para>
            Creates a new string buffer. The optional arguments are added
            to the string buffer. Attempting to add non-strings will
            result in undefined behavior.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>buffer = strbuf.concat(strbuf1, value)</option>
		  </term>
	      <listitem>
		  <para>
		  Concatenates the <literal>value</literal> (which has to be either 
		  a string or a string buffer) to <literal>strbuf1</literal>. This
		  is also the function serving as the string buffer's concatenation operator.
		  The above function call can thus also be expressed as: 
		  <literal>buffer = strbuf1 .. value</literal>
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>bool = strbuf.eqbuf(strbuf1, strbuf2)</option>
		  </term>
	      <listitem>
		  <para>
		  Compares <literal>strbuf1</literal> and <literal>strbuf2</literal>
		  for equality. For the function to return <literal>true</literal>, both values must be 
		  string buffers containing exactly the same strings. The <literal>eqbuf</literal> function is called to compare two strings for equality.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>strbuf.clear(strbuf)</option>
		  </term>
	      <listitem>
		  <para>
		  Deletes all strings in <literal>strbuf</literal>.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>string = strbuf.dump(strbuf, "delimiter")</option>
		  </term>
	      <listitem>
		  <para>
		  Dumps <literal>strbuf</literal>'s contents as string. The second 
		  parameter is used as a delimiter between the strings stored inside
		  <literal>strbuf</literal>. <literal>dump(strbuf, "\n")</literal> is 
		  used as the <literal>tostring</literal> function of string buffers.
		  </para>
	      </listitem>
	    </varlistentry>
	</variablelist>
	</sect2>
	<sect2 id="nse-lib-url">
	<title>URL Manipulation Functions</title>
        <indexterm><primary><varname>url</varname> NSE module</primary></indexterm>

	<para>URL manipulation functions have obvious uses. Fortunately
        there is already an implementation of URL generation functions
        inside the Lua <varname>socket</varname> package, which is fairly complete and
        <ulink
        url="http://www.cs.princeton.edu/~diego/professional/luasocket/old/luasocket-2.0-alpha/url.html">well
        documented</ulink>. For NSE, the <varname>url</varname> module was
        extended with two functions:</para>

	<variablelist>
	    <varlistentry>
	      <term><option>table = url.parse_query("query-string")</option>
		  </term>
	      <listitem>
		  <para>
		  This function takes a <replaceable>query-string</replaceable> of the form <literal>name1=value1&amp;name2=value2...</literal> and returns a table 
		  containing the name-value pairs, with the <literal>name</literal> 
		  as the key and the <literal>value</literal> as its associated value.
		  The table corresponding to the above <replaceable>query-string</replaceable> would have two 
		  entries: <literal>table["name1"]="value1"</literal> and 
		  <literal>table["name2"]="value2"</literal>.</para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>query_string = url.build_query(table)</option>
		  </term>
	      <listitem>
		  <para>
		  This is the inverse function to <literal>parse_query()</literal>.
		  </para>
	      </listitem>
	    </varlistentry>
	</variablelist>
	</sect2>
	<sect2 id="nse-lib-match">
	<title>Buffered Network I/O Helper Functions</title>
        <indexterm><primary><varname>match</varname> NSE module</primary></indexterm>
	<para>
	The <literal>match</literal> module was written to provide
	functions which can be used for delimiting data received by the
	<literal>receive_buf()</literal> function from the Network I/O API: 
	</para>
	<variablelist>
	    <varlistentry>
	      <term><option>start,end = match.regex("regexpattern")</option>
		  </term>
	      <listitem>
		  <para>
		  	This is actually a wrapper around NSE's PCRE library <literal>exec</literal> function (see <xref linkend="nse-pcre"/>), thus
			giving script developers the possibility to use regular expressions
			for delimiting instead of Lua's string patterns. If you want to get
			the data in chunks separated by <literal>pattern</literal> (which has to be a valid
			regular expression), you would write <literal>status, val =
			sockobj:receive_buf(match.regex("pattern"))</literal>.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>start,end = match.numbytes(number)</option>
		  </term>
	      <listitem>
		  <para>
		  	Takes a number as its argument and returns that
                        many bytes.  It can be used to get a buffered
                        version of
                        <literal>sockobj:receive_bytes(n)</literal> in
                        case a script requires more than one
                        fixed-size chunk, as the unbuffered version
                        may return more bytes than requested and thus
                        would require you to do the parsing on your
                        own.
		  </para>
	      </listitem>
	    </varlistentry>
	</variablelist>
	</sect2>
	<sect2 id="nse-lib-http">
	<title>HTTP Functions</title>
        <indexterm><primary><varname>http</varname> NSE module</primary></indexterm>
	<para>
	The <literal>http</literal> module provides functions for dealing with the client side of the http protocol.
	The functions reside inside the <literal>http</literal> namespace.
	The return value of each function in this module is a table with the following keys:
	<literal>status</literal>, <literal>header</literal> and <literal>body</literal>.

	<literal>status</literal> is a number representing the HTTP
	status code returned in response to the HTTP request. In case
	of an unhandled error, <literal>status</literal>
	is <literal>nil</literal>. The <literal>header</literal> value
	is a table containing key-value pairs of HTTP headers received
	in response to the request. The header names are in lower-case
	and are the keys to their corresponding header values
	(e.g. <literal>header.location =
	"http://nmap.org/"</literal>).  Multiple headers of the same
	name are concatenated and separated by
	commas. The <literal>body</literal> value is a string
	containing the body of the HTTP response.</para>

	<variablelist>
	    <varlistentry>
	      <term><option>table = http.get(host,port,path,[options])</option>
		  </term>
	      <listitem>
		  <para>
		  Fetches a resource with a <literal>GET</literal> request.
		  The first argument is either a string with the hostname or a
		  table like the host table passed by nmap. The second argument
		  is either the port number or a table like the port table passed
		  by nmap. The third argument is the path of the resource. The fourth
		  argument is a table for further options. The table may have 2 keys:
		  <literal>timeout</literal> and <literal>header</literal>.
		  <literal>timeout</literal> is the timeout used for the socket
		  operations. <literal>header</literal> is a table with additional
		  headers to be used for the request.
		  The function builds the request and calls <literal>http.request</literal>
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>table = http.request(host,port,request,[options])</option>
		  </term>
	      <listitem>
		  <para>
		  Sends <literal>request</literal> to <literal>host</literal>:<literal>port</literal>
		  and parses the answer.
		  The first argument is either a string with the hostname or a
		  table like the host table passed by nmap. The second argument
		  is either the port number or a table like the port table passed
		  by nmap. SSL is used for the request if either <literal>port.service</literal>
		  equals <literal>https</literal> or <literal>port.version.service_tunnel</literal>
		  equals <literal>ssl</literal>.  The third argument is the request. The fourth
		  argument is a table for further options. You can specify a timeout
		  for the socket operations with the timeout key.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>table = http.get_url(url,[options])</option>
		  </term>
	      <listitem>
		  <para>
		  Parses <literal>url</literal> and calls <literal>http.get</literal>
		  with the result.
		  The second argument is a table for further options. The table may have 2 keys:
		  <literal>timeout</literal> and <literal>header</literal>.
		  <literal>timeout</literal> is the timeout used for the socket
		  operations. <literal>header</literal> is a table with additional
		  headers to be used for the request.
		  </para>
	      </listitem>
	    </varlistentry>
	</variablelist>
	</sect2>

	<sect2 id="nse-lib-comm">
	<title>Common Communication Functions</title>
        <indexterm><primary><varname>comm</varname> NSE module</primary></indexterm>
	<para>
	The <literal>comm</literal> module provides functions for common network discovery
	tasks such as banner-grabbing and making a quick exchange of data.  These functions'
	return values are setup for use with exception handling via <literal>nmap.new_try()</literal>.
        <indexterm><primary>exceptions in NSE</primary></indexterm>
	</para>

	<para>
	These functions can all be passed a table of options, but it's not required.
	The relevant indexes for this table are <literal>bytes</literal>, <literal>lines</literal>,
	<literal>proto</literal> and <literal>timeout</literal>.  <literal>bytes</literal>
	is used to provide the minimum number of bytes required for a read.  <literal>lines</literal>
	does the same, but for the minimum number of lines.  If neither are provided, these
	functions attempt to read as many bytes as are available.  <literal>proto</literal>
	is used to set the protocol to communicate with, defaulting to <literal>"tcp"</literal> if not provided.
	<literal>timeout</literal> is used to set the socket timeout (see the socket function
	<literal>set_timeout()</literal> for details).
	</para>

	<variablelist>
	    <varlistentry>
	      <term><option>bool, response = comm.get_banner(host, port, [options])</option>
		  </term>
	      <listitem>
		  <para>
			This function simply connects to the specified port number on
			the specified host and returns any data received.
			<literal>bool</literal> is a Boolean value indicating success.
			If <literal>bool</literal> is true, then the second returned
			value is the response from the target host.  If <literal>bool</literal>
			is false, an error message is returned as the second value instead
			of a response.
		  </para>
	      </listitem>
	    </varlistentry>

	    <varlistentry>
	      <term><option>bool, response = comm.exchange(host, port, data, [options])</option>
		  </term>
	      <listitem>
		  <para>
			This function connects to the specified port number on the
			specified host, sends <literal>data</literal>, then waits for
			and returns the response, if any.  <literal>bool</literal> is a
			Boolean value indicating success.  If <literal>bool</literal> is
			true, then the second returned value is the response from the
			target host.  If <literal>bool</literal> is false, an error message
			is returned as the second value instead of a response.
		  </para>
	      </listitem>
	    </varlistentry>
	</variablelist>
	</sect2>

	<sect2 id="nse-lib-unpwdb">
	<title>Username/Password Database Functions</title>
        <indexterm><primary><varname>unpwdb</varname> NSE module</primary></indexterm>
	<para>
	The <literal>unpwdb</literal> module provides functions for easily obtaining
	usernames and/or passwords from a "database" (list).  The most obvious use
	for this library is brute-force attack.  The first two functions' return values
	are setup for use with exception handling via <literal>nmap.new_try()</literal>.
        <indexterm><primary>exceptions in NSE</primary></indexterm>
	</para>

	<variablelist>
	    <varlistentry>
	      <term><option>bool, response = unpwdb.usernames()</option></term>
	      <listitem>
		  <para>
			This function returns a closure which returns a new username
			with every call, or <literal>nil</literal> when the list is
			exhausted.  This closure takes an optional argument of
			<literal>"reset"</literal> to rewind the list to the beginning.
			You can specify your own username list with the script argument
			<literal>userdb</literal>.
		  </para>
	      </listitem>
	    </varlistentry>

	    <varlistentry>
	      <term><option>bool, response = unpwdb.passwords()</option></term>
	      <listitem>
		  <para>
			This function returns a closure which returns a new password
			with every call, or <literal>nil</literal> when the list is
			exhausted.  This closure takes an optional argument of
			<literal>"reset"</literal> to rewind the list to the beginning.
			You can specify your own password list with the script argument
			<literal>passdb</literal>.
		  </para>
	      </listitem>
	    </varlistentry>

	    <varlistentry>
	      <term><option>limit = unpwdb.timelimit()</option></term>
	      <listitem>
		  <para>
			This function returns the suggested number of seconds to
			attempt a brute force attack, based on Nmap's timing values
			(<option>-T4</option>, etc) and whether or not a user-defined
			list is used.  You can use the script argument
			<literal>notimelimit</literal> to make this function return
			<literal>nil</literal>, which means the brute-force should
			run until the list is empty.  If <literal>notimelimit</literal>
			is not used, be sure to still check for <literal>nil</literal>
			return values on the above two functions in case you finish
			before the time limit is up.
		  </para>
	      </listitem>
	    </varlistentry>

	</variablelist>
	</sect2>

	<sect2 id="nse-lib-datafiles">
	<title>Data File Parsing Functions</title>
        <indexterm><primary><varname>datafiles</varname> NSE module</primary></indexterm>
        <indexterm><primary>data files</primary><secondary>access to from NSE</secondary></indexterm>
	<para>
	The <literal>datafiles</literal> module provides functions for reading and parsing
	Nmap's data files (e.g. <filename>nmap-protocol</filename>, <filename>nmap-rpc</filename>,
	etc.).  These functions' return values are setup for use with exception handling via
	<literal>nmap.new_try()</literal>.
	</para>

	<variablelist>
	    <varlistentry>
	      <term><option>bool, table = datafiles.parse_protocols()</option>
		  </term>
	      <listitem>
		  <para>
			This function reads and parses Nmap's <filename>nmap-protocols</filename>
			file.  <literal>bool</literal> is a Boolean value indicating success.
			If <literal>bool</literal> is true, then the second returned
			value is a table with protocol numbers indexing the protocol
			names.  If <literal>bool</literal> is false, an error message
			is returned as the second value instead of the table.
		  </para>
	      </listitem>
	    </varlistentry>

	    <varlistentry>
	      <term><option>bool, table = datafiles.parse_rpc()</option>
		  </term>
	      <listitem>
		  <para>
			This function reads and parses Nmap's <filename>nmap-rpc</filename>
			file.  <literal>bool</literal> is a Boolean value indicating success.
			If <literal>bool</literal> is true, then the second returned
			value is a table with RPC numbers indexing the RPC names.  If
			<literal>bool</literal> is false, an error message is returned
			as the second value instead of the table.
		  </para>
	      </listitem>
	    </varlistentry>

	    <varlistentry>
	      <term><option>bool, table = datafiles.parse_services([protocol])</option>
		  </term>
	      <listitem>
		  <para>
			This function reads and parses Nmap's <filename>nmap-services</filename>
			file.  <literal>bool</literal> is a Boolean value indicating success.
			If <literal>bool</literal> is true, then the second returned
			value is a table containing two other tables:
			<literal>tcp{}</literal> and <literal>udp{}</literal>.
			<literal>tcp{}</literal> contains services indexed by TCP port
			numbers.  <literal>udp{}</literal> is the same, but for UDP.
			You can pass <literal>"tcp"</literal> or <literal>"udp"</literal> as an argument to
			<literal>parse_services()</literal> to only get the corresponding
			table.  If <literal>bool</literal> is false, an error message is
			returned as the second value instead of the table.
		  </para>
	      </listitem>
	    </varlistentry>
	</variablelist>

	</sect2>
	<sect2 id="nse-lib-stdnse">
	<title>Various Utility Functions</title>
        <indexterm><primary><varname>stdnse</varname> NSE module</primary></indexterm>
	<para>
	The <literal>stdnse</literal> library contains various handy
	functions which are too small to justify modules of their own:
	</para>
	<variablelist>
	    <varlistentry>
	      <term><option>stdnse.print_debug([verbosity,] format, ...)</option>
		  </term>
	      <listitem>
		  <para>
		  Wrapper function around <literal>print_debug_unformatted()</literal>
		  in the <literal>nmap</literal> namespace. The first optional numeric
          argument, <literal>verbosity</literal>, is
		  used as the necessary debug level to print the message (it defaults
		  to 1 if omitted). All remaining arguments are processed with
		  Lua's <literal>string.format()</literal> function, which provides a 
		  C-style printf interface.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>list = stdnse.strsplit("delimiter", "text")</option>
		  </term>
	      <listitem>
		  <para>
		  This function will certainly be appreciated by Perl programmers.
		  It takes two strings as arguments and splits the second one around
		  all occurrences of the first one, returning a list (table), which
          contains the substrings without the delimiting string.
		  </para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>string = stdnse.strjoin("delimiter", list)</option>
		  </term>
	      <listitem>
		  <para>
		  Inverse function to <literal>strsplit()</literal>. Basically this is
		  Lua's <literal>table.concat()</literal> function with the parameters
		  swapped for coherence.
		  </para>
	      </listitem>
	    </varlistentry>
        <varlistentry>
          <term><option>string = stdnse.tobinary(n)</option>
          </term>
          <listitem>
            <para>
              Converts the given number, <literal>n</literal>, to a string
              in a binary number format (e.g. 5 becomes "101").
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>string = stdnse.tooctal(n)</option>
          </term>
          <listitem>
            <para>
              Converts the given number, <literal>n</literal>, to a string
              in an octal number format (e.g. 9 becomes "11").
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>string = stdnse.tohex(n)</option>
          </term>
          <listitem>
            <para>
              Converts the given number, <literal>n</literal>, to a string
              in a hexadecimal number format (e.g. 10 becomes "a").
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>string = stdnse.make_buffer(socket, sep)</option>
          </term>
          <listitem>
            <para>
              This function operates on a socket attempting to read data.
              It separates the data by <literal>sep</literal> and, for each
              invocation, returns a piece of the separated data. Typically
              this is used to iterate over the lines of data received from a
              socket (<literal>sep = "\r?\n"</literal>). The returned string
              does not include the separator. It will return the final data
              even if it is not followed by the separator. Once an error or
              EOF is reached, it returns <literal>nil, msg</literal>.
              <literal>msg</literal> is what is returned by
              <literal>nmap.receive_lines()</literal>.
            </para>
          </listitem>
        </varlistentry>
	</variablelist>
	</sect2>
    <indexterm class="endofrange" startref="nse-library-indexterm"/>
  </sect1>

  <sect1 id="nse-api">
    <title>Nmap API</title>
    <indexterm class="startofrange" id="nse-nmap-indexterm"><primary><varname>nmap</varname> NSE module</primary></indexterm>
    <indexterm><primary>Nmap Scripting Engine (NSE)</primary><secondary>API</secondary></indexterm>
    <para>
      NSE scripts have access to several Nmap facilities for writing
      flexible and elegant scripts. The API provides target host
      details such as port states and version detection results.  It
      also offers an interface to the Nsock<indexterm><primary>Nsock</primary></indexterm>
      library
      for efficient network I/O.
    </para>

    <sect2 id="nse-api-arguments">
      <title>Information Passed to a Script</title>
      <para>
	An effective Nmap scripting engine requires more than just a
	Lua interpreter. Users need easy access to the information
	Nmap has learned about the target hosts. This data is passed
	as arguments to the NSE
	<literal>action</literal> method.<indexterm><primary sortas="action script variable">&ldquo;<varname>action</varname>&rdquo; script variable</primary></indexterm>
        The arguments, <literal>host</literal> and
	<literal>port</literal>, are Lua tables which contain
	information on the target against which the script is
	executed. The following list describes each variable in the
	<literal>host</literal> and <literal>port</literal> tables.
      </para>

      <para>
	<variablelist>
	  <varlistentry>
	    <term><option>host</option>
	      </term>
	    <listitem>
	      <para>
		This table is passed as a parameter to the rule and action
		functions. It contains information on the operating system run by
		the host (if the <option>-O</option> switch was supplied), the
		IP address and the host name of the scanned target.
	      </para>
	    </listitem>
	  </varlistentry>

	  <varlistentry>
	    <term><option>host.os</option>
	      </term>
	    <listitem>
	      <para>
		The <literal>os</literal> entry in the host table is
		an array of strings. The strings (maximally 8) are the
		names of the operating systems the target is possibly
		running. Strings are only entered in this array if the
		target machine is a perfect match for one or more OS
		database entries.  If Nmap was run without the
		<option>-O</option> option, then
		<literal>host.os</literal> is <literal>nil</literal>.
	      </para>
	    </listitem>
	  </varlistentry>
	  
	  <varlistentry>
	    <term><option>host.ip</option>
	      </term>
	    <listitem>
	      <para>Contains a string representation of the IP address of the
		target host. If the scan was run against a host name and the
		reverse DNS query returned more than one IP addresses then the
		same IP address is used as the one chosen for the scan.
	      </para>
	    </listitem>
	  </varlistentry>
	  
	  <varlistentry>
	    <term><option>host.name</option>
	      </term>
	    <listitem>
	      <para>Contains the reverse DNS entry of the scanned target host
		represented as a string. If the host has no reverse DNS entry,
		the value of the field is an empty string.
	      </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>host.targetname</option>
	      </term>
	    <listitem>
	      <para>Contains the name of the host as specified on the command line.
		If the target given on the command line contains a netmask or is an IP
		address the value of the field is <literal>nil</literal>.
	      </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>host.directly_connected</option>
	      </term>
	    <listitem>
	      <para> A Boolean value indicating whether or not the target host is
		  directly connected (i.e. on the same network segment).
		  </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>host.mac_addr</option>
	      </term>
	    <listitem>
	      <para>MAC address<indexterm><primary>MAC address</primary></indexterm>
                  of the destination host (6-byte long binary
		  string) or <literal>nil</literal>, if the host is not directly connected. 
		  </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>host.mac_addr_src</option>
	      </term>
	    <listitem>
	      <para>Our own MAC address, which was used to connect to the
		  host (either our network card's, or (with
		  <option>--spoof-mac</option>)<indexterm><primary><option>--spoof-mac</option></primary></indexterm>
                  the spoofed address).
		  </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>host.interface</option>
	      </term>
	    <listitem>
	      <para>A string containing the interface name
	      (dnet-style)<indexterm><primary>libdnet</primary></indexterm>
              through 
		  which packets to the host are sent.
		  </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>host.bin_ip</option>
	      </term>
	    <listitem>
	      <para>The target host's IPv4 address as 4 byte long binary value.
		  </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>host.bin_ip_src</option>
	      </term>
	    <listitem>
	      <para>Our host's (running Nmap) source IPv4 address as 4 byte long binary value.
		  </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>port</option>
	      </term>
	    <listitem>
	      <para>
		The port table is passed to the Lua script in the same
		fashion as the host table. It contains information about the port
		against which the script is running. If the script is run
		according to a host rule, then no port table is passed to the
		script. Port states on the target can still be requested from Nmap
		using the <literal>nmap.get_port_state()</literal> call.
	      </para>
	    </listitem>
	  </varlistentry>

	  <varlistentry>
	    <term><option>port.number</option>
	      </term>
	    <listitem>
	      <para>
		Contains the number of the currently scanned port.
	      </para>
	    </listitem>
	  </varlistentry>

	  
	  <varlistentry>
	    <term><option>port.protocol</option>
	      </term>
	    <listitem>
	      <para>
		Defines the protocol of the port. Valid values are
		<literal>"tcp"</literal> and <literal>"udp"</literal>.
	      </para>
	    </listitem>
	  </varlistentry>

	  
	  <varlistentry>
	    <term><option>port.service</option>
	      </term>
	    <listitem>
	      <para>
		Contains a string representation of the service running on
		<literal>port.number</literal> as detected by the Nmap service
		detection. If the <literal>port.version</literal> field is
		<literal>nil</literal> then Nmap has guessed the service based
		only on the port number. Otherwise this field is equal to
		<literal>port.version.name</literal>.
	      </para>
	    </listitem>
	  </varlistentry>
	  
	  <varlistentry>
	    <term><option>port.version</option>
	      </term>
	    <listitem>
	      <para>
		This entry is a table which contains information
		retrieved by the Nmap version scanning engine.  Some
		of the values (like service name, service type
		confidence, RPC related values) may be retrieved by
		Nmap even if a version scan was not required.  Values
		which were not retrieved default to
		<literal>nil</literal>.  The meaning of each value is given in the following table:</para>

<table id="scripting-tbl-port-version-values">
 <title><literal>port.version</literal> values</title>
 <tgroup cols="2">
 <colspec colwidth="2*" />
 <colspec colwidth="5*" />
   <thead><row>
    <entry>Name</entry>
    <entry>Description</entry>
   </row></thead>
   <tbody>
   <row>
    <entry><literal>name</literal></entry>
    <entry>Contains the service name Nmap will use for the port.</entry>
   </row>

   <row>
    <entry><literal>name_confidence</literal></entry>
    <entry>Evaluates how confident the version detection is about the accuracy of <literal>name</literal>, from 1 (least confident) to 10.</entry>
   </row>

   <row>
    <entry><literal>product</literal>, <literal>version</literal>, <literal>extrainfo</literal>, <literal>hostname</literal>, <literal>ostype</literal>, <literal>devicetype</literal></entry>
    <entry>These five variables are described in <xref linkend="vscan-versioninfo"/>.
   </entry>
   </row>

   <row>
    <entry><literal>service_tunnel</literal></entry>
    <entry>Contains the string <literal>"none"</literal> or <literal>"ssl"</literal> based on whether or not Nmap used SSL tunneling to detect the service.</entry>
   </row>

   <row>
    <entry><literal>service_fp</literal></entry>
    <entry>The service fingerprint, if any, is provided in this value.  This is described in 
<xref linkend="vscan-community"/>.
</entry>
   </row>

   <row>
    <entry><literal>rpc_status</literal></entry>

    <entry>Contains a string value of <literal>good_prog</literal> if
    we were able to determine the program number of an RPC service
    listening on the port, <literal>unknown</literal> if the port
    appears to be RPC but we couldn't determine the program
    number, <literal>not_rpc</literal> if the port doesn't appear be
    RPC, or <literal>untested</literal> if we haven't checked for RPC
    status.</entry>
   </row>

   <row>
    <entry><literal>rpc_program</literal>, <literal>rpc_lowver</literal>, <literal>rpc_highver</literal></entry>
    <entry>The detected RPC program number and the range of version
    numbers supported by that program.  These will be
    <literal>nil</literal> if <literal>rpc_status</literal> is
    anything other than <literal>good_prog</literal>.</entry>
   </row>
</tbody></tgroup></table>

	    </listitem>
	  </varlistentry>
	  
	  <varlistentry>
	    <term><option>port.state</option>
	      </term>
	    <listitem>
	      <para>
		Contains information on the state of the port.
		Service scripts are only run against ports in the
		<literal>open</literal> or
		<literal>open|filtered</literal> states, so
		<literal>port.state</literal> generally contains one
		of those values. Other values might appear if the port
		table is a result of the
		<literal>get_port_state</literal> function.  You can
		adjust the port state using the
		<literal>nmap.set_port_state()</literal> call.  This is
		normally done when an <literal>open|filtered</literal>
		port is determined to be <literal>open</literal>.
	      </para>
	    </listitem>
	  </varlistentry>


	</variablelist>
      </para>

      <para>
      Scripts also have access to some of Nmap&rsquo;s functions and state
      variables that are exposed through functions in the <literal>nmap</literal>
      table.
      <variablelist>
        <varlistentry>
          <term><option>nmap.debugging()</option>
            </term>
	  <listitem>
	    <para>
            Returns the
	    debugging level<indexterm><primary>debugging</primary><secondary>in NSE</secondary></indexterm>
            as a non-negative integer. The
            debugging level can be set with the
	    <option>-d</option><indexterm><primary><option>-d</option></primary></indexterm>
            option<bookex> (see <xref linkend="port-scanning-options-output"/>)</bookex>.
            </para>
          </listitem>
        </varlistentry> 
        <varlistentry>
          <term><option>nmap.have_ssl()</option>
            </term>
	  <listitem>
	    <para>
            Returns true if Nmap was compiled with
	    SSL support,<indexterm><primary>SSL</primary><secondary>in NSE</secondary></indexterm>
            false
            otherwise. This can be used to avoid sending SSL probes
            when SSL is not available.
            </para>
          </listitem>
        </varlistentry> 
        <varlistentry>
          <term><option>nmap.verbosity()</option></term>
	  <listitem>
	    <para>
            Returns the
	    verbosity level<indexterm><primary>verbosity</primary><secondary>in NSE</secondary></indexterm>
            as a non-negative integer. The
            verbosity level can be set with the
	    <option>-v</option><indexterm><primary><option>-v</option></primary></indexterm>
            option<bookex> (see <xref linkend="port-scanning-options-output"/>)</bookex>.
            </para>
          </listitem>
        </varlistentry> 
        <varlistentry>
          <term>
            <option>nmap.fetchfile(filename)</option>
          </term>
	  <listitem>
            <indexterm><primary>data files</primary><secondary>access to from NSE</secondary></indexterm>
	    <para>
            Allows access to Nmap's data files.  <literal>fetchfile()</literal>
            searches for the specified file and returns a string containing
            it's path if it is found and readable (to the process).  If the
            file is not found, not readable, or is a directory,
            <literal>nil</literal> is returned.  The call
<programlisting>
nmap.fetchfile("nmap-rpc")
</programlisting>
            will search for the data file <filename>nmap-rpc</filename> and,
            assuming it's found (which it should be), return a location like
            <filename>/usr/local/share/nmap/nmap-rpc</filename>.
            </para>
          </listitem>
        </varlistentry> 
        <varlistentry>
          <term><option>nmap.timing_level()</option>
            </term>
	  <listitem>
            <indexterm><primary>timing templates</primary><secondary>access to from NSE</secondary></indexterm>
	    <para>
            Returns the timing level as a non-negative integer.  Possible return
            values vary from 0 to 5, corresponding to the six built-in Nmap
            timing templates.  The timing level can be set with the
            <option>-T</option> option<bookex> (see <xref linkend="man-performance"/>)</bookex>.
            </para>
          </listitem>
        </varlistentry> 
      </variablelist>
      </para>
    </sect2>

    <sect2 id="nse-api-portmethods">
      <title>Target Information Retrieval by a Script</title>
      <para>
	Often the information passed to the script is not enough. Sometimes
	a script might want to correct target information or set it in the
	first place. The following API methods handle this.
      </para>

      <para>
	<variablelist>
	  <varlistentry>
	    <term><option>nmap.get_port_state(host, port, protocol)</option>
	      </term>
	    <listitem>
	      <para>
		The <literal>get_port_state()</literal> call takes a
		host table, a port table and a protocol
		(<literal>tcp</literal> or <literal>udp</literal>) and
		returns a port table for the queried port. The host
		and port table are similar in structure to the ones
		passed to the rule and action functions. The host
		table should have an IP address field. The port table
		needs a port number and a protocol field. A call could
		look like this:
<programlisting>
nmap.get_port_state({ip="127.0.0.1"}, {number="80", protocol="tcp"})
</programlisting>
		You can of course reuse the host and port tables
		passed to the port rule function. The purpose of this
		call is to be able to match scripts against more than
		one open port. For example if the target host has an
		open port 22 and a running identd server, then you can
		write a script which will only fire if both ports are
		open and there is an identification server on port
		113. While it is possible to specify IP addresses
		different to the currently scanned target, the result
		will only be correct if the target is in the currently
		scanned group of hosts.
		
	      </para>
	    </listitem>
	  </varlistentry>

	  <varlistentry>
	    <term><option>nmap.set_port_state(host, port, state)</option>
	      </term>
	    <listitem>
	      

<para>The <literal>set_port_state()</literal> call takes a host table,
a port table, and a port state (<literal>open</literal>
or <literal>closed</literal>). Using this method the final port state,
reflected in Nmap's results, can be changed for a target. This is
useful when Nmap detects a port as <literal>open|filtered</literal>
(i.e. unable to determine which), but the script successfully connects
to that port. In this case the script can set the port state
to <literal>open</literal>. Note that the port.state value, which was
passed to the script's action function will not be changed by this
call.</para>

	    </listitem>
	  </varlistentry>
	  
	  <varlistentry>
	    <term><option>nmap.set_port_version(host, port, probestate)</option>
	      </term>
	    <listitem>
	      <para>
		NSE scripts are sometimes able to determine the
		service name and application version listening on a
		port.  A whole script category
		(<literal>version</literal>) was designed for this
		purpose, as described in <xref linkend="nse-vscan"/>.
		The <literal>set_port_version</literal> function is
		used to record version information when it is
		discovered.</para>
		
		<para>This method takes a host and a port
		table as arguments. The third argument describes the
		state in which the script completed. It is a string
		which is one of: 
		<literal>hardmatched</literal>, 
		<literal>softmatched</literal>, 
		<literal>nomatch</literal>,
		<literal>tcpwrapped</literal>, or 
		<literal>incomplete</literal>.
		The <literal>hardmatched</literal> argument is almost
		always used, as it signifies a successful match.  The
		other possible states are generally only used for
		standard version detection rather than the NSE
		enhancement.</para>

		<para>The host and port arguments to this function
		should either be the tables passed to the
		<literal>action</literal> method or they should have
		the same structure.  The version detection fields this
		function looks at are <literal>name</literal>,
		<literal>product</literal>,
		<literal>version</literal>,
		<literal>extrainfo</literal>,
		<literal>hostname</literal>,
		<literal>ostype</literal>,
		<literal>devicetype</literal>, and
		<literal>service_tunnel</literal>.  All values in this
		table are optional. It is possible to pass a table in
		which all these values are set to
		<literal>nil</literal> or not to set the values at
		all. 
	      </para>
	    </listitem>
	  </varlistentry>
	</variablelist>
      </para>
    </sect2>
	<sect2 id="nse-aux-raw-packet">
      <title>Various Utility Functions for Raw Packet Support</title>
      <indexterm><primary>raw packets</primary><secondary>in NSE</secondary></indexterm>
      <para>
	  NSE has support for sending raw ethernet frames and capturing
	  packets. The following two functions may be handy in this context:
	  </para>
	<variablelist>
	  <varlistentry>
	    <term><option>nmap.clock_ms()</option>
	      </term>
	    <listitem>
	      <para>
	      Returns a number representing the current time as milliseconds 
		  since the start of the epoch (on most systems this is 01/01/1970).
		  </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>nmap.get_interface_link("interface_name")</option>
	      </term>
	    <listitem>
	      <para>
		  For the provided
		  dnet-style<indexterm><primary>libdnet</primary></indexterm>
		  <literal>interface_name</literal>,
		  <literal>nmap.get_interface_link()</literal> returns
		  what kind of link level hardware the interface
		  belongs. Return values are:
		  <literal>ethernet</literal>,
		  <literal>loopback</literal> or
		  <literal>p2p</literal>.  If the provided
		  <literal>interface_name</literal> is not one of
		  those types, <literal>nil</literal> is returned.
		  </para>
	    </listitem>
	  </varlistentry>
	</variablelist>
	</sect2>

    <sect2 id="nse-api-networkio">
      <title>Network I/O API</title>
      <para>
	To allow for efficient and parallelizable network I/O, NSE
	provides an interface to Nsock, the Nmap socket library. The
	smart callback mechanism Nsock uses is fully transparent to
	NSE scripts. The main benefit of NSE's sockets is that they
	never block on I/O operations, allowing many scripts to be run in parallel.
	The I/O parallelism is fully transparent to authors of NSE scripts.  
	In NSE you can either program as if you were using a single non
	blocking socket or you can program as if your connection is
	blocking. Seemingly blocking I/O calls still return once a
	specified timeout has been exceeded. Two flavors of Network I/O are
	supported: 
	</para>
	<sect3 id="nse-api-networkio-connect">
	  <title>Connect-style network I/O</title>
          <indexterm><primary>sockets in NSE</primary></indexterm>
	  <para>This part of the network API should be suitable for most
	  classical network uses: Users create a socket, connect it to a
	  remote address, send and receive data and close the socket again.
	  Everything up to the Transport layer (which is either TCP, UDP or
	  SSL) is handled by the library.  The following socket API methods 
	  are supported:
      </para>

      <para>
	<variablelist>
	  <varlistentry>
	    <term><option>nmap.new_socket()</option>
	      </term>
	    <listitem>
	      <para>
		The <literal>new_socket()</literal> Nmap call returns an
		NSE socket object which is the recommended method for network
		I/O. It provides facilities to perform communication using the
		UDP, TCP and SSL protocol in a uniform manner. 
	      </para>
	    </listitem>
	  </varlistentry>
	  
	  <varlistentry>
	    <term><option>status, error = socket_object:connect(hostid, port, [protocol])</option>
            </term>

	    <listitem>
	    
	      <para>
		The connect method of NSE socket objects will put
		the socket in a state ready for communication. It
		takes as arguments a host descriptor (either an IP
		address or a host name), a port number and optionally
		a protocol. The protocol must be one of
		<literal>tcp</literal>, <literal>udp</literal> or
		<literal>ssl</literal>. By default the connect call
		will attempt to open a TCP connection. On success the
		returned value of status is
		<literal>true</literal>. If the connection attempt has
		failed, the error value contains a description of the
		error condition stored as a string.
		Those strings are 
		taken from the <function> gai_strerror()</function> 
		C function. They are (with the errorcode in parentheses):</para>
		<itemizedlist>
			<listitem>
			<para><quote>Address family for hostname not supported</quote> (<literal>EAI_ADDRFAMILY</literal>)</para>
			</listitem>
			<listitem>
			<para><quote>Temporary failure in name resolution</quote> (<literal>EAI_AGAIN</literal>)</para>
			</listitem>
			<listitem>
			<para><quote>Bad value for ai_flags</quote> (<literal>EAI_BADFLAGS</literal>)</para>
			</listitem>
			<listitem>
			<para><quote>Non-recoverable failure in name resolution</quote> (<literal>EAI_FAIL</literal>)</para>
			</listitem>
			<listitem>
			<para><quote>ai_family not supported</quote> (<literal>EAI_FAMILY</literal>)</para>
			</listitem>
			<listitem>
			<para><quote>Memory allocation failure</quote> (<literal>EAI_MEMORY</literal>)</para>
			</listitem>
			<listitem>
			<para><quote>No address associated with hostname</quote> (<literal>EAI_NODATA</literal>)</para>
			</listitem>
			<listitem>
			<para><quote>Name or service not known</quote> (<literal>EAI_NONAME</literal>)</para>
			</listitem>
			<listitem>
			<para><quote>Servname not supported for ai_socktype</quote> (<literal>EAI_SERVICE</literal>)</para>
			</listitem>
			<listitem>
			<para><quote>ai_socktype not supported</quote> (<literal>EAI_SOCKTYPE</literal>)</para>
			</listitem>
			<listitem>
			<para><quote>System error</quote> (<literal>EAI_SYSTEM</literal>)</para>
			</listitem>
		</itemizedlist>
<para>In addition to these standard system error based messages are the following two NSE-specific errors:</para>
		<itemizedlist>
			<listitem>
                        <indexterm><primary>SSL</primary><secondary>in NSE</secondary></indexterm>
			<para><quote>Sorry, you don't have OpenSSL.</quote> occurs
			if <literal>ssl</literal> is passed as third argument, but Nmap was compiled 
			without OpenSSL support.</para>
			</listitem>
			<listitem>
			<para><quote>invalid connection method</quote> occurs if 
			the second parameter is not one of <literal>tcp</literal>, <literal>udp</literal>, <literal>ssl</literal>.</para>
			</listitem>
		</itemizedlist>
	    </listitem>
	  </varlistentry>
	  
	  <varlistentry>
	    <term><option>status, error = socket_object:send(data)</option>
	      </term>
	    <listitem>
	      <para>
		The send method sends the data contained in the
		<literal>data</literal> string through an open
		connection. On success the returned value of status is
		<literal>true</literal>.  If the send operation
		has failed, the error value contains a description of
		the error condition stored as a string. The error strings are:
		<itemizedlist>
		<listitem>
			<para><quote>Trying to send through a closed socket</quote>&mdash;if there was no 
			call to socket_object:connect before the send operation.</para>
		</listitem>
		<listitem>
			<para><quote>TIMEOUT</quote>&mdash;if the operation took longer than the 
			specified timeout for the socket.</para>
		</listitem>
		<listitem>
			<para><quote>ERROR</quote>&mdash;if an error occurred inside the underlying 
			Nsock library.</para>
		</listitem>
		<listitem>
			<para><quote>CANCELLED</quote>&mdash;if the operation was cancelled.</para>
		</listitem>
		<listitem>
			<para><quote>KILL</quote>&mdash;if for example the script scan is aborted due
			to a faulty script.</para>
		</listitem>
		<listitem>
			<para><quote>EOF</quote>&mdash;if an EOF was read&mdash;will probably not occur 
			for a send operation.</para>
		</listitem>

		</itemizedlist>
	      </para>
	    </listitem>
	  </varlistentry>
	  
	  <varlistentry>
	    <term><option>status, value = socket_object:receive()</option>
	      </term>
	    <listitem>
	      <para>
		The receive method does a non-blocking receive operation on
		an open socket. On success the returned value of
		<literal>status</literal> is
		<literal>true</literal> and the received data is stored in
		<literal>value</literal>. If receiving data has failed,
		<literal>value</literal> contains a description of the error
		condition stored as a string. A failure occurs for example
		if receive is called on a closed socket. The receive call
		returns to the NSE script all the data currently stored
		in the receive buffer of the socket. Error conditions
		are the same as for the send operation.
	      </para>
	    </listitem>
	  </varlistentry>
	  
	  <varlistentry>
	    <term><option>status, value = socket_object:receive_lines(n)</option>
	      </term>
	    <listitem>
	      <para>
		Tries to receive at least <replaceable>n</replaceable>
		lines from an open connection. A line is a string
		delimited with <literal><quote>\n</quote></literal> characters.  If
		it was not possible to receive at least
		<replaceable>n</replaceable> lines before the operation times
		out a TIMEOUT error occurs. On the other hand, if more
		than <replaceable>n</replaceable> lines were received, all are
		returned, not just <replaceable>n</replaceable>. Use
        <literal>stdnse.make_buffer</literal> to guarantee one line
        is returned per call. On success
		the returned value of <replaceable>status</replaceable> is
		<literal>true</literal> and the received data is
		stored in <replaceable>value</replaceable>. If the connection
		attempt has failed, <replaceable>value</replaceable> contains
		a description of the error condition stored as string.
		Error conditions are the same as for the send operation.

	      </para>
	    </listitem>
	  </varlistentry>
	  
	  <varlistentry>
	    <term><option>status, value = socket_object:receive_bytes(n)</option>
	      </term>
	    <listitem>
	      <para>
		Tries to receive at least <replaceable>n</replaceable>
		bytes from an open connection. On success the returned
		value of <replaceable>status</replaceable> is <literal>true</literal> and the
		received data is stored in
		<replaceable>value</replaceable>. If operation fails,
		<replaceable>value</replaceable> contains a description of the
		error condition stored as a string. Similarly to
		<literal>receive_lines()</literal>
		<replaceable>n</replaceable> is the minimum amount of
		characters we would like to receive. If more arrive,
		we get all of them. If fewer than <replaceable>n</replaceable> characters arrive
		before the operation times out, a TIMEOUT error occurs.
		Other error conditions are the same as for the send operation. 

	      </para>
	    </listitem>
	  </varlistentry>
	  
	  <varlistentry>
	    <term><option>status, value = socket_object:receive_buf(func/"string", keeppattern)</option>
	      </term>
	    <listitem>
	      <para>

	      The <literal>receive_buf</literal> method reads data
	      from the network until it encounters the given delimiter
	      string (or matches the function passed in). Only data
	      which occurs before the delimiter is returned, and the
	      delimiter is then erased.  This function continues to
	      read from the network until the delimiter is found or
	      the function times out.  If data is read beyond the
	      delimiter, that data is saved in a buffer for the next
	      call to <literal>receive_buf</literal>.  This buffer is
	      cleared on subsequent calls to other Network I/O API
	      functions.</para>

                <para>The <literal>receive_buf</literal> method takes
                  two arguments.  The first one is either a string or
                  a function.  Strings are passed to
                  Lua's <literal><ulink role="hidepdf"
                  url="http://www.lua.org/manual/5.1/manual.html#5.4">string.find</ulink></literal>
                  function as the second (pattern) parameter, with the
                  buffer data being searched. If the
                  first <literal>receive_buf</literal> argument is a
                  function, it is expected to take exactly one
                  parameter (the buffer) and its return values must be
                  in the same format as <literal>string.find</literal>
                  (offsets to the start and the end of the delimiter
                  inside the buffer, or <literal>nil</literal> if the
                  delimiter is not found).  The nselib
		  <literal>match.lua</literal> module (see
		  <xref linkend="nse-lib-match"/>) provides functions
		  for matching against regular expressions or byte
		  counts.  These functions are suitable as arguments
		  to <literal>receive_buf</literal>.</para>

		  <para>The second argument
		  to <literal>receive_buf</literal> is a Boolean value
		  which indicates whether the delimiting pattern
		  should be returned along with the received data or
		  discarded.  The delimiter is included if <literal>true</literal> is passed as the <literal>keeppattern</literal> argument.</para>

                  <para>The return values of <literal>receive_buf</literal> are the same as the other <literal>receive*</literal> functions.  A <literal>(status, val)</literal> tuple is returned.  The <literal>status</literal> is <literal>true</literal> if the request was successful.  The <literal>val</literal> variable contains the returned data, or an error message if the call failed.</para>

                  <para>Possible error messages include those described previously for the other
		  <literal>receive*</literal> functions as well as the
		  following:

		<itemizedlist>
		<listitem>
			<para><quote>Error inside splitting-function</quote>&mdash;if the first argument was
			a function which caused an error while being called.
			</para>
		</listitem>
		<listitem>
			<para><quote>Error in <literal>string.find</literal> (<literal>nsockobj:receive_buf</literal>)!</quote>&mdash;if a string
			was provided as the first argument, and string.find() yielded an
			error while being called.</para>
		</listitem>
		<listitem>
			<para><quote>Expected either a function or a string!</quote>&mdash;if the 
			first argument was neither a function nor a string.</para>
		</listitem>
		<listitem>
			<para><quote>Delimiter has negative size!</quote>&mdash;if the returned start offset
			is greater than the end offset.</para>
		</listitem>
		</itemizedlist>
		  </para>

		</listitem>
	  </varlistentry>
	  
	  <varlistentry>
	    <term><option>status, err = socket_object:close()</option>
	      </term>
	    <listitem>
	      <para>
		Closes an open connection. On success the returned value of
		<literal>status</literal> is <literal>true</literal>. If the connection 
		attempt has failed, <literal>value</literal> contains a description 
		of the error condition stored as a string. Currently the only error 
		message is: <quote>Trying to close a closed socket</quote>, which is issued if the socket
		has already been closed. Sockets are subject to garbage collection. 
		Should you forget to close a socket, it will get closed before it gets
		deleted (on the next occasion Lua's garbage collector is run).
		However since garbage collection cycles are difficult to predict, it 
		is considered good practice to close opened sockets.		
	      </para>
	    </listitem>
	  </varlistentry>

	  <varlistentry>
	    <term><option>status,localip,localport,remoteip,remoteport=socket_object:get_info()</option>
	      </term>
	    <listitem>
	      <para>
		This function returns information about the socket
		object. It returns 5 values. If an error occurred, the
		first value is <literal>nil</literal> and the second
		value describes the error condition. Otherwise the
		first value describes the success of the operation and
		the remaining 4 values describe both endpoints of the
		TCP connection. If you put the call in a <literal>try()</literal> statement
		the status value is consumed. The call can be used for example if
		you want to query an authentication server.
	      </para>
	    </listitem>
	  </varlistentry>

	  
	  <varlistentry>
	    <term><option>socket_object:set_timeout(t)</option>
	      </term>
	    <listitem>
	      <para>
		Sets the time, in milliseconds, after which input and
		output operations on a socket should time out and
		return. The default value is 30,000 (30 seconds). The lowest
		allowed value is 10&nbsp;ms, since this is 
		the granularity of NSE network I/O.
	      </para>
	    </listitem>
	  </varlistentry>

	</variablelist>
      </para>
    </sect3>
	<sect3 id="nse-api-networkio-raw">
	  <title>Raw packet network I/O</title>
          <indexterm><primary>raw packets</primary><secondary>in NSE</secondary></indexterm>
	  <para>For those cases where the connection oriented approach is too inflexible,
	  NSE provides script developers with a more powerful option:
	  raw packet network I/O. The greater flexibility comes, however, at
	  the cost of a slightly more complex API. Receiving raw packets is
	  accomplished via a wrapper around
	  Libpcap<indexterm><primary>libpcap</primary></indexterm>
          inside the Nsock library.<indexterm><primary>Nsock</primary></indexterm>
          In order to keep the
	  capturing efficient it works in a three tiered approach: Opening a
	  device for capturing, registering listeners to it and receiving
	  packets. With each call to <literal>pcap_open()</literal> you have
	  to provide a callback function, which receives the packet (along with
	  it's layer 2 and 3 headers) and is used to compute a so-called
	  packet hash. Each call to <literal>pcap_register()</literal> takes a
	  binary string as argument. For every packet captured the computed
	  hash is matched against all registered strings. 
	  Those scripts for which the compare yields true are then provided
	  with the packet as a return value to <literal>pcap_receive()</literal>. 
	  The more general the packet hash computing function is kept, 
	  the more scripts may receive the packet and proceed with their
	  execution.  To use the packet capturing inside your script you have to 
	  create a socket with
	  <literal>nmap.newsocket()</literal> and later close the socket with <literal>socket_object:close()</literal>&mdash;just
	  like with the connection-based network I/O. A more detailed description 
	  of the functions for packet capturing follows:
      </para>

      <para>
	<variablelist>
	  <varlistentry>
	    <term><option>socket_object:pcap_open(device, snaplen, promisc,
		test_function, bpf)</option>
	      </term>
	    <listitem>
	      <para>
		The <literal>pcap_open()</literal> call opens the socket for
		packet capturing. The parameters are:</para>
<itemizedlist>
<listitem><para><literal>device</literal>&mdash;the dnet-style interface name of the device you want to capture from</para></listitem>
<listitem><para><literal>snaplen</literal>&mdash;defines the length of each packet you want to capture (similar to the <option>-s</option> option to <command>tcpdump</command>)</para></listitem>
<listitem><para><literal>promisc</literal>&mdash;should be set to <literal>1</literal> if the interface should activate promiscuous mode, and zero otherwise</para></listitem>
<listitem><para><literal>test_function</literal>&mdash;callback function used to compute the <literal>packet hash</literal></para></listitem>
<listitem><para><literal>bpf</literal>&mdash;a string describing a Berkeley packet filter expression (like those provided to <command>tcpdump</command>)</para></listitem>
</itemizedlist>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>socket_object:pcap_register(packet-hash)</option>
	      </term>
	    <listitem>
	      <para>
		  Starts the listening for incoming packages. The provided
		  <literal>packet-hash</literal> is a binary string which has to
		  match the hash returned by the
		  <literal>test_function</literal> parameter provided to
		  <literal>pcap_open()</literal>. If you want to receive all
		  packets, just provide the empty string (<literal>""</literal>).
		  There has to be a call to <literal>pcap_register()</literal>
		  before a call to <literal>pcap_receive()</literal>.
	      </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>status, packet_len, l2_data, l3_data = socket_object:pcap_receive()</option>
	      </term>
	    <listitem>
	      <para>
		  Receives a captured packet. If successful, the return values are:</para>
<itemizedlist>
<listitem><para><literal>status</literal>&mdash;a Boolean with the value <literal>true</literal></para></listitem>
<listitem><para><literal>packet_len</literal>&mdash;the length of the captured packet which was received (this may be smaller than the actual packet length since packets are truncated when the Libpcap snaplen parameter is smaller than the total packet length)</para></listitem>
<listitem><para><literal>l2_data</literal>&mdash;data from the second OSI layer (e.g. ethernet headers)</para></listitem>
<listitem><para><literal>l3_data</literal>&mdash;data from the third OSI layer (e.g. IPv4 headers)</para></listitem>
</itemizedlist>

<para>Should an error or timeout occur, while waiting for a packet the
return values are: <literal>nil,error_message,nil,nil</literal>, where
error_message describes the occurred error.</para>

	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>socket_object:pcap_close()</option>
	      </term>
	    <listitem>
	      <para>Closes the pcap device.
	      </para>
	    </listitem>
	  </varlistentry>
	</variablelist>  
	</para>
	<para>
	Receiving raw packets is a great feature, but it is also only the
	half job. Now for sending raw packets: To accomplish this NSE has
	access to a wrapper around the
	<literal>dnet</literal> library.<indexterm><primary>libdnet</primary></indexterm>
	Currently NSE has the ability to send raw ethernet frames via the
	following API:
	</para>
	<para>
	<variablelist>
	  <varlistentry>
	    <term><option>dnet_object=nmap.new_dnet()</option>
	      </term>
	    <listitem>
	      <para>
	      Creates and returns a new dnet_object, which can be used to
		  send raw packets.
		  </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>dnet_object:ethernet_open(interface_name)</option>
	      </term>
	    <listitem>
	      <para>Opens the interface defined by the provided 
		  <replaceable>interface_name</replaceable> for sending ethernet frames
		  through it. An error (<quote>device is not valid ethernet
		  interface</quote>) is thrown in case the provided argument
		  is not valid.
		  </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>dnet_object:ethernet_send(packet)</option>
	      </term>
	    <listitem>
	      <para>
		  Sends the provided data as ethernet frame across the previously
		  opened interface. Note that you have to provide the packet
		  including IP header and ethernet header. If there was no
		  previous valid call to <literal>ethernet_open()</literal> an
		  error is thrown (<quote>dnet is not valid opened ethernet
		  interface</quote>).
		  </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>dnet_object:ethernet_close()</option>
	      </term>
	    <listitem>
	      <para>Closes the interface. The only error which may be thrown
		  is the same as for the <literal>ethernet_send()</literal>
		  operation.
		  </para>
	    </listitem>
	  </varlistentry>
	</variablelist>
	</para>
	</sect3>
	</sect2>

    <sect2 id="nse-mutex">
      <title>Thread Mutexes</title>
      <indexterm><primary>threads in NSE</primary></indexterm>
      <indexterm><primary>mutexes in NSE</primary></indexterm>
      <para>
        Each thread made for a script (e.g. anonFTP.nse) will yield to other
        scripts whenever it makes a call on network objects (sending/receiving
        data). Some scripts need finer control over threads' execution. An
        example is the <literal>whois.nse</literal> script which queries
	whois<indexterm><primary>whois</primary></indexterm>
        servers for each target. Because many concurrent queries often result in
        getting one's IP banned for abuse and a query may return additional
        information for targets other threads are running against, it is useful
        to have other threads pause while one thread is conducting a query.
      </para>
      <para>
        To solve this problem, there is an nmap function,
        <literal>mutex</literal>, that provides a
        <ulink url="http://en.wikipedia.org/wiki/Mutual_exclusion">mutex</ulink>
        usable by scripts. The mutex allows for only one thread to be working
        on an object. Threads waiting to work on this object are put in the
        waiting queue until they can get a "lock" on the mutex. A solution for
        the <literal>whois.nse</literal> problem above is to have each thread
        block on a mutex for <xref linkend="nse-format-id">script's ID field
        </xref>, thus ensuring only one thread is working so its results can
        be shared with other scripts which may not need to run and so queries
        to the whois servers are staggered.
      </para>
      <variablelist>
        <varlistentry>
          <term>
            <option>mutex = nmap.mutex(object)</option>
            
          </term>
          <listitem>
            <para>
              Returns a function that works on a mutex for the object passed.
              This object can be any
              <ulink role="hidepdf" url="http://www.lua.org/manual/5.1/manual.html#2.2">
                Lua data type
              </ulink> except <literal>nil</literal>,
              <literal>booleans</literal>, and <literal>numbers</literal>.
              The returned function allows you to lock, try to lock, and
              release the mutex. Its first and only parameter must be one of the following:
            </para>
            <itemizedlist>
              <listitem>
                <para>
                  <literal>"lock"</literal>&mdash;Make a blocking lock on the
                  mutex. If the mutex is busy
                  (another thread has a lock on it), then the thread will
                  yield and wait. The function returns with the mutex
                  locked.
                </para>
              </listitem>
              <listitem>
                <para>
                  <literal>"trylock"</literal>&mdash;Makes a non-blocking lock 
                  on the mutex. If the mutex is
                  busy then it immediately returns with a return value of
                  <literal>false</literal>. Otherwise the mutex locks the
                  mutex and returns <literal>true</literal>.
                </para>
              </listitem>
              <listitem>
                <para>
                  <literal>"done"</literal>&mdash;Releases the mutex and allows
                  another thread to lock it.
                  If the thread does not have a lock on the mutex, an
                  error will be raised.
                </para>
              </listitem>
              <listitem>
                <para>
                  <literal>"running"</literal>&mdash;Returns the thread locked
                  on the mutex or <literal>nil</literal> if the mutex is not
                  locked. This should only be used for debugging as it
                  interferes with finished threads from being
                  collected.
                </para>
              </listitem>
            </itemizedlist>
          </listitem>
        </varlistentry>
      </variablelist>
      <example id="nse-mutex-handling">
        <title>Mutex manipulation</title>
        <programlisting>
id = "My Script's Unique ID";

local mutex = nmap.mutex(id);
function action(host, port)
  mutex "lock";
  -- do stuff
  mutex "done";
  return script_output;
end
        </programlisting>
      </example>
    </sect2>

    <sect2 id="nse-exceptions">
      <title>Exception Handling</title>
      <indexterm><primary>exceptions in NSE</primary></indexterm>
      <para>
	  NSE provides an exception handling mechanism not present in
	  the plain Lua language. The exception handling is tailored
	  specifically for network I/O operations. The mechanism
	  follows a functional programming paradigm rather than an
	  object oriented programming paradigm. To create an exception
	  handler the <literal>nmap.new_try()</literal> API method is
	  used. This method returns a function, which takes a function
	  as an argument. If the function passed as an argument raises
	  an exception, then the script execution is aborted and no
	  output is produced. Optionally you can pass a function to
	  the <literal>new_try()</literal> method which will be called
	  if an exception is caught. In this function you can perform
	  required clean up operations.</para>

      <para>
	  <xref linkend="nse-exception-handling" xrefstyle="select: label nopage"/> shows cleanup
	  exception handling at work.  A new function named
	  <literal>catch</literal> is defined to simply close the
	  newly created socket in case of an error.  It is then used
	  to protect connection and communication attempts on that
	  socket. If no catch function is specified, execution of the
	  script aborts without further ado&mdash;open sockets
	  will remain open. If the verbosity level is at least one
	  or if the scan is performed in debugging mode a description
	  of the uncaught error condition is printed on standard output.
	  Note that it is currently not easily possible to group several
	  statements in one try block. It is also important to remember
	  that if the socket is not closed it will occupy memory
	  until the next run of Lua's garbage collector.

	</para>

        <example id="nse-exception-handling">
          <title>Exception handling example</title>
<programlisting>
local result, socket, try, catch

result = ""
socket = nmap.new_socket()
catch = function() 
socket:close() 
end
try = nmap.new_try(catch)

try(socket:connect(host.ip, port.number))
result = try(socket:receive_lines(1));
try(socket:send(result))
</programlisting>
       </example>
	
	  <para>
	  Writing a function which is treated properly by the
	  try/catch mechanism is straightforward. The function should
	  return multiple values.  The first value should be a Boolean
	  which is <literal>true</literal> upon successful completion of the function and
	  <literal>false</literal> otherwise. If the function completed successfully the try
	  construct consumes the indicator value and returns the
	  remaining values.  If the function failed then the second
	  returned value must be a string describing the error
	  condition. Note that if the value is not <literal>nil</literal> it is
	  treated as <literal>true</literal> so you can return your
	  value in the normal case and return <literal>nil, <replaceable>error description</replaceable></literal>
	  if an error occurs.
	</para>

    </sect2>

    <sect2 id="nse-api-registry">
      <title>The Registry</title>
      <indexterm><primary>registry (NSE)</primary></indexterm>
      <para>
	The registry is a normal Lua table. What is special about it
	is that it is visible by all scripts and it retains its state
	between script executions. Nmap does not scan every host
	specified on the command line at the same time, it puts them
	in smaller groups and these groups are scanned in parallel.  The
	registry is rebuilt for every group, so information stored
	there is only deleted after NSE finishes processing the
	current target group. This implies of course that the registry
	is transient&mdash;it is not stored between Nmap executions. Every
	script can read the registry and write to it. If a script is
	running after another script, it can read some information in
	the registry which was left by the first script. This feature
	is particularly powerful in combination with the run level
	concept. A script with a higher run level can rely on entries
	left behind for it by scripts with lower run levels.
      <indexterm><primary>run level of scripts</primary></indexterm>
        Remember
	however that the registry can be written by all scripts
	equally, so choose the keys for your entries wisely. The
	registry is stored in <literal>nmap.registry</literal>. The
	behavior of the registry allows caching of already calculated
	data. The cache can be seen by all scripts until the registry
	is rebuilt with the next target group. <!-- If for example you have
	compiled a regular expression, you can store the compiled
	expression in the registry so that scripts which need the same
	pattern do not have to recompile it. -->

      </para>
    </sect2>
    <indexterm class="endofrange" startref="nse-nmap-indexterm"/>
  </sect1>

  <sect1 id="nse-tutorial">
    <title>Script Writing Tutorial</title>
    <indexterm class="startofrange" id="nse-tutorial-indexterm"><primary>Nmap Scripting Engine (NSE)</primary><secondary>tutorial</secondary></indexterm>

    <para>
      Suppose that you are convinced of the power of NSE. How do you
      go about writing your own script?  Let's say
      that you want to extract information from an identification
      server.<indexterm><primary>auth service</primary></indexterm>
      Nmap used to have this functionality but it was removed
      because of inconsistencies in the code base. Fortunately, the
      protocol identd uses is pretty simple. Unfortunately, it is too
      complicated to be expressible in Nmap's version detection
      language. Let's look at how the identification protocol
      works. First you connect to the identification server. Next you
      send a query of the form <literal><replaceable>port-on-server</replaceable>,
      <replaceable>port-on-client</replaceable></literal> terminated with a new line
      character. The server should then respond with a string of the
      form <literal><replaceable>port-on-server</replaceable>, <replaceable>port-on-client</replaceable>:<replaceable>response-type</replaceable>:<replaceable>address-information</replaceable></literal>.  In case of an error the address
      information is omitted. This description is sufficient for our
      purposes, for more details refer to <ulink role="hidepdf" url="http://www.rfc-editor.org/rfc/rfc1413.txt">RFC 1413</ulink>.  The protocol cannot be modeled in Nmap's version
      detection language for two reasons. The first is that you need
      to know both the local and the remote port of a
      connection. Version detection does not provide this data.  The
      second, more severe obstacle, is that you need two open
      connections to the target&mdash;one to the identification server and
      one to the port you want to query. Both obstacles are easily
      overcome with NSE.  </para>

    <para>
      The anatomy of a script is described in <xref linkend="nse-scripts"/>.
      In this section we will show how the described structure is utilized.
    </para>
    
    <sect2 id="nse-tutorial-head">
      <title>The Head</title>
      <para>

      The head of the script is essentially its meta information. This
      includes the
      fields: <literal>id</literal>, <literal>description</literal>, <literal>categories</literal>, <literal>runlevel</literal>, <literal>author</literal>
      and <literal>license</literal>. We are not going to change the
      run level, or worry about the author and license fields for now.
      The <literal>id</literal> of a script should uniquely identify
      it. If it is absent, the path to the script will be used as an
      id. We recommend to choose an id which concisely identifies the
      purpose of the script, since the ID is printed before the
      script's results in Nmap output.
      </para>

      <para>
<indexterm><primary sortas="Service Owner script">&ldquo;<literal>Service Owner</literal>&rdquo; script</primary></indexterm>
<indexterm><primary sortas="id script variable">&ldquo;<varname>id</varname>&rdquo; script variable</primary></indexterm>
<programlisting>
id = "Service Owner"
</programlisting>
      </para>
      <para>
	The description field should contain a sentence or two describing what the script does.  If anything about the script results might confuse or mislead users, and you can't eliminate the issue by improving the script or results text, it should be documented in the <literal>description</literal> string.
      </para>
      <para>
<indexterm><primary sortas="description script variable">&ldquo;<varname>description</varname>&rdquo; script variable</primary></indexterm>
<programlisting>
description = "Opens a connection to the scanned port, opens a connection to \
port 113, queries the owner of the service on the scanned port and prints it."
</programlisting>
      </para>
      <para>

 	Users must tell the Lua interpreter that the string
	continues on the following line by ending the line with a
	backslash (&lsquo;<literal>\</literal>&rsquo;). They must also decide what
	categories the script belongs to. This script is a good
	example of a script which cannot be categorized clearly. It is
	<literal>safe</literal><indexterm><primary><literal>safe</literal> script category</primary></indexterm>
        because we are not using the service
	for anything it was not intended for.  On the other hand, it
	is <literal>intrusive</literal><indexterm><primary><literal>intrusive</literal> script category</primary></indexterm>
        because we connect to a
	service on the target and therefore potentially give out
	information about us.  To solve this dilemma we will place our
	script in two categories:

      </para>
<indexterm><primary sortas="categories script variable">&ldquo;<varname>categories</varname>&rdquo; script variable</primary></indexterm>
<programlisting>
categories = {"safe", "intrusive"}
</programlisting>
    </sect2>

    <sect2 id="nse-tutorial-rule">
      <title>The Rule</title>
      <para>
	The rule section is a Lua method which decides when the
	script's action should be performed and when it should be
	skipped. Usually this decision is based on the host and port
	information passed to the rule function. In the case of the
	identification script it is slightly more complicated than
	that. To decide whether to run the identification script on a
	given port we need to know if there is an identification
	server running on the target machine. Or more formally: the
	script should be run if (and only if) the currently scanned TCP port is open and
	TCP port 113 is also open. For now we will rely on the fact that
	identification servers listen on TCP port 113. Unfortunately NSE
	only gives us information about the currently scanned port.

	To find out if port 113 is open we are going to use the
	<literal>nmap.get_port_state()</literal> method. If the identd
	port was not scanned, the <literal>get_port_state</literal>
	function returns <literal>nil</literal>. So we need to make
	sure that the table is not <literal>nil</literal>. We also
	check if both ports are in the <literal>open</literal> state.
	If this is the case, the action is executed, otherwise we skip
	the action.
      </para>
      <para>
<indexterm><primary sortas="portrule script variable">&ldquo;<varname>portrule</varname>&rdquo; script variable</primary></indexterm>
<programlisting>
portrule = function(host, port)
  local ident_port = { number=113, protocol="tcp" }
  local identd = nmap.get_port_state(host, ident_port)

  if identd ~= nil and identd.state == "open" and port.state == "open" then
    return true
  else
    return false
  end
end
</programlisting>
      </para>

      <para>
	This rule is <emphasis>almost</emphasis> correct, but still
	slightly buggy. Can you find the bug?  It is a pretty subtle
	one. The problem is that this script fires on any kind of open
	port, TCP or UDP. The <literal>connect()</literal> method on
	the other hand assumes a TCP protocol unless it is explicitly
	told to use another protocol. Since the identification service
	is only defined for TCP connections, we need to narrow down
	the range of ports which fire our script. Our new rule only
	runs the script if the port is open, we are looking at a TCP
	port, and TCP port 113 is open. Writing the new and
	improved port rule is left as an exercise to the reader (or
	peek at the script in the latest Nmap distribution).
      </para>


    </sect2>

    <sect2 id="nse-tutorial-action">
      <title>The Mechanism</title>
      <para>
	At last we implement the actual functionality. The script will
	first connect to the port on which we expect to find the
	identification server, then it will connect to the port we
	want information about. Afterward we construct a query string
	and parse the response. If we received a satisfactory
	response, we return the retrieved information.
      </para>
      <para>
	First we need to create two socket objects. These objects
	represent the sockets we are going to use. By using object methods
	like 
	<literal>open()</literal>, 
	<literal>close()</literal>, 
	<literal>send()</literal> or
	<literal>receive()</literal> we can operate on the network
	socket. To avoid excessive error checking code we use NSE's
	exception handling mechanism.<indexterm><primary>exceptions in NSE</primary></indexterm>
        We create a function which will
	be executed if an error occurs and call this function
	<literal>catch</literal>. Using this function we generate
	a <literal>try</literal> function. The <literal>try</literal>
	function will call the <literal>catch</literal> function
	whenever there is an error condition in the tried block.
      </para>
      
      <para>
        <indexterm><primary sortas="action script variable">&ldquo;<varname>action</varname>&rdquo; script variable</primary></indexterm>
	<programlisting>
action = function(host, port)
local owner = ""

local client_ident = nmap.new_socket()
local client_service = nmap.new_socket()

local catch = function()
  client_ident:close()
  client_service:close()
end

local try = nmap.new_try(catch)

try(client_ident:connect(host.ip, 113))
try(client_service:connect(host.ip, port.number))

local localip, localport, remoteip, 
remoteport = client_service:get_info()

local request = port.number .. ", " .. localport .. "\n"

try(client_ident:send(request))

owner = try(client_ident:receive_lines(1))

if string.match(owner, "ERROR") then
  owner = nil
  --  owner = "Service owner could not be determined: " .. owner
else
 owner = string.match(owner, "USERID : .+ : (.+)\n", 1)
end

try(client_ident:close())
try(client_service:close())

return owner
end
</programlisting>
      </para>

<para>Note that because we know that the remote port is stored
in <literal>port.number</literal>, we could have ignored the last two
return values of <literal>client_service:get_info()</literal> like
this:</para>

<programlisting>
local localip, localport  = client_service:get_info()
</programlisting>

<para>In this example we avoided telling the user if the service responded with an error.  Instead we commented that line out and assigned <literal>nil</literal> to the owner variable.  NSE scripts generally only return messages when they succeed.</para>

    </sect2>
    <indexterm class="endofrange" startref="nse-tutorial-indexterm"/>
  </sect1>

  <sect1 id="nse-documentation">
    <title>Script Documentation Writing</title>
    <indexterm class="startofrange" id="nse-documentation-indexterm"><primary>Nmap Scripting Engine (NSE)</primary><secondary>Documentation Writing</secondary></indexterm>

    <para>
      Scripts are used by more than just the author, so it is useful to have
      documentation for users and other developers to browse through
      when looking
      for scripts to run against a host. For this reason,
      NSE provides a system
      that will produce complete documentation for its library
      of scripts.
    </para>
    <para>
      The documentation
      is put in comments around your code. The trigger for a
      documentation comment is <literal>---</literal>. Only the first
      line in the block of comments should use the prefix trigger;
      subsequent lines should use Lua's standard single line comment
      prefix: <literal>--</literal>.
    </para>
    <para>
      The first paragraphs are used as a description for the code
      following the comments. The first sentence of this description
      is used as a summary (for a brief description in the index).
      Optional tags can follow the description using the format
      <literal>@tag ...</literal>. See
      <xref linkend="nse-documentation-tags"/> for more information.
    </para>
    <para>
      The first comment block not followed by a function or table
      definition will
      be used for a file's comment. This comment will serve for documenting
      general information concerning the script. The description of this
      tag should explain the script's purpose and what it does.
      The file comment should also
      contain an output tag showing expected output for the script and
      an args tag for any arguments the script expects.
    </para>

    <sect2 id="nse-documentation-format">
      <title>Documentation Format</title>
      <para>
      Documentation for a script consists of three basic parts: a file
      comment, a function or table comment, and standard NSE script fields
      (e.g. <literal>description</literal> or <literal>author</literal>).
      A file comment is the first documentation comment in a script and
      is not followed by a function definition or table. The following
      example demonstrates a file comment:
<programlisting>
--- Checks if an FTP server allows anonymous logins.
-- @output
-- |_ Anonymous FTP: Anonymous login allowed"
</programlisting>
      The standard NSE fields are used to gather other information about a
      script such as the author or its categories:
<programlisting>
author = "Eddie Bell"
license = "Same as Nmap--See http://nmap.org/book/man-legal.html"
id = "Anonymous FTP"
categories = {"default", "intrusive"}
</programlisting>
      These fields will be present in the final documentation. The
      description field is used for the User Summary if a description in
      the file comment or the file comment itself is absent. It is best
      if the description field is succinct while the file comment has an
      extensive description of the script.
      </para>
      <para>
      Finally, a documentation comment may precede a function or
      table definition:
<programlisting>
--- Convert two bytes into a 16bit number.
--@param data String of data.
--@param idx Index in the string (first of two consecutive bytes).
--@return 16 bit number represented by the two bytes.
function bto16(data, idx)
  local b1 = string.byte(data, idx)
  local b2 = string.byte(data, idx+1)
  -- (b2 &amp; 0xff) | ((b1 &amp; 0xff) &lt;&lt; 8)
  return bit.bor(bit.band(b2, 255), bit.lshift(bit.band(b1, 255), 8))
end
</programlisting>
      You may also choose to document the action, portrule, or hostrule
      functions. They are separated from other functions in the final
      documentation for improved visibility.
      </para>
    </sect2>
    <sect2 id="nse-documentation-tags">
      <title>Tags</title>
      <para>
        The following tags can be used:
      </para>
      <variablelist>
        <varlistentry>
          <term><option>@param</option></term>
          <listitem>
            <para>
              Describe function parameters. The first alphanumeric word
              must be the parameter being documented.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@see</option></term>
          <listitem>
            <para>
              Refer to other functions or tables.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@return</option></term>
          <listitem>
            <para>
              Describe the return values of a function.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@usage</option></term>
          <listitem>
            <para>
              Describe the usage of a function or variable.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@description</option></term>
          <listitem>
            <para>
              The description of a function or table; this is normally
              inferred from the beginning of the documentation comment.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@name</option></term>
          <listitem>
            <para>
              The name of the function or table definition. This should
              be used only if the documentation system cannot infer the
              name by code analysis.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@class</option></term>
          <listitem>
            <para>
              Like <literal>@name</literal>, if the name of the variable
              is not being correctly inferred through code analysis, you
              may specify the class (function or table) explicitly.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@field</option></term>
          <listitem>
            <para>
              Describe a table field definition.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@release</option></term>
          <listitem>
            <para>
              Free format string to describe the module or file release.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@args</option></term>
          <listitem>
            <para>
              This tag is special to the file comment. It allows you specify
              arguments to the script (via --script-args) that your script
              uses. The first alphanumeric word (literally matching in Lua
              <literal>([%w%p]+)</literal>) is used as the name for
              the argument, the rest its description.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@summary</option></term>
          <listitem>
            <para>
              This allows you to specify the summary explicitly different
              from the first sentence of the description.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@output</option></term>
          <listitem>
            <para>
              This tag is special to thie file comment. It allows you to
              show typical output of a script. You may use "\n" to
              force a line break where you please.
            </para>
          </listitem>
        </varlistentry>
      </variablelist>
    </sect2>
  </sect1>

  <sect1 id="nse-vscan">
    <title>Version Detection Using NSE</title>
    <indexterm class="startofrange" id="nse-sample-indexterm"><primary>Nmap Scripting Engine (NSE)</primary><secondary>sample scripts</secondary></indexterm>
    <indexterm><primary>version detection</primary><secondary>using NSE</secondary></indexterm>
    <para>

      The version detection system built into Nmap was designed to
      efficiently recognize the vast majority of protocols with a
      simple pattern matching syntax.  Some protocols require a more
      complex approach, and a generalized scripting language is
      perfect for this.  Skype&nbsp;v2 is one such protocol.  It pretends to
      be an HTTP server, requiring multiple queries to determine its
      true nature.  NSE has been integrated into Nmap's version
      detection framework to handle these cases. The scripts which
      extend the version scanner belong to the reserved category
      <literal>version</literal>.<indexterm><primary><varname>version</varname> script category</primary></indexterm>
      This category cannot be run from
      the command line.  It is only executed if the user has required a
      version scan. The following listing shows a simple script which
      demonstrates the use of the NSE version detection API. If either
      the TCP port 80 is open or the service has been determined to be
      HTTP, the script is triggered. Although it could be extended to
      recognize different HTTP servers, its only purpose is to show off
      the version detection API. It is not advisable to use NSE for
      version detection in the simple case of HTTP servers. The
      version detection variables have been filled with dummy entries
      to illustrate their effect on the Nmap output.</para> 


      <para>
<programlisting> 
description = "Demonstration of a version detection NSE script. It checks \
and reports the version of a remote web server. For real life purposes it is \
better to use Nmap version detection (-sV)."<indexterm><primary sortas="description script variable">&ldquo;<varname>description</varname>&rdquo; script variable</primary></indexterm>
author = "Diman Todorov &lt;diman.todorov@gmail.at&gt;"<indexterm><primary>Todorov, Diman</primary></indexterm><indexterm><primary sortas="author script variable">&ldquo;<varname>author</varname>&rdquo; script variable</primary></indexterm>
license = "Same as Nmap--See http://nmap.org/book/man-legal.html"<indexterm><primary sortas="license script variable">&ldquo;<varname>license</varname>&rdquo; script variable</primary></indexterm>

id = "HTTP version"<indexterm><primary sortas="id script variable">&ldquo;<varname>id</varname>&rdquo; script variable</primary></indexterm>

categories = {"version"}<indexterm><primary sortas="categories script variable">&ldquo;<varname>categories</varname>&rdquo; script variable</primary></indexterm>

runlevel = 1.0<indexterm><primary sortas="runlevel script variable">&ldquo;<varname>runlevel</varname>&rdquo; script variable</primary></indexterm>

portrule = function(host, port)<indexterm><primary sortas="portrule script variable">&ldquo;<varname>portrule</varname>&rdquo; script variable</primary></indexterm>
  if (port.number == 80
      or port.service == "http" )
      and port.protocol == "tcp" 
  then
    return true
  else
    return false
  end
end

action = function(host, port)<indexterm><primary sortas="action script variable">&ldquo;<varname>action</varname>&rdquo; script variable</primary></indexterm>
  local query = "GET / HTTP/2.1\r\n"
  query = query .. "Accept: */*\r\n"
  query = query .. "Accept-Language: en\r\n"
  query = query .. "User-Agent: Nmap NSE\r\n"
  query = query .. "Host: " .. host.ip .. ":" .. port.number .. "\r\n\r\n"

  local socket = nmap.new_socket()
  local catch = function()
    socket:close()
  end

  local try = nmap.new_try(catch)

  try(socket:connect(host.ip, port.number))
  try(socket:send(query))

  local response = ""
  local lines
  local status
  local value

  while true do
    status, lines = socket:receive_lines(1)

    if not status or value then
      break
    end

    response = response .. lines
    value = string.match(response, "Server: (.-)\n")
  end

  try(socket:close())

  if value then
    port.version.name = "[Name]"
    port.version.name_confidence = 10
    port.version.product = "[Product]"
    port.version.version = "[Version]"
    port.version.extrainfo = "[ExtraInfo]"
    port.version.hostname = "[HostName]"
    port.version.ostype = "[OSType]"
    port.version.devicetype = "[DeviceType]"

    port.version.service_tunnel = "none"
    port.version.fingerprint = nil
    nmap.set_port_version(host, port, "hardmatched")
  end
end
      </programlisting>
    </para>
    <para>
	This is what the output of this script looks like:
      <screen>
$ nmap -sV localhost -p 80

Starting Nmap ( http://nmap.org )
Interesting ports on localhost (127.0.0.1):
PORT   STATE SERVICE VERSION
80/tcp open  [Name]  [Product] [Version] ([ExtraInfo])
Service Info: Host: [HostName]; OS: [OSType]; Device: [DeviceType]

Nmap finished: 1 IP address (1 host up) scanned in 9.317 seconds
</screen>
    </para>
    <para>
      The name variable denotes the detected protocol name.
      The product, version and extrainfo variables are used
      to produce a human readable description of the server
      version. The remaining variables provide information deduced
      from the output of the server concerning the target host.
    </para>
  </sect1>

  <sect1 id="nse-example-scripts">
    <title>Example Script</title>

    <sect2 id="nse-example-script-finger">
      <title>Finger-Test Script</title>
      <indexterm><primary sortas="Finger Results script">&ldquo;<literal>Finger Results</literal>&rdquo; script</primary></indexterm>
	  <para>The finger script (<filename>finger.nse</filename>) is a perfect
      example of how short typical NSE scripts are.
	  </para>

 <para>first the information fields are filled out, note that the 
<literal>id</literal> field is kept short, this is important since it is 
printed in Nmap's output. A detailed description of what the script
actually does should go in the <literal>description</literal> field.</para>
<programlisting>
id="Finger Results"<indexterm><primary sortas="id script variable">&ldquo;<varname>id</varname>&rdquo; script variable</primary></indexterm>

description="attempts to get a list of usernames via the finger service"<indexterm><primary sortas="description script variable">&ldquo;<varname>description</varname>&rdquo; script variable</primary></indexterm>

author = "Eddie Bell &lt;ejlbell@gmail.com&gt;"<indexterm><primary>Bell, Eddie</primary></indexterm><indexterm><primary sortas="author script variable">&ldquo;<varname>author</varname>&rdquo; script variable</primary></indexterm>

license = "Same as Nmap--See http://nmap.org/book/man-legal.html"<indexterm><primary sortas="license script variable">&ldquo;<varname>license</varname>&rdquo; script variable</primary></indexterm>
</programlisting>

<para>The <literal>categories</literal> field is a table
containing all the categories the script belongs to&mdash;These are used for
script selection through the <option>--script</option> option.</para>

<programlisting>
categories = {"discovery"}
</programlisting>

<para>You can use the facilities provided by the nselib (<xref
linkend="nse-library"/>) with <literal>require</literal>.  Here
we want to use shorter port rules.</para>

<programlisting>
require "shortport"
</programlisting>

<para>We want to run the script against the finger service.  So we
test whether it is using the well-known finger port (<literal>79/tcp</literal>), or
whether the service is named <quote>finger</quote> based on version
detection results or in the port number's listing
in <filename>nmap-services</filename>.</para>


<para>We want to check whether the service behind the port is finger,
or whether it runs on finger's well-known port 79. Through this we can
use the information gathered during the version scan (if finger runs
on a non-standard port) or still run against at least the port we
expect it, should the version detection information not be available.</para>

<programlisting>
portrule = shortport.port_or_service(79, "finger")<indexterm><primary sortas="portrule script variable">&ldquo;<varname>portrule</varname>&rdquo; script variable</primary></indexterm>

action = function(host, port)<indexterm><primary sortas="action script variable">&ldquo;<varname>action</varname>&rdquo; script variable</primary></indexterm>
  local socket = nmap.new_socket()
  local results = ""
  local status = true
</programlisting>

<para>The function <literal>err_catch()</literal> will be called for
clean up, through NSE's exception handling mechanism. Here it only
closes the previously opened socket (which should be enough in most
cases).</para>

<programlisting>
local err_catch = function()
  socket:close()
end
</programlisting>

<para>The clean up function gets registered for exception handling via
a call to <literal>nmap.new_try()</literal></para>

<programlisting>
  local try = nmap.new_try(err_catch())
</programlisting>

<para>The script sets a timeout of 5000, which is equivalent to 5
seconds. Should any operation require more time we'll receive a
<literal>TIMEOUT</literal> error message.</para>

<programlisting>
  socket:set_timeout(5000)
</programlisting>

<para>To make use of the exception handling we need to wrap calls to those functions which might return an error, inside <literal>try()</literal></para>

<programlisting>
  try(socket:connect(host.ip, port.number, port.protocol))
  try(socket:send("\n\r"))
</programlisting>

<para>The call to <literal>receive_lines()</literal> is not wrapped
in <literal>try()</literal>, because we don't want to abort the script
just because we didn't receive the data we expected. Note that if
there is less data than requested (100 lines), we will still receive
it and the status will be <literal>true</literal>&mdash;subsequent
calls would yield a <literal>false</literal> status.</para>

<programlisting>
  status, results = socket:receive_lines(100)
  socket:close()
</programlisting>

<para>The script returns a string if the call to <literal>receive_lines()</literal> was successful, otherwise it returns <literal>nil</literal>.</para>

<programlisting>
  return results
  end
</programlisting>
</sect2>
    <indexterm class="endofrange" startref="nse-sample-indexterm"/>
  </sect1>
  <sect1 id="nse-implementation">
    <title>Implementation</title>
    <indexterm><primary>Nmap Scripting Engine (NSE)</primary><secondary>implementation</secondary></indexterm>
	<para>
	Now how does all this work? The following section describes
	some interesting aspects of NSE. While the focus primarily lies on 
	giving script writers a better feeling of what happens with scripts, it 
	should also provide a starting point for understanding (and extending) the 
	NSE sources.
	</para>
    <sect2 id="nse-implementation-init">
      <title>Initialization Phase</title>
      <para>
      During its initialization stage, Nmap loads the Lua interpreter, including its provided libraries. These libraries are documented in the <ulink url="http://www.lua.org/manual/5.1/manual.html">Lua Reference Manual</ulink>.  Here is a summary:</para>
		<itemizedlist>
		<listitem>
			<para>The <emphasis>package</emphasis> library (namespace: 
			<literal>package</literal>)&mdash;Lua's 
			<ulink role="hidepdf" url="http://www.lua.org/manual/5.1/manual.html#5.3">package-lib</ulink> provides (among others) the <literal>require</literal> function, used to load modules from the
			nselib.
			</para>
		</listitem>
		<listitem>
			<para>The <emphasis>table</emphasis> library (namespace: 
			<literal>table</literal>)&mdash;The 
			<ulink role="hidepdf" url="http://www.lua.org/manual/5.1/manual.html#5.5">table manipulation library</ulink> contains many functions used
			to operate on <literal>tables</literal>&mdash;Lua's central data 
			structure.
			</para>
		</listitem>
		<listitem>
			<para>The <emphasis>I/O</emphasis> library (namespace: 
			<literal>io</literal>)&mdash;The 
			<ulink role="hidepdf" url="http://www.lua.org/manual/5.1/manual.html#5.7">Input/Output library</ulink> offers functions such as reading files and reading the output from programs you execute.
			</para>
		</listitem>
		<listitem>
			<para>The <emphasis>OS</emphasis> library (namespace: 
			<literal>os</literal>)&mdash;The 
			<ulink role="hidepdf" url="http://www.lua.org/manual/5.1/manual.html#5.8">Operating System library</ulink> provides facilities of the operating system, including filesystem operations (renaming/removing files, temporary file creation) and access to the environment.</para>
		</listitem>
		<listitem>
			<para>The <emphasis>string</emphasis> library (namespace: 
			<literal>string</literal>)&mdash;The 
			<ulink role="hidepdf" url="http://www.lua.org/manual/5.1/manual.html#5.4">
			string library </ulink> helps you with functions used to manipulate
			strings inside Lua. Functions include: printf-style 
			string formatting, pattern matching using Lua-style patterns, 
			substring extraction, etc.
			</para>
		</listitem>
		<listitem>
			<para>The <emphasis>math</emphasis> library (namespace: 
			<literal>math</literal>)&mdash;Numbers in Lua usually correspond to the <literal>double</literal> C type, so the <ulink role="hidepdf" url="http://www.lua.org/manual/5.1/manual.html#5.6">math library</ulink> provides access to rounding functions, trigonometric functions, random number generation, and more.</para>
		</listitem>
		<listitem>
			<para>The <emphasis>debug</emphasis> library (namespace: 
			<literal>debug</literal>)&mdash;The 
			<ulink role="hidepdf" url="http://www.lua.org/manual/5.1/manual.html#5.9">debug library</ulink> provides you with a somewhat lower level API
			to the Lua interpreter. Through it you can access functions along 
			the execution stack, get function closures and object metatables,
			etc. 
			</para>
		</listitem>
	    </itemizedlist>

<para>In addition to loading the libraries provided by Lua, the functions in the <literal>nmap</literal> namespace are loaded. The search paths are the same directories that Nmap searches for its data files and scripts, except that the <literal>nselib</literal> directory is appended to each. In this step the provided script arguments are stored inside the registry.<indexterm><primary>registry (NSE)</primary></indexterm></para>


		<para>
		The next phase of NSE initialization is loading the chosen
		scripts, which are the arguments provided to the 
		<option>--script</option><indexterm><primary><option>--script</option></primary></indexterm>
                option or <literal>default</literal>, in 
		case of a default script scan.  The string
		<literal>version</literal><indexterm><primary><varname>version</varname> script category</primary></indexterm>
		is appended, if version detection was enabled.
		The arguments afterwards are tried to be 
		interpreted as script categories. This is done via a Lua C function
		in <filename>nse_init.cc</filename> called <literal>entry</literal>.
        Inside <filename>script.db</filename>,<indexterm><primary><filename>script.db</filename></primary><seealso><option>--script-updatedb</option></seealso></indexterm>
        for each category of a script,
        there is a call to <literal>Entry</literal>. If the category was chosen
        then the script is loaded. Every argument of
        <option>--script</option> that could not be interpreted as a category
        is loaded as a file or directory. If the file or directory could not
        be located, then an error is raised and the Script Engine aborts.
        </para>
        
		<para>
        All of the <literal>.nse</literal> files inside a loaded directory are 
        loaded as files. Each file loaded is executed by Lua. If a
        <emphasis>portrule</emphasis> is present, then it is saved in the
        <emphasis>porttests</emphasis> table with a portrule key and file
        closure value. Otherwise, if the script has a
        <emphasis>hostrule</emphasis>, then it is saved in the <emphasis>hosttests</emphasis> table
        in the same manner.
        </para>
    </sect2>
    
	<sect2 id="nse-implementation-match">
      <title>Matching of Scripts to Targets</title>
      <para>
	  After the initialization is finished the
	  <literal>hostrules</literal><indexterm><primary sortas="hostrule script variable">&ldquo;<varname>hostrule</varname>&rdquo; script variable</primary></indexterm>
          and <literal>portrules</literal><indexterm><primary sortas="portrule script variable">&ldquo;<varname>portrule</varname>&rdquo; script variable</primary></indexterm>
          are evaluated for each host in the current
	  target group. At this check a list is built which contains the combinations of scripts and the hosts they will run against.

It should be noted that the rules of all chosen scripts are
checked against all hosts and their
<literal>open</literal><indexterm><primary><literal>open</literal> port state</primary></indexterm>
and <literal>open|filtered</literal><indexterm><primary><literal>open|filtered</literal> port state</primary></indexterm>
ports.
Therefore it is advisable to leave the rules as simple as possible and
to do all the computation inside the <literal>action</literal>, as a script will only be
executed if it is run against a specific target. After the check those script-target combinations
get their own <ulink url="http://www.lua.org/manual/5.1/manual.html#2.11">Lua thread</ulink>. A
thread running against a host will have only a hostrule passed to the action closure whereas
a thread running against a port will have both a hostrule and portrule passed. Each thread
is stored with information relevant to the thread. This information
includes the runlevel, target, target port (if applicable), host and port tables
(passed to action), its type (running against a host or port), and its id.
The mainloop function will work on each runlevel grouping of threads in order.
</para>

    </sect2>

	<sect2 id="nse-implementation-run">
      <title>Running Scripts</title>

      <para>
        Nmap is able to perform NSE script scanning in
	parallel<indexterm><primary>parallelism</primary><secondary>in NSE</secondary></indexterm>
        by making use of Lua language features.  In particular,
	  <ulink url="http://www.lua.org/manual/5.1/manual.html#2.11">coroutines
	  </ulink> offer collaborative multi-threading so scripts can suspend themselves at defined points, and allow other coroutines 
	  to execute. Since network I/O, especially waiting for responses from
	  remote host, is the part of scripts which would consume most time with
	  waiting, this is the point where scripts suspend themselves and let
	  others execute. Each call to some of the functions of the Nsock wrapper 
	  causes the calling script to yield (pause). Once the request is
      processed by the Nsock library, the 
	  callback causes the script to be pushed from the waiting queue to the
	  running queue, which will eventually let it resume its operation.</para>
      <para>
      The mainloop function will maintain two sets of threads, running and
      waiting. Threads will be
      moved back and forth between the sets; when a thread yields, it
      is moved to the waiting group. Threads run in the running set will either
      yield, complete, or error. After all scripts are resumed in the running
      set, mainloop will place all yielded threads ready to be
      run in the running set. Threads are made "ready" by calling 
      <literal>process_waiting2running</literal>. This process of running
      threads and moving paused threads to the waiting and running sets is
      repeated until no threads exist in either waiting or running.
	  </para>
    </sect2>
  </sect1>
<indexterm class="endofrange" startref="nse-indexterm"/>
