Lua Extensions

In addition to the significant built-in capabilities of Lua, we have written or integrated several extensions to make NSE scripts more powerful and convenient to write. These modules are compiled and installed along with Nmap. They have their own directory, nselib, which is installed in the configured datadir. Scripts need only require the default modules in order to use them. The default modules are described in the following sections.

Bitwise Logical Operations

Lua does not provide bitwise logical operations. Since they are often useful for low-level network communication, Reuben Thomas' bitwise operation library for Lua has been integrated into NSE. The arguments to the bitwise operation functions should be integers. The number of bits available for logical operations depends on the data type used to represent Lua numbers—this is typically 8-byte IEEE floats (double), which give 53 bits (the size of the mantissa). This implies that the bitwise operations won't work (as expected) for numbers larger than 10¹⁴. You can use them with 32-bit wide numbers without any problems. Operations involving 64-bit wide numbers, however, may not return the expected result. The logical operations start with “b” (for bit) to avoid clashing with reserved words; although xor isn't a reserved word, it seemed better to use bxor for consistency. In NSE the bitwise functions are in the bit namespace.

bit.bnot(a): Returns the one's complement of a.
bit.band(w1,...): Returns the bitwise and of the w variables.
bit.bor(w1,...): Returns the bitwise or of the w variables.
bit.bxor(w1,...): Returns the bitwise xor of the w variables.
bit.lshift(a,b): Returns a shifted left b places—padded with zeros.
bit.rshift(a,b): Returns a shifted logically right b places.
bit.arshift(a,b): Returns a shifted arithmetically right b places.
bit.mod(a,b): Returns the integer remainder of a divided by b.

Binary Data Handling

A problem script authors often face is the necessity of encoding values into binary data. For example after analyzing a protocol the starting point to write a script could be a hex dump, which serves as a preamble to every sent packet. Although it is possible work with the functionality Lua provides, it's not very convenient. Therefore the Binlib has been added to NSE, based on lpack by Luiz Henrique de Figueiredo. The Binlib functions take a format string to encode and decode binary data. The operators of the format string are shown in Table 9.1.

Table 9.1. Binlib format string operators

Operator	Description
`H`	hex string
`B`	bit string
`x`	null byte
`z`	zero-terminated string
`p`	string preceded by length byte
`P`	string preceded by length word
`a`	string preceded by length size_t
`A`	string
`f`	float
`d`	double
`n`	Lua number
`c`	char
`C`	byte = unsigned char
`s`	short
`S`	unsigned short
`i`	int
`I`	unsigned int
`l`	long
`L`	unsigned long
`<`	little endian modifier
`>`	big endian modifier
`=`	native endian modifier

Note that the endian operators work as modifiers to all the characters following them in the format string.

bin.pack(fmt, p1, ...): Returns a binary packed string. The format string describes how the parameters (p1, ...) will be interpreted. Numerical values following operators stand for operator repetitions and need an according amount of parameters. Operators expect appropriate parameter types.
bin.unpack(fmt, data, [init]): Returns values read from the binary data string. First result is the position, at which unpack stopped. This can be used as init value for subsequent calls. The following results are the values according to the format string. Numerical values in the format string are interpreted as repetitions like in pack, except if used with A, B or H, in which cases the number tells unpack how many bytes to read. Unpack stops if either the format string or the binary data string are exhausted.

Perl Compatible Regular Expressions

One of Lua's quirks is its string patterns. While they have great performance and are tightly integrated into the Lua interpreter, they are very different in syntax and not as powerful as standard regular expressions. So we have integrated Perl compatible regular expressions into Lua using PCRE and a modified version of the Lua PCRE library written by Reuben Thomas and Shmuel Zeigerman. These are the same sort of regular expressions used by Nmap version detection. The main modification to their library is that the NSE version only supports PCRE expressions instead of both PCRE and POSIX patterns. In order to maintain a high script execution speed, the library interfacing with PCRE is kept very thin. It is not integrated as seamlessly as the Lua string pattern API. This allows script authors to decide when to use PCRE expressions versus Lua patterns. The use of PCRE involves a separate pattern compilation step, which saves execution time when patterns are reused. Compiled patterns can be cached in the NSE registry and reused by other scripts. The PCRE functions reside inside the pcre namespace.

Warning

PCRE has a history of security vulnerabilities allowing attackers who are able to compile arbitrary regular expressions to execute arbitrary code. More such vulnerabilities may be discovered in the future. These have never affected Nmap because it doesn't give attackers any control over the regular expressions it uses. Similarly, NSE scripts should never build regular expressions with untrusted network input. Matching hardcoded regular expressions against the untrusted input is fine.

The following documentation is derived from that supplied by the PCRE Lua lib.

pcre.new(pattern, flags, locale)

Returns a compiled regular expression. The first argument is a string describing the pattern, such as ^foo$. The second argument is a number describing which compilation flags are set. The compilation flags are set bitwise. If you want to set the 3rd (corresponding to the number 4) and the 1st (corresponding to 1) bit for example you would pass the number 5 as a second argument. The compilation flags accepted are those of the PCRE C library. These include flags for case insensitive matching (1), matching line beginnings (^) and endings ($) even in multiline strings (i.e. strings containing “\n”) (2) and a flag for matching across line boundaries (4). No compilation flags yield a default value of 0. The third (optional) argument is a string describing the locale which should be used to compile the regular expression. The variable is a string which is passed to the C standard library function setlocale. For more information on this argument refer to the documentation of setlocale. The resulting compiled regular expression is ready to be matched against strings. Compiled regular expressions are subject to Lua's garbage collection. Generally speaking, my_regex = pcre.new("<pcre-pattern>",0,"C") should do the job most of the time.

pcre.flags()

Returns a table of the available PCRE option flags (numbers) keyed by their names (strings). Possible names of the available strings can be retrieved from the documentation of the PCRE library used to link against Nmap. The key is the option name in the manual minus the PCRE prefix. PCRE_CASELESS becomes CASELESS for example.

pcre.version()

Returns the version of the PCRE library in use as a string. For example 6.4 05-Sep-2005.

pcre_obj:match(string, start, flags)

Returns the start point and the end point of the first match of the compiled regular expression pcre_obj in the string. A third returned value is a table which contains false in the positions where the pattern did not match. If named sub-patterns were used, the table also contains substring matches keyed by their sub-pattern name. Should no match be found the function returns nil. The second and third arguments are optional. The second argument is a number specifying where the engine should start trying to apply the pattern. The third argument specifies execution flags for the pattern. If you want to see if a given string matches a certain expression you could use:

s = pcre_obj:match("string to be searched", 0,0);
if(s) code_to_be_done_on_match end

pcre_obj:exec(string, start, flags)

This function is like match() except that a table returned as a third result contains offsets of substring matches rather than substring matches themselves. That table will not contain string keys, even if named sub-patterns are used. For example, if the whole match is at offsets 10, 20 and substring matches are at offsets 12, 14 and 16, 19 then the function returns the following: 10, 20, {12,14,16,19}

pcre_obj:gmatch(string, func, n, ef)

Tries to match the regular expression <pcre_obj> against <string> up to <n> times (or as many as possible if <n> is either not given or is not a positive number), subject to execution flags ef. Each time there is a match, <func> is called as <func(m, t)>, where <m> is the matched string and <t> is a table of substring matches. This table contains false in the positions where the corresponding sub-pattern did not match. If named sub-patterns are used then the table also contains substring matches keyed by their correspondent sub-pattern names (strings). If <func> returns a true value, then gmatch immediately returns; gmatch returns the number of matches made.

IP Operations

The ipOps module provides some functions for manipulating IPv4 addresses. The functions reside inside the ipOps namespace.

bool = ipOps.isPrivate("ip-string"): checks whether an IP address, provided as a string in dotted-quad notation, is part of the non-routed private IP address space, as described in RFC 1918. These addresses are the well-known 10.0.0.0/8, 192.168.0.0/16 and 172.16.0.0/12 networks.
DWORD = ipOps.todword("ip-string"): returns the IP address as DWORD value (i.e. the IP <a.b.c.d> becomes (((a*256+b)*256+c)*256+d) )
a,b,c,d = ipOps.get_parts_as_number("ip-string"): returns 4 numbers corresponding to the fields in dotted-quad notation. For example, ipOps.get_parts_as_number("192.168.1.1") returns 192,168,1,1.

Short Portrules

Since portrules are mostly the same for many scripts, the shortport module provides functions for the most common tests. The arguments in brackets ([]) are optional. If no proto is provided, tcp is used. The default state is open

shortport.portnumber(port,[proto],[state]): The port argument is either a number or a table of numbers which are interpreted as port numbers, against which the script should run.
shortport.service(service,[proto],[state]): The service argument is either a string or a table of strings which are interpreted as service names (e.g. "http", "https", "smtp" or "ftp") against which the script should run. These service names are determined by Nmap's version scan or (if no version scan information is available) the service assigned to the port in nmap-services (e.g. "http" for TCP port 80).
shortport.port_or_service(port,service,[proto],[state]): This is a combination of the above functions, since many scripts explicitly try to run against the well-known ports, but want also to run against any other port which was discovered to run the named service. A typical example for this function is: portrule = shortport.port_or_service(22,"ssh").

Functional Programming Style List Operations

People used to programming in functional languages, such as Lisp or Haskell, appreciate their handling of lists very much. The listop module tries to bring much of the functionality from functional languages to Lua using Lua's central data structure, the table, as a base for its list operations. Highlights include a map function applying a given function to each element of a list.

bool = listop.is_empty(list): Returns true if the given list is empty.
bool = listop.is_list(value): Returns true if the given value is a list (or rather a table).
list = listop.map(function, list): The provided function is applied to each element of the list separately. The returned list contains the results of each function call. For example listop.map(tostring,{1,2,true}) returns {"1","2","true"}.
value = listop.apply(function, list): All of the elements in the list are passed to a call of function. The result is then returned. For example listop.apply(math.max,{1,5,6,7,50000}) yields 50000.
list = listop.filter(predicate, list): Returns a list containing only those elements for which the predicate returns true. The predicate has to be a function, which takes an element of the list as argument and the result of which is interpreted as a Boolean value. If it returns true (or rather anything besides false and nil) the argument is appended to the return value of filter. For example: listop.filter(isnumber,{1,2,3,"foo",4,"bar"}) returns {1,2,3,4}.
list = listop.flatten(list): Since a list can itself contain lists as elements, flatten returns a list which only contains values that are not themselves lists. For example: listop.flatten({1,2,3,"foo",{4,5,{"bar"}}}) returns {1,2,3,"foo",4,5,"bar"}.
list = listop.append(list1, list2): Returns a list containing all elements of list1 appended by all elements of <list2>.
list = listop.cons(value1, value2): Returns a list containing <value1> appended by <value2>, which may be of any type.
list = listop.reverse(list): Returns a list containing all elements of the given list in inverted order.
value = listop.car(list): Returns the first element of the given list.
value = listop.ncar(list,n): Returns the nth (or first if n is omitted) element of the given list.
value = listop.cdr(list): Returns a list containing all elements but the first of the given list.
value = listop.ncdr(list, n): Returns a list containing all elements but the first n of the given list, where n is 2 if it is omitted.

String Buffer Operations

Lua's string operations are very flexible and offer an easy-to-use way to manipulate strings. Concatenation using the .. operator is such an operation. The drawback of the built-in API however is the way it handles concatenation of many string values. Since strings in Lua are immutable values, each time you concatenate two strings both get copied into the result string. The strbuf module offers a workaround for this problem, while maintaining the nice syntax. This is accomplished by overloading the concatenation operator (..) the equality operator (==) and the tostring operator. By overloading these operators, we reduce the overhead of using a string buffer instead of a plain string to wrap the first literal string assigned to a variable inside a strbuf.new() call. Afterwards you can append to the string buffer, or compare two string buffers for equality just as you would do with normal strings. When looking at the details there are some more restrictions/oddities: The concatenation operator requires its left-hand value to be a string buffer. Therefore, if you want to prepend a string to a given string buffer you have to create a new string buffer out of the string you want to prepend. The string buffer's tostring operator concatenates the strings inside the buffer using newlines by default, since this appears to be the separator used most often.

buffer = strbuf.new(...): Creates a new string buffer. The optional arguments are added to the string buffer. Attempting to add non-strings will result in undefined behavior.
buffer = strbuf.concat(strbuf1, value): Concatenates the value (which has to be either a string or a string buffer) to strbuf1. This is also the function serving as the string buffer's concatenation operator. The above function call can thus also be expressed as: buffer = strbuf1 .. value
bool = strbuf.eqbuf(strbuf1, strbuf2): Compares strbuf1 and strbuf2 for equality. For the function to return true, both values must be string buffers containing exactly the same strings. The eqbuf function is called to compare two strings for equality.
strbuf.clear(strbuf): Deletes all strings in strbuf.
string = strbuf.dump(strbuf, "delimiter"): Dumps strbuf's contents as string. The second parameter is used as a delimiter between the strings stored inside strbuf. dump(strbuf, "\n") is used as the tostring function of string buffers.

URL Manipulation Functions

URL manipulation functions have obvious uses. Fortunately there is already an implementation of URL generation functions inside the Lua socket package, which is fairly complete and well documented. For NSE, the url module was extended with two functions:

table = url.parse_query("query-string"): This function takes a <query-string> of the form name1=value1&name2=value2... and returns a table containing the name-value pairs, with the name as the key and the value as its associated value. The table corresponding to the above <query-string> would have two entries: table["name1"]="value1" and table["name2"]="value2".
query_string = url.build_query(table): This is the inverse function to parse_query().

Buffered Network I/O Helper Functions

The match module was written to provide functions which can be used for delimiting data received by the receive_buf() function from the Network I/O API:

start,end = match.regex("regexpattern"): This is actually a wrapper around NSE's PCRE library exec function (see the section called “Perl Compatible Regular Expressions”), thus giving script developers the possibility to use regular expressions for delimiting instead of Lua's string patterns. If you want to get the data in chunks separated by pattern (which has to be a valid regular expression), you would write status, val = sockobj:receive_buf(match.regex("pattern")).
start,end = match.numbytes(number): Takes a number as its argument and returns that many bytes. It can be used to get a buffered version of sockobj:receive_bytes(n) in case a script requires more than one fixed-size chunk, as the unbuffered version may return more bytes than requested and thus would require you to do the parsing on your own.

HTTP Functions

The http module provides functions for dealing with the client side of the http protocol. The functions reside inside the http namespace. The return value of each function in this module is a table with the following keys: status, header and body. status is a number representing the HTTP status code returned in response to the HTTP request. In case of an unhandled error, status is nil. The header value is a table containing key-value pairs of HTTP headers received in response to the request. The header names are in lower-case and are the keys to their corresponding header values (e.g. header.location = "http://nmap.org/"). Multiple headers of the same name are concatenated and separated by commas. The body value is a string containing the body of the HTTP response.

table = http.get(host,port,path,[options]): Fetches a resource with a GET request. The first argument is either a string with the hostname or a table like the host table passed by nmap. The second argument is either the port number or a table like the port table passed by nmap. The third argument is the path of the resource. The fourth argument is a table for further options. The table may have 2 keys: timeout and header. timeout is the timeout used for the socket operations. header is a table with additional headers to be used for the request. The function builds the request and calls http.request
table = http.request(host,port,request,[options]): Sends request to host:port and parses the answer. The first argument is either a string with the hostname or a table like the host table passed by nmap. The second argument is either the port number or a table like the port table passed by nmap. SSL is used for the request if either port.service equals https or port.version.service_tunnel equals ssl. The third argument is the request. The fourth argument is a table for further options. You can specify a timeout for the socket operations with the timeout key.
table = http.get_url(url,[options]): Parses url and calls http.get with the result. The second argument is a table for further options. The table may have 2 keys: timeout and header. timeout is the timeout used for the socket operations. header is a table with additional headers to be used for the request.

Common Communication Functions

The comm module provides functions for common network discovery tasks such as banner-grabbing and making a quick exchange of data. These functions' return values are setup for use with exception handling via nmap.new_try().

These functions can all be passed a table of options, but it's not required. The relevant indexes for this table are bytes, lines, proto and timeout. bytes is used to provide the minimum number of bytes required for a read. lines does the same, but for the minimum number of lines. If neither are provided, these functions attempt to read as many bytes as are available. proto is used to set the protocol to communicate with, defaulting to "tcp" if not provided. timeout is used to set the socket timeout (see the socket function set_timeout() for details).

bool, response = comm.get_banner(host, port, [options]): This function simply connects to the specified port number on the specified host and returns any data received. bool is a Boolean value indicating success. If bool is true, then the second returned value is the response from the target host. If bool is false, an error message is returned as the second value instead of a response.
bool, response = comm.exchange(host, port, data, [options]): This function connects to the specified port number on the specified host, sends data, then waits for and returns the response, if any. bool is a Boolean value indicating success. If bool is true, then the second returned value is the response from the target host. If bool is false, an error message is returned as the second value instead of a response.

Username/Password Database Functions

The unpwdb module provides functions for easily obtaining usernames and/or passwords from a "database" (list). The most obvious use for this library is brute-force attack. The first two functions' return values are setup for use with exception handling via nmap.new_try().

bool, response = unpwdb.usernames(): This function returns a closure which returns a new username with every call, or nil when the list is exhausted. This closure takes an optional argument of "reset" to rewind the list to the beginning. You can specify your own username list with the script argument userdb.
bool, response = unpwdb.passwords(): This function returns a closure which returns a new password with every call, or nil when the list is exhausted. This closure takes an optional argument of "reset" to rewind the list to the beginning. You can specify your own password list with the script argument passdb.
limit = unpwdb.timelimit(): This function returns the suggested number of seconds to attempt a brute force attack, based on Nmap's timing values (-T4, etc) and whether or not a user-defined list is used. You can use the script argument notimelimit to make this function return nil, which means the brute-force should run until the list is empty. If notimelimit is not used, be sure to still check for nil return values on the above two functions in case you finish before the time limit is up.

Data File Parsing Functions

The datafiles module provides functions for reading and parsing Nmap's data files (e.g. nmap-protocol, nmap-rpc, etc.). These functions' return values are setup for use with exception handling via nmap.new_try().

bool, table = datafiles.parse_protocols(): This function reads and parses Nmap's nmap-protocols file. bool is a Boolean value indicating success. If bool is true, then the second returned value is a table with protocol numbers indexing the protocol names. If bool is false, an error message is returned as the second value instead of the table.
bool, table = datafiles.parse_rpc(): This function reads and parses Nmap's nmap-rpc file. bool is a Boolean value indicating success. If bool is true, then the second returned value is a table with RPC numbers indexing the RPC names. If bool is false, an error message is returned as the second value instead of the table.
bool, table = datafiles.parse_services([protocol]): This function reads and parses Nmap's nmap-services file. bool is a Boolean value indicating success. If bool is true, then the second returned value is a table containing two other tables: tcp{} and udp{}. tcp{} contains services indexed by TCP port numbers. udp{} is the same, but for UDP. You can pass "tcp" or "udp" as an argument to parse_services() to only get the corresponding table. If bool is false, an error message is returned as the second value instead of the table.

Various Utility Functions

The stdnse library contains various handy functions which are too small to justify modules of their own:

stdnse.print_debug([verbosity,] format, ...): Wrapper function around print_debug_unformatted() in the nmap namespace. The first optional numeric argument, verbosity, is used as the necessary debug level to print the message (it defaults to 1 if omitted). All remaining arguments are processed with Lua's string.format() function, which provides a C-style printf interface.
list = stdnse.strsplit("delimiter", "text"): This function will certainly be appreciated by Perl programmers. It takes two strings as arguments and splits the second one around all occurrences of the first one, returning a list (table), which contains the substrings without the delimiting string.
string = stdnse.strjoin("delimiter", list): Inverse function to strsplit(). Basically this is Lua's table.concat() function with the parameters swapped for coherence.
string = stdnse.tobinary(n): Converts the given number, n, to a string in a binary number format (e.g. 5 becomes "101").
string = stdnse.tooctal(n): Converts the given number, n, to a string in an octal number format (e.g. 9 becomes "11").
string = stdnse.tohex(n): Converts the given number, n, to a string in a hexidecimal number format (e.g. 10 becomes "a").
string = stdnse.make_buffer(socket, sep): This function operates on a socket attempting to read data. It separates the data by sep and, for each invocation, returns a piece of the separated data. Typically this is used to iterate over the lines of data received from a socket (sep = "\r?\n"). The returned string does not include the separator. It will return the final data even if it is not followed by the separator. Once an error or EOF is reached, it returns nil, msg. msg is what is returned by nmap.receive_lines().