During its initialization stage, Nmap loads the Lua interpreter, including its provided libraries. These libraries are documented in the Lua Reference Manual. Here is a summary:

In addition to loading the libraries provided by Lua, the functions in the nmap namespace are loaded. The search paths are the same directories that Nmap searches for its data files and scripts, except that the nselib directory is appended to each. In this step the provided script arguments are stored inside the registry.

The next phase of NSE initialization is loading the chosen scripts, which are the arguments provided to the --script option or default, in case of a default script scan. The string version is appended, if version detection was enabled. The arguments afterwards are tried to be interpreted as script categories. This is done via a Lua C function in nse_init.cc called entry. Inside script.db, for each category of a script, there is a call to Entry. If the category was chosen then the script is loaded. Every argument of --script that could not be interpreted as a category is loaded as a file or directory. If the file or directory could not be located, then an error is raised and the Script Engine aborts.

All of the .nse files inside a loaded directory are loaded as files. Each file loaded is executed by Lua. If a portrule is present, then it is saved in the porttests table with a portrule key and file closure value. Otherwise, if the script has a hostrule , then it is saved in the hosttests table in the same manner.

After the initialization is finished the hostrules and portrules are evaluated for each host in the current target group. At this check a list is built which contains the combinations of scripts and the hosts they will run against. It should be noted that the rules of all chosen scripts are checked against all hosts and their open and open|filtered ports. Therefore it is advisable to leave the rules as simple as possible and to do all the computation inside the action, as a script will only be executed if it is run against a specific target. After the check those script-target combinations get their own Lua thread. A thread running against a host will have only a hostrule passed to the action closure whereas a thread running against a port will have both a hostrule and portrule passed. Each thread is stored with information relevant to the thread. This information includes the runlevel, target, target port (if applicable), host and port tables (passed to action), its type (running against a host or port), and its id. The mainloop function will work on each runlevel grouping of threads in order.

Nmap is able to perform NSE script scanning in parallel by making use of Lua language features. In particular, coroutines offer collaborative multi-threading so scripts can suspend themselves at defined points, and allow other coroutines to execute. Since network I/O, especially waiting for responses from remote host, is the part of scripts which would consume most time with waiting, this is the point where scripts suspend themselves and let others execute. Each call to some of the functions of the Nsock wrapper causes the calling script to yield (pause). Once the request is processed by the Nsock library, the callback causes the script to be pushed from the waiting queue to the running queue, which will eventually let it resume its operation.

The mainloop function will maintain two sets of threads, running and waiting. Threads will be moved back and forth between the sets; when a thread yields, it is moved to the waiting group. Threads run in the running set will either yield, complete, or error. After all scripts are resumed in the running set, mainloop will place all yielded threads ready to be run in the running set. Threads are made "ready" by calling process_waiting2running. This process of running threads and moving paused threads to the waiting and running sets is repeated until no threads exist in either waiting or running.

This section gives a short walk-through for adding nselib modules written in C (or C++) to Nmap's build system, since this has shown to be sometimes tedious. Writing C modules is described at length in Programming in Lua, Second Edition. Basically C modules consist of the functions they provide to Lua, which have to be of type lua_CFunction. Additionally they have to contain a function which is used to actually open the module. By convention these function names are luaopen_<modulename>. A good starting point for writing such modules is provided by bit.c inside the nselib/ subdirectory of Nmap's source tree. bit is a C module already provided by the nselib. C modules basically are shared libraries which get loaded at runtime by Lua.

The Unix build system uses libtool for compilation in a platform independent way. As long as the new module does not depend on foreign libraries, you should only need to add <modulename>.so to the all and clean targets in Makefile.in and copy and adapt the lines from bit.so. If your module does have dependencies you will most probably have to add checks for those libraries to configure.ac and put the dependencies inside the libtool commands in Makefile.in.

Of course, theory and practice are rarely the same. Most of the trouble building nselib actually comes from the complications of building shared libraries and not nselib itself. Linking with static libraries (e.g. libnbase) sometimes leads to problems with exporting symbols on some platforms (in our case the x86_64-linux platform). To our knowledge no such problems occur when linking against already existing shared libraries.

The Windows build system requires C module developers to create a MS Visual Studio Project file for their module (<modulename>.vcproj) inside the nselib subdirectory. On Windows you have to include the liblua/ subdirectory as an additional include path as well as a library search path. In addition you have to tell the project to link against the liblua.lib static library provided with Nmap. Other properties of the project should be the same as for other nselib C modules (e.g. see nse_bitlib.vcproj). Afterwards you have to include the newly created project file in Nmap's Visual Studio solution file (mswin32\nmap.sln) and make sure that nse_bitlib.vcproj depends on your project, because it is there that nselib modules get copied to their final destinations and are included in Nmap.