Lisp Executable

The goal of this Common Lisp library is to provide a convenient abstraction to creating executable programs that can interact easily with other tools that accompany the Unix shell. The abstraction should be familiar and straight forward to use.

The library consists of

• a language for defining programs that accept command line arguments.
• the ability to automatically generate an executable for a defined program.
• an ASDF:COMPONENT so that executables can be built using an ASDF:OPERATION.

The library provides two different styles of passing command line arguments to the defined program

Program

A program which accepts options and arguments with options appearing anywhere on the command line.

Dispatcher

A program which dispatches to other programs. Example software of this style would be xargs, git and svn.

Lastly, the automatic executable creation relies heavily on ASDF. It is assumed that all programs are declared within an ASDF system.

Automatic executable creation has been implemented for SBCL, ECL, CLISP, CCL and CMUCL.

• Create an ASDF system to manage your program.

  ;; File lisp-executable-example.asd
  
  (eval-when (:compile-toplevel :load-toplevel :execute)
    (asdf:load-system "lisp-executable"))
  
  (defsystem lisp-executable-example
    :components ((:modules "example/"
                           :serial t
                           :components ((:file "main")
                                        (lisp-executable:executable "example-program" :program ("LISP-EXECUTABLE.EXAMPLE" "EXAMPLE-PROGRAM"))))))  
  

• Create main.lisp and specify your program.

    ;; File example/main.lisp
  (defpackage "LISP-EXECUTABLE.EXAMPLE"
    (:use "COMMON-LISP"
          "LISP-EXECUTABLE"))
  (in-package "LISP-EXECUTABLE.EXAMPLE")
  
  (define-program example-program (&options help)
    (cond
      (help
       (format t "Help has arrived."))
      (t
       (format t "You are doomed.")))
    (terpri))
  

• Create the executable.

  (asdf:oos 'lisp-executable:create-executables-op "lisp-executable-example")
  

• Invoke from the command line.

  $ example/example-program
  You are doomed.
  $ example/example-program --help
  Help has arrived.
  $
  

The API is separated in to two parts (i) the definition of the command line arguments the program accepts (ii) the implementation on how to associate command line arguments to the program argument symbols.

The following terms are used in this document to describe the various components of programs that accept command line arguments:

Command line program

The program that is being invoked from the command line.

Command line arguments

The list of arguments being passed to the program.

Program option (or option)

A command line argument that modifies how the command line program performs its task. Program options can accept parameters.

The first type of command line program introduced is one where the program options can appear at any position within the command line arguments. For example, the following two invocations of the command line program process-file are equivalent

$ process-file input output -v
$ process-file -v input output

To define this type of program you use the macro LISP-EXECUTABLE:DEFINE-PROGRAM.

(defmacro define-program (program-name program-lambda-args &body body))

The symbol PROGRAM-NAME is used to identify the command line program. The format of PROGRAM-LAMBDA-ARGS is presented in the next section. Finally, the code that uses the command line arguments is placed in BODY.

A dispatcher program is one in which the operation to be performed is determined from the command line. For example, the program git has a number of commands which are all accessed via git

$ git init
$ git status
$ git reset

and so on. The goal of the dispatcher program is to easily define these types of programs.

The key difference between a dispatcher program and the program defined in the previous section is in the handling of the command line options. Any option occurring before an argument is an option to the dispatcher and any option occurring after an argument is a option to the dispatched program.

An example dispatcher program can be defined as follows

(define-dispatcher-program git (&options help &arguments command &others others)
  (cond
    ((or help (null command))
     (print-usage))
    (command
     (alexandria:switch (command :test #'string-equal)
       ("init"
        (program-apply 'git/init others))
       ("commit"
        (program-apply 'git/commit others))
       (t
        (error "Don't know how to perform command ~A" command))))))

The declarations IDENTIFIERS, CONVERSION-FUNCTION and REDUCING-POLICY can be used within the DEFINE-DISPATCHER-PROGRAM form as well.

A defined program can be tested by using the functions PROGRAM-FUNCALL and PROGRAM-APPLY. The arguments passed to these functions must be of type string. The identification of options and non option arguments is handled by the object bound to *COMMAND-LINE-ARGUMENTS-READER*.

(define-program my-program (&options help (file file-value) &arguments what-to-do)
  (list help file-value what-to-do))

(setf *command-line-arguments-reader* 'gnu-style)

(program-funcall 'my-program "hello-there")
; => (NIL NIL "hello-there")
(program-funcall 'my-program "--help")
; => (T NIL NIL)
(program-funcall 'my-program "--file=good-program")
; => (NIL "good-program" NIL)

The function PROGRAM-APPLY is to PROGRAM-FUNCALL as the Common Lisp function APPLY is to FUNCALL.

If you want to test the program without considering how options are read from the command line, the functions PROGRAM-FUNCALL-WITH-ALIST and PROGRAM-FUNCALL-WITH-PLIST can be used.

(program-funcall-with-alist 'my-program '((help t)))
(program-funcall-with-plist 'my-program 'help t)

(program-funcall-with-alist 'my-program '((file t) (file-value "input.txt")))
(program-funcall-with-plist 'my-program '(file t file-value "input.txt"))

The object bound to the symbol *COMMAND-LINE-ARGUMENTS-READER* represents the method in which the command line arguments are identified. As of writing, GNU-STYLE is the only implemented style of identifying options and arguments from strings.

The GNU style uses the following templates for options

-h

A short option with identifier h.

--help

A long option with identifier help.

--debug=1

A long option with identifier debug and parameter 1.

--file input.txt

A long option with identifier file and parameter input.txt. Valid for mandatory parameter options only.

-f input.txt

A short option with identifier f and parameter input.txt. Valid for mandatory parameter options only.

--

Terminate option processing. i.e. All options found after this delimiter will be treated as non option arguments.

One of the features of the LISP-EXECUTABLE library is that it is possible to generate an executable from a command line program definition.

The function provided to do this is CREATE-EXECUTABLE.

(define-program my-program (&options help)
  (cond
    (help
     (format t "Help has arrived."))
    (t
     (format t "You are doomed."))))

(create-executable 'my-program "/tmp/my-program" :asdf-system "system-containing-my-program")

The keyword :asdf-system is important as CREATE-EXECUTABLE uses this argument to initialize a new lisp machine in order to create the program. The need for a separate process is that the machine specific function equivalent to SAVE-LISP-MACHINE on some lisps actually kills the currently executing process. e.g. SB-EXT:SAVE-LISP-AND-DIE on SBCL.

The building of an executable can also be specified in the ASDF system definition by using the LISP-EXECUTABLE:EXECUTABLE ASDF component.

(eval-when (:compile-toplevel :load-toplevel :execute)
  (asdf:load-system "lisp-executable"))

(defsystem lisp-executable-example
  :author "Mark Cox"
  :serial t
  :components ((:modules "example/"
                         :serial t
                         :components ((:file "main")
                                      (lisp-executable:executable "example-program" :program ("LISP-EXECUTABLE.EXAMPLE" "EXAMPLE-PROGRAM"))))))

The keyword argument :PROGRAM contains the symbol path to the program. From the above example, an executable will be created in the directory "example/" with the name "example-program". When the executable is executed, it will invoke the program LISP-EXECUTABLE-EXAMPLE::EXAMPLE-PROGRAM.

To build the executable, you perform the LISP-EXECUTABLE:CREATE-EXECUTABLES-OP operation on the system.

(asdf:oos 'lisp-executable:create-executables-op "lisp-executable-example")

As mentioned previously, you can specify a function to convert the string found on the command line to its expected type within the program. For convenience, there are some built in conversion functions that use the lisp reader with type checking and a coercion. These are

• CL:NUMBER
• CL:REAL
• CL:FLOAT
• CL:SINGLE-FLOAT
• CL:DOUBLE-FLOAT
• CL:RATIONAL
• CL:INTEGER
• CL:FIXNUM
• CL:UNSIGNED-BYTE
• CL:SIGNED-BYTE
• CL:RATIO

The compound type specifiers for the above can also be specified, for example, the argument HOW-MANY-TIMES should be greater than equal to 0.

(define-program counter (&options (how-many-times how-many-times-value))
  (declare (conversion-function (integer 0) how-many-times)))

Another special built in conversion function is CL:KEYWORD, which converts the string in to a keyword

(define-program file-processor (&options (if-exists if-exists-value))
  (declare (conversion-function keyword if-exists)
           (type (member nil :error :supersede) if-exists-value)))

The lisp-executable library outlined above is tested using the LISP-EXECUTABLE-TESTS system. These tests can be executed by issuing

(asdf:test-system "lisp-executable")  

The tests require the LISP-UNIT unit testing library (version 0.9.1 and above). The version available from QUICKLISP should always be sufficient.

Thank you to the people who develop Solarized CSS and Org mode.

Lastly, big thanks to the Lisp community for all their excellent work.