(archive 'newLISPer)

February 17, 2007

newLISP in ConTeXt

Filed under: newLISP — newlisper @ 13:09
Tags:

I’ve been spending some time recently updating the Introduction to newLISP, which is my attempt at producing a beginner’s guide to the language – the document I wanted to read but couldn’t find when I started using newLISP. Most of the work this time has been behind the scenes, rather than on the pages, and I haven’t spent much time covering the new functionality in the 9.1 release, which has just been released.

For this revision, I’ve produced the document using a publishing system called ConTeXt. While a discussion about the merits of ConTeXt is probably not interesting or relevant here, I think it’s worth mentioning one cool thing: it’s possible to include newLISP code in a document, and have that code executed when the PDF document is generated.

This is how it works.

When you want to evaluate some newLISP code, you put it between two tags in your source document, like this:

\startNLE
(println (date (date-value) 0 “%Y-%m-%d %H:%M:%S“))
\stopNLE

These peculiar backslash-prefixed commands will look all too familiar to all ConTeXt (and TeX) users – this is the standard way of indicating document structure and attributes. When the document is ‘printed’ to PDF, ConTeXt processes the code between the \startNLE and \stopNLE tags, and includes the output in the document.

To make this happen, you need to define a module that tells ConTeXt what to do when it sees these tags. The module looks like this:

\startmodule[NLprocess]
\unprotect
\def\NLbufferprefix{nl-}
\newcounter\nofNLfiles
\def\NLfile{%
\TEXbufferfile{\NLbufferprefix\nofNLfiles}}
\def\startNLE
{\doglobal\increment\nofNLfiles\dostartbuffer
[\NLbufferprefix\nofNLfiles][startNLE][stopNLE]}
\def\stopNLE
{\doifmode{*\v!first}\runNLE
\input \NLfile.out
}
\def\runNLE
{\executesystemcommand{./process-context-eval.lsp \NLfile}}
\protect \doifnotmode{demo}{\endinput}
\stopmodule

Aaaargh! This lives in a file called m-nl.tex. It’s ConTeXt (or TEX) code, rather than newLISP, of course. I don’t understand it at all, but it seems to be writing a buffer to a numbered file, calling an external command, and then finally reading in the contents of the resulting .out file.

This module needs to be referenced at the start of your source document, like this:

\usemodule[nl]
\def\NL{NL}

As for the process-context-eval.lsp file, this is the newLISP file that processes the code passed to it, and sends its output to standard output, which is stored to the .out file.

#!/usr/bin/newlisp
(device (open (string ((main-args) 2) “.out“) “write“))
(set 'c (string (eval-string s true)))
(exit)

Once all this has been set up (and you’ve got to enable ‘write18’ support, which gives ConTeXt permission to run external code) it’s easy to include and execute newLISP code in your documents.

There are a few things to remember. The most important is that quite a few characters, including at least the following:

{ } \ $# ! | @ ^ % / < > ~ & ? _ ’ are not accepted directly by ConTeXt, and have to be ‘escaped’ in some way. Since there are quite a few, it’s probably a good idea to write some code that replaces all of them with acceptable alternatives, rather than you having to do it for each print statement: (set 'uuid1 (uuid)) (set 'uuid2 (uuid)) (replace “{“ raw-string uuid1) (replace “}“ raw-string uuid2) (replace {} raw-string {\letterbackslash{}}) (replace {$} raw-string {\letterdollar{}} )
(replace {#} raw-string {\letterhash{}})
(replace {!} raw-string {\letterexclamationmark{}} )
(replace {|} raw-string {\letterbar{}})
(replace {@} raw-string {\letterat{}})
(replace {^} raw-string {\letterhat{}})
(replace “%“ raw-string {\letterpercent{}} )
(replace “/“ raw-string {\letterslash{}} )
(replace “<“ raw-string {\letterless{}} )
(replace “>“ raw-string {\lettermore{}} )
(replace “~“ raw-string {\lettertilde{}} )
(replace “&“ raw-string {\letterampersand{}})
(replace “?“ raw-string {\letterquestionmark{}})
(replace “_“ raw-string {\letterunderscore{}})
(replace “’“ raw-string {\lettersinglequote{}})
(replace uuid1 raw-string {\letteropenbrace{}})
(replace uuid2 raw-string {\letterclosebrace{}})

The uuid function generates a string that is guaranteed not to exist, and which replaces a left or right brace. I couldn’t think of a better way to do this.

One obvious use for this feature is to insert things like date stamps and version numbers. But, more useful for the writer, is that you can generate information programmatically rather than manually. For one thing, the information can then be guaranteed to be up to date. A list of reserved symbols, for example, or the contents of a particular file or URL – these would always be the current versions, automatically included. (Programmers have had this for years; few writing tools allow for this.)

I quite like the idea of generating some random information for inclusion into the final document. Perhaps those quotes that some authors love to include at the start of each chapter could be extracted?

It’s also intriguing to consider using this technique for processing code examples: each example could be checked, evaluated, and the output formatted and included after the code, which could also be pretty-printed and syntax-coloured. Unfortunately, a surprisingly high proportion of the examples in my document wouldn’t evaluate: some continue from previous examples, and others assume that certain files or symbols already exist. And occasionally you want to write code that doesn’t work – to illustrate how not to do something.

Or perhaps we could generate some of the text itself. I’ve looked through this list but haven’t found any way of generating coherent generic explanatory material programmatically. Looks like I’ll have to write it the old-fashioned way…!