(archive 'newLISPer)

February 28, 2008

Tufte sparklines in newLISP

Filed under: newLISP — newlisper @ 21:22

That’s a weird headline! At least, it is if you’re not into infographics. If you are, though, you’ll probably already know what I’m going to write about. For the uninitiated, you can read a good introduction at Edward Tufte’s site.

A sparkline is a little word-sized graph that shows a trend or sequence. Because our brains are so good at reading lower-case words, they also find it easy to absorb data trends if the information is represented in a wavy word-sized graphic – a sparkline. For example, this sparkline

shows the changing size of the posts in this blog over time. And this one shows the time between each post since I first started in December 2006. Even though I haven’t labelled the axes or provided additional information, you can easily pick out a message from the data – if there’s a message to pick out. At least, that’s the theory behind sparklines. I’ve been playing around with sparklines for fun, using newLISP. Here are my thoughts so far.

At first I assumed that the best way to draw sparklines would be to generate an SVG sparkline and embed it into the HTML. After all, SVG is a standard vector format using standard XML, and it should display well in all web browsers that follow current standards. Unfortunately, the standards guys have apparently painted themselves into a corner on this one, because XML isn’t compatible with HTML (or XHTML) to the extent that you can just put one inside the other. One of the possible solutions is to change the way pages are served to browser clients, so I decided to try another approach.

Another drawback with the automatic generation of graphs from data is that you don’t get a chance to play with your graph before publishing it. Sparklines, like anything else, can benefit from a little tweaking to make them look just right.

Here’s my first attempt at a rudimentary Sparkline generator running in newLISP-GS:

At the top, there’s the data-entry text box. Anything can be typed in here – non-numeric material is filtered out. This means that you can paste in comma-separated lists as well as plain newLISP-style lists. (This batch of numbers shows the “Coefficients of the ‘3rd order’ mock theta function phi(q)” from the strange and wonderful On-Line Encyclopedia of Integer Sequences.

Next, there’s the sparkline. The width of this sparkline ‘canvas’ changes according to how much data you input. There is, though, a limit to how much data can be plotted; in a 300 pixel wide window there’s room for a 300 element set. That’s probably enough for most sparklines. If you want more, you can edit the script to give you a wider window and canvas – but if you do, all the controls will move…

Below is a plethora of sliders … probably too many. The top one changes the height of the sparkline canvas, so you can squeeze it down to 14 pixels and fit it easily into text. The checkbox at the bottom shows the location of the x-axis.


February 21, 2008


Filed under: newLISP — newlisper @ 16:27

I’ve moved the Lambdapress project to Google Code. Click on the icon at the bottom right to see the project in its new home. So now you can join in!

Google Code is not the easiest of things to get working – I’m not up on this whole svn/code management thing – but it proposes a few advantages that I’m now starting to appreciate. Recently I spent some time looking for a piece of code that I’d deleted some months ago, and if the project had been in a subversion system like Google Code I could have found it more easily.

I’m also hoping that, one day, googling for ‘newLISP’ will find some of this web site. Currently I think I’ve reached page 8 or 9…

February 12, 2008

A newLISP CGI web page in 10 easy steps

Filed under: newLISP — newlisper @ 19:36

I recently needed an odd sort of calculator. It had to work in multiples of 7.5 and 1.5, and add numbers differently according to the day of the week. I was starting to wonder how I could avoid using a Microsoft Excel spreadsheet to do the job; I’m not a big fan of MicroVBasicSoft or whatever it’s called these days. Then I realised that I might be able to do it by writing a newLISP CGI web page instead.

I’m still struggling to understand the basics of CGI and newLISP. Learning CGI isn’t that easy when most of the examples are written in Perl. So I thought this would be good way to get some practice. For the purposes of this post, though, I’ve changed the example to a simple table that adds rows and columns and puts the totals in the last column and bottom row. Let’s see if I can describe it in 10 easy steps.

Step 1: prepare ingredients

This one is easy:

#!/usr/bin/env newlisp
(load {/usr/share/newlisp/modules/cgi.lsp})

I want the cgi.lsp module – part of the standard newLISP install – mainly for the CGI:get function, which retrieves values from the incoming form. I’ve not looked at cgi.lsp much, so I don’t know what’s in it.

Step 2: set up a data store

This step sets up a small data storage area. The data submitted by the user will be loaded into the store for subsequent processing. I use a list of lists rather than an array because:

  1. Lists are more flexible than arrays, and just as quick if they’re not too big.

  2. Er, that’s it, really.

(set 'rows 10 'columns 10)
(set 'data (dup (dup 0 rows) columns))

To access the value of the cell at row 3 and column 4, I can use the implicit addressing technique: (data 3 4).

Step 3: process the data

In step 3, I can write the functions that actually do something with the data. When these functions are called, there will be data in the data store.

; get column n of data
(define (column n a-list)
  (map (fn (f) (nth (f n))) a-list))

(define (sum-rows)
    ; in each row, add cells and put value in final cell
    (for (r 0 (- rows 1))
       (nth-set (data r (- columns 1)) (apply add (chop (data r))))))

(define (sum-columns)
    (for (c 0 (- columns 1))
       (nth-set (data (- rows 1) c) (apply add (chop (column c data))))))

(define (class r c)
      ((= c (- columns 1)) "total")
      ((= r (- rows 1)) "total")
      (true "cell")))

The class function returns a suitable CSS class for a cell given its location. I can use this when building the HTML form to assign different colours to the various input boxes.

Step 4: extract data from the incoming form

This step feels like it’s in the wrong place. I want to extract data from the incoming form, because it contains the numbers that the user has typed into the input boxes. But what form? What boxes? I haven’t created them yet – they won’t get created until step 8. Well, I think it’s confusing!

; extract data from the form
(for (r 0 (- rows 1))
   (for (c 0 (- columns 1))
      (set 'raw (float (CGI:get (string r c))))
      (if (number? raw)
          (nth-set (data r c) raw))))

The data store is filled with numbers.

Step 5: do the calculation

If the user clicks the ‘Calculate’ button to submit the form, a call to CGI:get {recalc} will return true, so the main action part of the script can take place here.

(when (CGI:get {recalc}) (sum-rows) (sum-columns))

I haven’t yet defined that form, though…

Step 6: tidy up

Before the data store’s contents are displayed, I want to change the zeroes to spaces, purely for aesthetic reasons. I’ve used the set-ref-all function (new for version 9.3) to convert them.

(set-ref-all (data 0) "")

Step 7: generate HTML, part 1

The output consists of a header line followed by the HTML page. Most of the HTML is a simple form that uses the POST method to send the completed form to this script.

(print "Content-type: text/html\r\n\r\n")

(print [text]<html>
<head><title>A newLISP calculator</title>
<style>#calc .cell {background-color: #ffeeff; }
#calc .total { background-color: #ddffff; }
<form id="calc" action="index.cgi" method="POST">

Step 8: generate HTML, part 2

I’m in the middle of the form. Instead of generating the form manually, I’ve got newLISP to do it for me. Each input box is named after its row and column, and is assigned a class name for styling. There’s a simple data-fetch function in it too. newLISP’s format function works fine here – originally I used a more complicated template function but this works well.

(for (r 0 (- rows 1))
  (for (c 0 (- columns 1))
    (println (format [text]<input type="text" class="%s" name="%s" size="6" value="%s">[/text] 
        (class r c) (string r c) (string (data r c))) ))

   ; next row
   (println "

Step 9: generate HTML, part 3

The form has a ‘Calculate’ button which sends the ‘recalc’ message to this script – I dealt with the message in step 5 but this is where the message is defined.

    <p><input type="submit" name="recalc" value="Calculate"></p>


Step 10: Serve hot

It looks like I made it in 10 steps. To test the script, start up a newLISP web server and point it at the right place. This is what I do on my Mac:

$ sudo newlisp -http -d 80 -w ~/lisp/cgi-test/

And then visit http://localhost/index.cgi in a browser and try it out. This last step is probably different on other platforms

Any tips or advice from the CGImen or CGIwomen out there will be welcome!

February 9, 2008

Hello newLISP 9.3

Filed under: newLISP — newlisper @ 16:55

This post should have been uploaded some days ago. I’ve been trying to get round to doing it, but life keeps getting in the way…

The latest version of newLISP, 9.3, is now available, and there are some great new features.

The new when function is going to replace if in a lot of the scripts I’ve written. I’m sure a lot of newLISP users have made the following mistake at least once:

(if expression

where they’re expecting that both actions will be carried out if the expression is true. But with if it’s either the first or the second. The syntax of if requires you to construct a block if you want two or more expressions to be evaluated in sequence:

(if expression

But now there’s when, which will make this form easier to read and write:

(when expression

There are also many new powerful techniques for processing items in nested lists, including set-ref-all. For a great introduction to these, see Jeff’s post at his newly relocated artful code site.

I haven’t managed to prepare a major revision of my Introduction to newLISP for this release, but I tried to include examples of most of the new functions I was able to test. It’s difficult to keep up with newLISP’s pace of development sometimes. But it’s much more interesting to describe a moving target than a static one! As each release adds more powerful functions, some of the examples I wrote last year has been rendered obsolete or – in a few cases – inoperative. For example, my example of writing a macro now provides an unnecessary alternative to the built-in when function.

Apparently, *when* was added to the development builds of newLISP is September 2007, but I never noticed it!

Another function that promises to be one of my favourites is find-all. find-all isn’t a new function – it used to work on strings – but now it works with lists, and with lists of lists (or association lists). It can also, when working with lists, take action expressions and comparison functions.

First, here’s the familiar original form of find-all working on a string:

(set 't {There are 60 seconds in a minute, 60 seconds in an hour, and 24 hours in a day.})
(find-all {\d+} t)
;-> ("60" "60" "24")

Notice that find-all in this form takes regular expressions without you having to tell it to.

Another form of find-all lets you search a list of lists for a list that matches a pattern. You can specify the pattern with wildcard characters. For example, here’s a list of lists:

(set 'symphonies 
  '((Beethoven 9)
    (Haydn 104)
    (Mozart 41)
    (Mahler 10)
    (Wagner 1)
    (Schumann 4)
    (Shostakovich 15)
    (Bruckner 9)))

To find all the lists that end with 9, use the match pattern ‘(? 9), where the question mark matches a single item:

(find-all '(? 9) symphonies)
;-> ((Beethoven 9) (Bruckner 9))

You can also use this form with an ‘action expression’ at the end, after the list:

(find-all '(? 9) symphonies (println (first $0) { wrote 9 symphonies.}))
Beethoven wrote 9 symphonies.
Bruckner wrote 9 symphonies.

Here, the action expression uses $0 to refer to each matched element in turn. If you’ve used replace in newLISP, you’ll be familiar with this idea.

There’s a third form. For searching a plain list, rather than a list of lists, the syntax is different. Supply four arguments: the search key, the list, an action expression, and the functor, which is the comparison function you want to use for matching the search key. This time, don’t use match-style patterns:

(set 'food '("bread" "cheese" "onion" "pickle" "lettuce"))
(find-all "onion" food (print $0 { }) >)
bread cheese lettuce

Here, find-all is looking for the string “onion” in the list food. It’s using the > function as a comparison function, so it will find anything that “onion” is greater than. For strings, ‘greater than’ means appearing later in the default ASCII sorting order, so that “cheese” is greater than “bread” but less than “onion”. Notice that, unlike other functions that let you provide comparison functions (namely find, ref, ref-all, ref-set, replace when used with lists, set-ref, set-ref-all, and sort), the comparison function must be supplied.

With the < function, the result is a list of things that “onion” is less than:

(find-all "onion" food (print $0 { }) <)

With different comparison functions, and the ability to use an action expression for each ‘successful’ comparison, you can do quite a lot with a single find-all.

Choose a word:

(set 'word "moonstarer")

Select a dictionary. On Unix it’s probably in /usr/share/dict/words; Windows users can probably find something suitable here.

(set 'dictionary (parse (read-file {/usr/share/dict/words})))

Define a test for the anagram-ness of two words:

(define (anagrams? w1 w2)
  (= (sort (explode w1)) (sort (explode w2))))

Then use find-all to look for anagrams of the word:

(find-all word dictionary (println $0) anagrams?)


newLISP does all the hard work for you. Notice how the comparison function receives two arguments automatically, one from the list, and the other the contents of *word*. Come to think of it, I don’t need to continually explode and sort the first word, do I?. I’ll leave you to improve it!

Create a free website or blog at WordPress.com.