(archive 'newLISPer)

January 20, 2006

Bayesian functions in the latest newLISP

Filed under: newLISP — newlisper @ 17:02
Tags:

>

The latest development version of newLISP contains two new Bayesian statistical functions: bayes-train analyses datasets for word frequencies, saving the results in a newLISP context, and bayes-query uses this context to generate the probability of a given piece of text belonging to one or other of the datasets. Here’s a version of the example in the manual, which estimates the probability that a given piece of text can be considered as spam – a typical use for Bayesian analysis tools.

First, we obtain two sets of data, one good, one bad: I dumped a dozen or so desirable and undesirable email messages into a pair of text files. parse converts each file into a list of symbols:

(set 'spam-data (parse (read-file "/Users/me/spam.txt") {\s+} 0))
(set 'nospam-data (parse (read-file "/Users/me/nospam.txt") {\s+} 0))

Then we use the bayes-train function to produce a context containing all the words in these lists, together with their frequency data:

(bayes-train spam-data nospam-data 'Lexicon)

Next, we can save the resulting context in a text file:

(save "lex.lsp" 'Lexicon)

so that we can load it again later:

(load "lex.lsp")

The training process takes a few seconds, so it makes sense to do it once, and then load the context when you want to analyse some text.

With the context loaded, we can use the bayes-query function to analyse a piece of text against the previously-analysed data:

(set 'q1 (bayes-query (parse "newLISP is fine open source software") Lexicon))
(set 'q2 (bayes-query (parse "Office XP is cheap at the moment") Lexicon))

Each result is a pair of numbers in a list: the first number is the probability that the phrase belongs in the first dataset (and is therefore, in this case, not spam), and the second number the probability that it belongs in the second, spam dataset. The two numbers add up to 1.

(println (format "%5f" (first q1)))
(println (format "%5f" (first q2)))

with the following results:

0.000090
0.999998

So the phrase “newLISP is fine open source software” scores a tiny 0.000090 out of 1, and so is not considered to be similar to the spam text I used for training, whereas the phrase “Office XP is cheap at the moment” certainly scores like some of the spam email I receive – I receive dozens of messages like this each month, so the statistics have clearly produced the right results this time.

Consult the newLISP manual for all the options and formulas.

I discovered these functions just after writing my previous entry about analysing novels. If I use two of the novels for training, I can find out the probability that a piece of text was written by one or other of the authors. Plainly newLISP is an excellent choice for this sort of activity!

Advertisements

3 Comments »

  1. >The new ‘bayes-train’ and ‘bayes-query’ are not yet in the offical release but in the current development version 8.7.8. Also when doing really big files its a lot faster to parse and train tokens line by line instead of all at once. Its the parseing/tokenizing which is faster doing it in small portions versus one big chunk. The next development release 8.7.8 will have improved documentation and examples how to do this. Meanwhile I post the improved Bayes documentation here:http://newlisp.org/downloads/development/bayesian.html

    Comment by Lutz — January 20, 2006 @ 19:31 | Reply

  2. >The link was cut of, here the link in two lines:http://www.newlisp.org/downloads/ development/bayesian.html

    Comment by Lutz — January 20, 2006 @ 19:33 | Reply

  3. >Thanks Lutz – Yes, I know that these are very new, and development version only at the moment, but I thought it worth mentioning since I’d only just posted something v similar and they looked pretty cool…By the time anyone else reads this post, it will probably be newLISP version 9! ;-)

    Comment by newlisper — January 20, 2006 @ 20:52 | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: