I wanted to try meta-sexp package. It's a PEG-like parser library. Now, it has a method create-parser-context which can only take string or string-stream. (This is where the parsed content comes from.) However, open and friends create file-stream.
I've looked into flexi-streams library, but couldn't find a straight-forwad way to do it (I could use octets-to-string to create a buffer, feeding it to a bidirectional stream etc... but it's a lot of plumbing). I feel like I'm missing something obvious, and it should be as easy as calling some flexi-stream wrapper function.
I think that you do need to work with the way of transforming streams from the system meta-sexp readme:
meta-sexp uses in-memory string vectors, instead of commonly used
streams, for efficiently stepping backward and forward through the
input.
So meta-sexp works normally with characters and strings, it has implemented some type checkers:
CL-USER> (ql:quickload :meta-sexp)
To load "meta-sexp":
Install 1 Quicklisp release:
meta-sexp
Downloading http://beta.quicklisp.org/archive/meta-sexp/2010-10-06/meta-sexp-0.1.6.tgz
##########################################################################
; Loading "meta-sexp"
[package meta-sexp]......
(:META-SEXP)
CL-USER> (use-package :meta-sexp)
T
CL-USER> (alpha? #\c)
T
CL-USER> (alpha? #\3)
NIL
CL-USER> (digit? #\3)
3 (2 bits, #x3, #o3, #b11)
Then you can use this defining rules to work with strings like the example in the readme:
CL-USER> (defrule integer? (&aux (sign 1) d (num 0)) ()
(:? (:or (:and "-" (:assign sign -1))
"+"))
(:+ (:assign d (:type digit?))
(:assign num (+ (* num 10)
(- (char-code d) #.(char-code #\0)))))
(:return (* sign num)))
CL-USER> (defrule integer? (&aux (sign 1) d (num 0)) ()
(:? (:or (:and "-" (:assign sign -1))
"+"))
(:+ (:assign d (:type digit?))
(:assign num (+ (* num 10)
(- (char-code d) #.(char-code #\0)))))
(:return (* sign num)))
now let's suppose that we have this file
parser-test.txt:
patata
123
calimero
34
32
we can read as follows (read-line returns a string):
CL-USER> (with-open-file (stream "parser-test.txt")
(loop for line = (read-line stream nil)
while line
do (print line)))
"patata"
"123"
"calimero"
"34"
"32"
NIL
then apply the parser:
CL-USER>
; No values
CL-USER> (with-open-file (stream "parser-test.txt")
(loop for line = (read-line stream nil)
while line
do (print (integer? (create-parser-context line)))))
NIL
123
NIL
34
32
NIL
you can read files, by character, by line ..., take a look at hypersec function read; finally I would recommend this nice post with a big lesson of lisp Lisp: It's Not About Macros, It's About Read
Related
I'm on Day 3 of trying to learn Common Lisp (SBCL) by doing Advent of Code. I understand there is more than one type of return. I'm wondering if someone could explain to me why the following function will return nil (this makes sense)
(defun fn (n)
(cond ((zerop n)
(return-from fn nil))
(t
(write-line "Hello World")
(fn (- n 1)))))
but the following function will return "Hello World" (this does not make sense to me).
(defun fn (n)
(cond ((zerop n)
(return-from fn nil))
(t
(fn (- n 1))
(write-line "Hello World"))))
I found a great post covering some aspects of SBCL's return behaviour here, but to my understanding it doesn't seem to address this particular detail.
EDIT: a loop call is a more sensible way of writing this function, but it is not the way that I discovered this behaviour. My suspicion is that this behaviour arises because fn is called recursively.
(I started writing this before Sylwester's answer, which is mostly better I think.)
A critical difference between Lisp-family languages and many other languages is that Lisp-family languages are 'expression languages'. What this means technically is that languages like (say) C or Python there are two sorts of constructs:
expressions, which have values;
statements, which do not;
While in Lisp-family languages there is one sort of thing: expressions, which have values. Lisp-family languages are sometimes called 'expression languages' as a result of this.
This makes a huge difference if you want to write functions, which are things which return values (a function call is an expression in other words).
Conventional languages (Python as an example)
In a language which is not an expression language then if you're defining a function and find yourself in the middle of some construct which is a statement and you want to return a value, you have to use some special magic construct, often called return to do that. So in Python, where conditionals are statements, you might write:
def fib(n):
if n < 2:
return n
else:
return fib(n - 1) + fib(n - 2)
And in fact you have to use return because the body of a function definition in Python is a series of statements, so to return any kind of value at all you need to use return.
In fact, Python (and C, and Java &c &c) have a special form of a conditional which is an expression: in Python this looks like this:
def fib(n):
return n if n < 2 else (fib(n - 1) + fib(n - 2)
It looks different in C but it does the same thing.
But you still need this annoying return (OK, only one of them now) & that brings to light another feature of such languages: if some place in the syntax wants a statement you generally need to have a statement there, or if you can put an expression there its value just gets dropped. So you can try something like this:
def fib(n):
n if n < 2 else (fib(n - 1) + fib(n - 2)
And that's syntactically OK -- the expression gets turned into a statement -- but it fails at runtime because the function no longer returns a useful value. In Python you can get around this if you want people to hate you:
fib = lambda n: n if n < 2 else fib(n - 1) + fib(n - 2)
Python people will hate you if you do this, and it's also not useful, because Python's lambda only takes expressions so what you can write is crippled.
Lisp
Lisp has none of this: in Lisp everything is an expression and therefore everything has a value, you just need to know where it comes from. There is still return (in CL, anyway) but you need to use it much less often.
But, of course people often do want to write programs which look like 'do this, then do this, then do this', where most of the doing is being done for side-effect, so Lisps generally have some kind of sequencing construct, which lets you just have a bunch of expressions one after the other, all but (typically) one of which get evaluated for side-effect. In CL the most common sequencing construct is called progn (for historical reasons). (progn ...) is an expression made of other expressions, and its value is the value of last expression in its body.
progn is so useful in fact that a bunch of other constructs have 'implicit progns' in them. Two examples are function definitions (the body of defun is an implicit progn) and cond (the body of a cond-clause is an implicit `progn).
Your function
Here is your function (first version) with its various parts notated
(defun fn (n)
;; the body of fn is an implicit progn with one expression, so
;; do this and return its value
(cond
;; the value of cond is the value of the selected clause, or nil
((zerop n)
;; the body of this cond clause is an implicit progn with on
;; expression so do this and ... it never returns
(return-from fn nil))
(t
;; the body of this cond clause is an implicit progn with two expressions, so
;; do this for side-effect
(write-line "Hello World")
;; then do this and return its value
(fn (- n 1)))))
Here is the second version
(defun fn (n)
;; the body of fn is an implicit progn with one expression, so
;; do this and return its value
(cond
;; the value of cond is the value of the selected clause, or nil
((zerop n)
;; the body of this cond clause is an implicit progn with on
;; expression so do this and ... it never returns
(return-from fn nil))
(t
;; the body of this cond clause is an implicit progn with two expressions, so
;; do this for side-effect
(fn (- n 1))
;; then do this and return its value
(write-line "Hello World"))))
So you can see what is happening here: in the first version the value that gets returned is either nil or the value of the recursive call (also nil). In the second version the value that gets returned is either nil or whatever write-line returns. And it turns out that write-line returns the value of its argument, so that's what you get if you call it with an integer greater than zero.
Why have return-from at all in Lisp?
One thing that should be immediately clear from this whole expression-language thing is that you hardly ever need to explicitly return something in Lisp: you just have an expression that computes the value you want. But there are two good uses (which perhaps are really the same use) of explicit returns.
The first is that sometimes you are doing some big search for something in the form of a bunch of nested loops and at some point you just want to say 'OK, found it, here's the answer'. You can do that in one of two ways: you can carefully structure your loops so that once you find what you're after they all terminate nicely and the value gets passed back up, or you can just say 'here's the answer'. The latter thing is what return-from does: it just says 'I'm done now, unwind the stack carefully and return this':
(defun big-complicated-search (l m n)
(dotimes (i l)
(dotimes (j m)
(dotimes (k n)
(let ((it (something-involving i j k l m n)))
(when (interesting-p it)
(return-from big-complicated-search it)))))))
And return-from does this in the right way:
(defun big-complicated-file-search (file1 file2)
(with-open-file (f1 file1)
(with-open-file (f2 file2)
...
(when ...
(return-from big-complicated-search found)))))
When you call this, and when the thing is found, return-from will make sure that the two files you have opened are properly closed.
The second, which is really almost the same thing, is that sometimes you need to just give up, and return-from is a good way of doing this: it returns immediately, deals with clean-ups (see above) and is generally a nice way of saying 'OK, I give up now'. At first blush this seems like something you would do with some kind of exception-handling system, but in fact there are two critical differences:
in an exception-handling system (which CL has, of course), you need some kind of exception to raise so you might need to invent something;
exception-handling systems are dynamic not lexical: if you raise an exception then the thing that gets to handle it is hunted for up the stack dynamically: this means that you're at the mercy of anyone who stuck a handler in the way and it also typically rather slow.
Finally the exceptional-return-via-error-handling-mechanism is just, well, horrid.
Your code:
(defun fn (n)
(cond ((zerop n) (return-from fn nil))
(t (write-line "Hello World") (fn (- n 1)))
)
)
There is a bunch of things slightly wrong with above code:
(defun fn (n)
(cond ((zerop n) (return-from fn nil)) ; 1) the return from is not needed
(t (write-line "Hello World") (fn (- n 1))) ; 2) this line is not correctly
; indented
) ; 3) dangling parentheses Don't. Never.
; also: incorrect indentation
)
the first cond clause already returns a value, just write nil as the return value. Then the whole cond returns this value. It is very rare that you need return or return-from from a cond clause.
use an editor to indent your code. In GNU Emacs / SLIME the command control-meta-q will indent the expression. For help about the editor commands in the current mode see: control-h m for mode help.
indent correctly and don't use dangling parentheses. They are useless in Lisp. Learn to use the editor to correctly indent code - that's much more useful than to place wrongly indented parentheses on their own line. tab indents the current line.
It's more useful to format the code like this for a beginner:
(defun fn (n)
(cond ((zerop n)
(return-from fn nil))
(t
(write-line "Hello World")
(fn (- n 1)))))
Code then will look more like a prefix tree.
Also don't forget to disable inserting tabs in GNU Emacs Put this into your emacs init file: (setq-default indent-tabs-mode nil). You can evaluate Emacs Lisp expressions also on the fly with meta>-:.
Now according to 1. above code is usually written as:
(defun fn (n)
(cond ((zerop n)
nil)
(t
(write-line "Hello World")
(fn (- n 1)))))
When n is zero, the first clause is selected and its last value is returned. The other clauses are not looked at -> cond returns nil -> the function fn returns nil.
Usually I would write above recursive function like this:
(defun fn (n)
(unless (zerop n)
(write-line "Hello World")
(fn (- n 1))))
unless returns nil if (zerop n) is true. Another variant:
(defun fn (n)
(when (plusp n)
(write-line "Hello World")
(fn (- n 1))))
You CAN use return-from, but in case it was not clear: you don't need it most of the time.
Unlike C language family Lisp has the feature that everything is expressions. That means you "return" the result of an expression. eg.
(+ (if (< x 0)
(- x)
x)
3)
Here the result of the if is that it will be the absolute value of x. Thus if x is -5 or 5 the result of the expression is 8. You can write abs like this:
(defun my-abs (v)
(if (< v 0)
(- v)
v))
Notice I do not use return. THe result of the if is the last expression and that means the result of that is the result of my-abs.
Your two functions can be written like this:
(defun fn1 (n)
(cond
((zerop n) nil)
(t (write-line "Hello World") (fn1 (- n 1)))))
And
(defun fn2 (n)
(cond
((zerop n) nil)
(t (fn2 (- n 1)) (write-line "Hello World"))))
Needless to say (write-line "Hello World") returns its argument in addition to print the argument. Thus whenever it is the last expression it will be the result.
For every n above 0 it will do the recursions first and each end every one except the first will return "Hello World". If you call (fn2 0) the result is nil, the same as fn1.
EDIT
One might ask what is the purpose of return and return-from when there obviously is little use for it. If you want something else than the default result in a loop macro the common way to do it by finally clause.
(defun split-by (test list &key (return-form #'values))
"Split a list in two groups based on test"
(loop :for e :in list
:if (funcall test e)
:collect e :into alist
:else
:collect e :into blist
:finally (return (funcall return-form alist blist))))
(split-by #'oddp '(1 2 3 4) :return-form #'list)
; ==> ((1 3) (2 4))
Another way is if you are doing recursion and want to cancel everything when you know the result you can use return-from:
(defun find-tree-p (needle haystack &key (test #'eql))
"search the tree for element using :test as comparison"
(labels ((helper (tree)
(cond ((funcall test tree needle)
(return-from find-tree t))
((consp tree)
(helper (car tree))
(helper (cdr tree)))
(t nil))))
(helper haystack)))
(find-tree '(f g) '(a b c (d e (f g) q) 1 2 3) :test #'equal)
; ==> (f g) ; t
Now if you hadn't done return-from you would have had logic to check the returned value to see if you needed to continue or not. If you want to process elements and don't want to pass twice to check validity before computing the result, you can just start computing and use return-from as a call/cc. This function can be used to map over lists of lists and it stops at the shortest list so needs to become () when the first sublist is empty:
(defun cdrs (lists)
"return the cdrs if all elements are cons, () otherwise"
(loop :for list :in lists
:when (null list) :do (return-from cdrs '())
:collect (cdr list)))
(cdrs '((a) (b) (c))) ; ==> (nil nil nil)
(cdrs '((a) (b) ())) ; ==> ()
I wrote some function that could replace the function read of common lisp
(defun my-read (stream &rest args)
(declare (ignore args))
(funcall (my-get-macro-character (read-char stream))))
Is there a way to use this function as default reader?
You can't redefine the built in functions1, but you can define a package that shadows cl:read and defines a new function my:read, so that when you use that package, it looks like it's the default read function. E.g., something like this:
CL-USER> (defpackage #:my-package
(:use "COMMON-LISP")
(:shadow #:read)
(:export #:read))
;=> #<PACKAGE "MY-PACKAGE">
CL-USER> (defun my-package:read (&rest args)
(declare (ignore args))
42)
;=> MY-PACKAGE:READ
CL-USER> (defpackage #:another-package
(:use #:my-package "COMMON-LISP")
(:shadowing-import-from #:my-package #:read))
;=> #<PACKAGE "ANOTHER-PACKAGE">
CL-USER> (in-package #:another-package)
;=> #<PACKAGE "ANOTHER-PACKAGE">
ANOTHER-PACKAGE> (read)
;=> 42
Actually, as Rainer Joswig noted in the comments, even though it's undefined behavior (see 11.1.2.1.2 Constraints on the COMMON-LISP Package for Conforming Programs), there often are ways to redefine some of the Common Lisp function, For instance, in SBCL you can use unlock-package, as shown in redefining built-in function. CLISP has package locks. Other implementations may have similar functionality.
One approach is to use set-macro-character on all "valid" input characters in a readtable. (This is okay if you only accept ASCII input, but I don't know if it would be practical for full Unicode.)
Something like this:
(defun replace-default-read-behavior (rt fn)
(loop for c across
" !\"#$%&'()*+,-./0123456789:;<=>?#ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~"
do (set-macro-character c fn t rt)))
(defun my-parser (stream char)
(format t "custom read: ~A~A" char (read-line stream)))
(defun my-get-macro-character (char)
(declare (ignore char))
#'my-parser)
(defun my-read (stream char)
(funcall (my-get-macro-character char) stream char))
(defvar *my-readtable* (copy-readtable ()))
(replace-default-read-behavior *my-readtable* #'my-read)
(let ((*readtable* *my-readtable*))
(read-from-string "foo"))
custom read: foo ; printed
NIL ; returned
3
Does Racket use memoization when computing large amounts of numbers from an infinite stream? So, for example, if I printed out (aka, computed and displayed) the first 400 numbers on the infinite stream of integers:
(1 2 3 ... 399 400)
And right after I asked to print the first 500 numbers on this infinite stream. Would this second set of computations use memoization? So the first 400 numbers would not be computed again?
Or does this functionality need to be coded by the user/obtained from libraries?
The built-in racket/stream library uses lazy evaluation and memoization to draw elements from a stream:
(require racket/stream)
(define (print-and-return x)
(displayln "drawing element...")
x)
(define (in-range-stream n m)
(if (= n m)
empty-stream
(stream-cons (print-and-return n) (in-range-stream (add1 n) m))))
(define s (in-range-stream 5 10))
(stream-first s)
(stream-first s)
(stream-first (stream-rest s))
The expressions passed to stream-cons are not evaluated until requested either with stream-first or stream-rest. Once evaluated, they are memoized. Notice that despite the four stream operations performed on s, only two `"drawing element..." messages are displayed.
You can use the memoize package.
GitHub source: https://github.com/jbclements/memoize/tree/master
raco pkg install memoize
Using it is as simple as replacing define with define/memo. To quote its example:
(define (fib n)
(if (<= n 1) 1 (+ (fib (- n 1)) (fib (- n 2)))))
> (time (fib 35))
cpu time: 513 real time: 522 gc time: 0
14930352
> (define/memo (fib n)
(if (<= n 1) 1 (+ (fib (- n 1)) (fib (- n 2)))))
> (time (fib 35))
cpu time: 0 real time: 0 gc time: 0
14930352
Also, it is generally quite easy to implement memoization yourself using a Racket hash-table.
For other Scheme implementations that use SRFI 41 streams, those streams also fully memoise all the materialised elements.
In fact, in my Guile port of SRFI 41 (which has been in Guile since 2.0.9), the default printer for streams will print out all the elements so materialised (and nothing that isn't):
scheme#(guile-user)> ,use (srfi srfi-41)
scheme#(guile-user)> (define str (stream-from 0))
scheme#(guile-user)> (stream-ref str 4)
$1 = 4
scheme#(guile-user)> str
$2 = #<stream ? ? ? ? 4 ...>
Any of the elements that aren't being printed out as ? or ... have already been memoised and won't be recomputed. (If you're curious about how to implement such a printer, here's the Guile version.)
Well, I am trying to get some things in racket, I am currently studying streams. I try to create a function that will edit a stream, for example add a pair of (int, element) in place of each stream's element.
For example initial_stream : <1,2,3, …>
edited_stream : <(int . 1) (int . 2) (int . 3) ….. >
I wrote this but it seems to enter an endless loop (with int=>13)
(define (stream-add-zero s)
(cons (cons 13 (car (s))) (stream-add-zero (cdr (s)))))
Thanks in advance.
If by "stream" you mean Racket's lazy stream data structure, this can be done with an application of stream-map.
(define initial-stream (in-naturals))
(define edited-stream (stream-map (λ (i) (cons 'int i)) initial-stream))
;; sanity check
(require rackunit)
(check-equal? (stream-ref edited-stream 3) '(int . 3))
This is assuming the int in your edited_stream was a symbol.
Can anybody explain an example in Paul Graham's ANSI Common Lisp page 110?
The example try to explain the use &rest and lambda to create functional programming facilities. One of them is a function to compose functional arguments. I cannot find anything explaining how it worked. The code is as follows:
(defun compose (&rest fns)
(destructuring-bind (fn1 . rest) (reverse fns)
#'(lambda (&rest args)
(reduce #'(lambda (v f) (funcall f v))
rest
:initial-value (apply fn1 args)))))
The usage is:
(mapcar (compose #'list #'round #'sqrt)
'(4 9 16 25))
The output is:
((2) (3) (4) (5))
Line 2 and 6 look especially like magic to me.
The compose function returns a closure that calls each of the functions from last to first, passing on the result of each function call to the next.
The closure resulting from calling (compose #'list #'round #'sqrt) first calculates the square root of its argument, rounds the result to the nearest integer, then creates a list of the result. Calling the closure with say 3 as argument is equivalent to evaluating (list (round (sqrt 3))).
The destructuring-bind evaluates the (reverse fns) expression to get the arguments of compose in reverse order, and binds its first item of the resulting list to the fn1 local variable and the rest of the resulting list to the rest local variable. Hence fn1 holds the last item of fns, #'sqrt.
The reduce calls each the fns functions with the accumulated result. The :initial-value (apply fn1 args) provides the initial value to the reduce function and supports calling the closure with multiple arguments. Without the requirement of multiple arguments, compose can be simplified to:
(defun compose (&rest fns)
#'(lambda (arg)
(reduce #'(lambda (v f) (funcall f v))
(reverse fns)
:initial-value arg)))
destructuring-bind combines destructors with binding. A destructor is a function that lets you access a part of a data structure. car and cdr are simple destructors to extract the head and tail of a list. getf is a general destructor framework. Binding is most commonly performed by let. In this example, fns is (#'list #'round #'sqrt) (the arguments to compose), so (reverse fns) is (#'sqrt #'round #'list). Then
(destructuring-bind (fn1 . rest) '(#'sqrt #'round #'list)
...)
is equivalent to
(let ((tmp '(#'sqrt #'round #'list)))
(let ((fn1 (car tmp))
(rest (cdr tmp)))
...))
except that it doesn't bind tmp, of course. The idea of destructuring-bind is that it's a pattern matching construct: its first argument is a pattern that the data must match, and symbols in the pattern are bound to the corresponding pieces of the data.
So now fn1 is #'sqrt and rest is (#'round #'list). The compose function returns a function: (lambda (&rest args) ...). Now consider what happens when you apply that function to some argument such as 4. The lambda can be applied, yielding
(reduce #'(lambda (v f) (funcall f v))
'(#'round #'list)
:initial-value (apply #'sqrt 4)))
The apply function applies fn1 to the argument; since this argument is not a list, this is just (#'sqrt 4) which is 2. In other words, we have
(reduce #'(lambda (v f) (funcall f v))
'(#'round #'list)
:initial-value 2)
Now the reduce function does its job, which is to apply #'(lambda (v f) (funcall f v)) successively to the #'round and to #'list, starting with 2. This is equivalent to
(funcall #'list (funcall #'round 2))
→ (#'list (#'round 2))
→ '(2)
Okay, here goes:
It takes the functions given, reverses it (in your example, it becomes (#'sqrt #'round #'list)), then sticks the first item into fn1, and the rest into rest. We have: fn1 = #'sqrt, and rest = (#'round #'list).
Then it performs a fold, using (apply sqrt args) (where args are the values given to the resulting lambda) as the initial value, and with each iteration grabbing the next function from rest to call.
For the first iteration you end up with (round (apply sqrt args)), and the second iteration you end up with (list (round (apply sqrt args))).
Interestingly, only the initial function (sqrt in your case) is allowed to take multiple arguments. The rest of the functions are called with single arguments only, even if any particular function in the chain does a multiple-value return.
This example stumped me for a day. I could finally understand it by renaming some of the arguments and commenting each line before it made sense. Below is what helped me explain it to myself.
In the book example using the call:
(mapcar (compose #'list #'round #'sqrt) '(4 9 16 25))
The parameter functions becomes (#'LIST #'ROUND #'SQRT)
(defun compose (&rest functions)
(destructuring-bind (fx . fxs) (reverse functions)
;; fx becomes #'SQRT
;; fxs becomes '(#'ROUND #'LIST)
#'(lambda (&rest args) ; This is the function returned as result.
;; The args parameter will be (4) on the mapcar's first
;; iteration on the (4 9 16 25) list passed in the call:
;; (mapcar #'(compose #'List #'round #'sqrt) '(4 9 16 25)) => ((2) (3) (4) (5))
;; or e.g. the (4) in (funcall (compose #'list #'sqrt '(4)) => (2.0)
;; Note that args is not ((#'ROUND #'LIST)).
(reduce #'(lambda (x y) (funcall y x))
;; fxs is (#'ROUND #'LIST) - captuted as closure since it is now
;; locally unbound.
fxs
;; Initial value is: (apply #'SQRT '(4) => 2.0.
;; In Paul Graham's example, the mapcar passes
;; each square number individually.
;; The reverse order of parameters in the second lambda
;; first invokes: (ROUND 2.0) => 2
;; and then invokes: (LIST 2) => (2)
:initial-value (apply fx args)))))