Common Lisp output file streams SBCL - stream

I am on SBCL on debian.
For some reason if I use this:
(with-open-file (output (open #p"file.txt"
:direction :output
:if-exists :overwrite))
(format output "test")))
Where file.txt is a plain text file.
I get the error
#<SB-SYS:FD-STREAM for "file /home/me/file.txt" {1004A90813}> is not
a character output stream.
Even using :element-type 'character doesn't save me. I haven't been able to get any output stream opened by any method. If I try to use write-bit it says that it isn't a binary output stream. No other write functions work either, such as write-sequence or write-line. They all return this error. How do I fix this?

I've made the important points bold. The problem is actually more tricky then one might think:
Let's look at the form.
First mistake: it's not indented correctly. Let's indent:
(with-open-file (output (open #p"file.txt"
:direction :output
:if-exists :overwrite))
(format output "test")))
Now we can see more mistakes. An additional parentheses
(with-open-file (output (open #p"file.txt"
:direction :output
:if-exists :overwrite))
(format output "test"))) ; <- additional parenthesis
But more important:
(open #p"file.txt"
:direction :output
:if-exists :overwrite)
Above opens a file for writing output and returns a stream.
WITH-OPEN-FILE does also open a file. So you try to open the file TWICE, first for writing..
(with-open-file (output stream)
(format output "test")))
Above opens a file for reading. You have opened the file twice: first for writing, then for reading.
Now you try to write with FORMAT to an input stream.
The slightly surprising part is this: both open and with-open-file can take a file stream as a file spec. If it gets a file stream as a file spec, then the associated pathname is used for the open operation.
So, as mentioned in another answer, this would be more correct:
(with-open-file (output #p"file.txt"
:direction :output
:if-exists :supersede)
(format output "Hello"))
SBCL error message:
#<SB-SYS:FD-STREAM for "file /home/me/file.txt" {1004A90813}>
is not a character output stream.
The point of the error message here is not that the stream is not a character stream. It's not an output stream. The stream actually is a character input stream! Thus calling FORMAT using the stream won't work. Let's write an assert to verify this:
CL-USER 18 > (with-open-file (output (open #p"/tmp/file.txt"
:direction :output
:if-does-not-exist :create
:if-exists :overwrite))
(assert (output-stream-p output) (output)
"The stream ~a is not an output stream!"
output)
(format output "test"))
Error: The stream #<STREAM::LATIN-1-FILE-STREAM /tmp/file.txt>
is not an output stream!
Your extra question: Why is the following form working?
(with-open-file (input (open #p"file.txt")) ...)
It just opens the file TWICE for reading.

Your usage of with-open-file is incorrect.
(with-open-file (output #p"file.txt"
:direction :output
:if-exists :supersede)
(format output "Hello"))

Related

How to open multiple URLs at the same time in an Emacs buffer?

I am using the Emacs editor together with the org-mode and evil-mode mainly for text handling and documentation. Often there is a topic where several different URLs to websites belong to.
Example: I have a text snippet on how to install Emacs:
*** install emacs
emacs - I want to try org-mode. What's the shortest path from zero to typing? - Stack Overflow
https://stackoverflow.com/questions/4940680/i-want-to-try-org-mode-whats-the-shortest-path-from-zero-to-typing
Index of /gnu/emacs/windows/emacs-26
http://ftp.gnu.org/gnu/emacs/windows/emacs-26/emacs-26.3-x86_64.zip
Installation target:
file://C:\Lupo_Pensuite\MyApps\emacs
How to
file://C:\Lupo_Pensuite\MyDocs\howto.txt
Is it possible to select the region and all the URLs are opened within my default web browser? And the file link is being opened by Windows Explorer? And the text file is opened with the associated editor?
Or even better: emacs is aware that the a.m. text snippet actually is a org-mode chapter. And regardless where within that chapter the cursor is positioned, something like M-x open-all-links-in-chapter is...opening all mentioned links in the current chapter.
Prio 1: is there something like that existing in emacs/org-mode/evil-mode already?
Prio 2: is there a elisp function you know which can achieve this use case?
Enviroment: Cygwin under Windows 10, emacs 26.3, org-mode 9.1.9
It turns out, that org-mode has this already built-in!
Today I was browsing the documentation of org-mode, wondering how exactly C-c C-o is working. That key combo is calling the emacs org-mode function "org-open-at-point". org-open-at-point is opening the URL where the cursor (in emacs speak: point) is positioned.
Now if a C-c C-o is pressed on a heading, then all URL's beneath that heading are opened! Which is exactly what I asked for from the beginning. Thanks a lot, NickD, for your constructive contributions!
Here the original help text:
When point is on a headline, display a list of every link in the entry, so it is possible to pick one, or all, of them.
Warning: used without thought, the following can bring your machine to its knees. I will add some more specific warnings at the end, but be careful!
The basic idea of the code below is to parse the buffer of an Org mode file, in order to get a parse tree of the buffer: that is done by org-element-parse-buffer. We can then use org-element-map to walk the parse tree and select only nodes of type link, applying a function to each one as we go. The function we apply, get-link, munges through the contents of the link node, extracting the type and path and returning a list of those two. Here's how it looks so far:
(defun get-link (x)
(let* ((link (cadr x))
(type (plist-get link :type))
(path (plist-get link :path)))
(if (or (string= type "http") (string= type "https"))
(list type path))))
(defun visit-all-http-links ()
(interactive)
(let* ((parse-tree (org-element-parse-buffer))
(links (org-element-map parse-tree 'link #'get-link)))
links))
Note that I only keep http and https links: you may want to add extra types.
This already goes a long way towards getting you what you want. In fact, if you load the file with the two functions above, you can try it on the following sample Org mode file:
* foo
** foo 1
http://www.google.com
https://redhat.com
* bar
** bar 2
[[https://gnome.org][Gnome]] is a FLOSS project. So is Fedora: https://fedoraproject.org.
* Code
#+begin_src emacs-lisp :results value verbatim :wrap example
(visit-all-http-links)
#+end_src
#+RESULTS:
#+begin_example
(("http" "//www.google.com") ("https" "//redhat.com") ("https" "//gnome.org") ("https" "//fedoraproject.com"))
#+end_example
and evaluating the source block with C-c C-c, you get the results shown.
Now all we need to do is convert each (TYPE PATH) pair in the result list to a real URL and then visit it - here's the final version of the code:
(defun get-link (x)
"Assuming x is a LINK node in an Org mode parse tree,
return a list consisting of its type (e.g. \"http\")
and its path."
(let* ((link (cadr x))
(type (plist-get link :type))
(path (plist-get link :path)))
(if (or (string= type "http") (string= type "https"))
(list type path))))
(defun format-url (x)
"Take a (TYPE PATH) list and return a proper URL. Note
the following works for http- and https-type links, but
might need modification for other types."
(format "%s:%s" (nth 0 x) (nth 1 x)))
(defun visit-all-http-links ()
(interactive)
(let* ((parse-tree (org-element-parse-buffer))
(links (org-element-map parse-tree 'link #'get-link)))
(mapcar #'browse-url (mapcar #'format-url links))))
We add a function format-url that does this: ("http" "//example.com") --> "http://example.com" and map it on the links list, producing a new list of URLS. Then we map the function browse-url (which is provided by emacs) on the resulting list and we watch the browser open them all.
WARNINGS:
If you have hundreds or thousands of links in the file, then you are going to create hundreds or thousands of tabs in your browser. Are you SURE your machine can take it?
If your links point to big objects, that's going to put another kind of memory pressure on your system. Are you SURE your machine can take it?
If your Org mode buffer is big, then org-element-parse-buffer can take a LONG time to process it. Moreover, even though there is a caching mechanism, it is not enabled by default because of bugs, so every time you execute the function you are going to parse the buffer AGAIN from scratch.
Every time you execute the function, you are going to open NEW tabs in your browser.
EDIT in response to questions in comments:
Q1: "visit-all-http-links opens all URLs in the file. My original question was, whether it is possible to open only the URLs which are being found in the current org-mode chapter."
A1: Doing just a region is a bit harder but possible, if you guarantee that the region is syntactically correct Org mode (e.g. a collection of headlines and their contents). You just write the region to a temporary buffer and then do what I did on the temp buffer instead of the original.
Here's the modified code using the visit-url function from Question 2:
(defun visit-all-http-links-in-region (beg end)
(interactive "r")
(let ((s (buffer-substring beg end)))
(with-temp-buffer
(set-buffer (current-buffer))
(insert s)
(let* ((parse-tree (org-element-parse-buffer))
(links (org-element-map parse-tree 'link #'get-link)))
(mapcar #'visit-url (mapcar #'format-url links))))))
(defun visit-all-http-links ()
(interactive)
(visit-all-http-links-in-region (point-min) (point-max)))
Very lightly tested.
Q2: "Every time I execute your function with your example URLs, the URLs are being opened with a different sequence - is it possible to open the URLs in that very sequence which is found in the org file?"
A2: The links are found deterministically in the order that they occur in the file. But the moment you call browse-url, all bets are off, because the URL now belongs to the browser, which will try to open each URL it receives in a separate tab and using a separate thread - in other words asynchronously. You might try introducing a delay between calls, but there are no guarantees:
(defun visit-url(url)
(browse-url)
(sit-for 1 t))
and then use visit-url instead of browse-url in visit-all-urls.

0x85 windows 1252 breaks line if file opened with utf-8 encoding

I have a file with an old format from the 70s used in Companies House (UK company registry).
I inherited a parser written 6 years ago which goes line by line and according to a set of conditions extracts the information from the line and inserts them into a dictionary.
There is a weird character that is breaking a line.
I copied this line to a new file awk '{if(NR==33411) print $0}' PROD216_1950_ew_1.dat > broken and opend broken in vim.
Turns out that weird character is read by vim a <85>.
The result is that everything after MAYFIELD is read as a new line.
Below the line in question:
000376702103032986930001 1993010119941024 193709 0105<BARRY ALEXANDER<GROSVENOR<<<<MAYFIELD 3<41 PLANTATION ROAD<THE PEAK<<HONG KONG<BANK EXECUTIVE<BRITISH<<
in vim becomes
000376702103032986930001 1993010119941024 193709 0105<BARRY ALEXANDER<GROSVENOR<<<<MAYFIELD <85>3<41 PLANTATION ROAD<THE PEAK<<HONG KONG<BANK EXECUTIVE<BRITISH<<
I am using codecs to read this file with a context manager, which I thought was the way of going about it -
Is there anything I am missing? What is that <85>?
with codecs.open(filepath, 'r', 'utf-8') as fh:
for line in fh:
linetype = determine_line_type(line)
if linetype == 'header':
continue
elif linetype == 'company':
do stuff...
elif linetype == 'officer':
do stuff...
vim shows <85> to indicate a hex 85 byte that is invalid in the current encoding (i.e., the encoding it's using to decode the file).
My guess is that the file's encoding is Windows-1252, in which hex 85 denotes the ellipsis character.
So the solution for your parser might be as simple as changing 'utf-8' to 'cp1252' in the codecs.open call.
After going around for some time here and here I came up with this solution, which works.
with open(filepath, encoding='utf-8') as fh:
for line in fh:
byteline = bytearray(line, encoding='utf-8').replace(b'\xc2\x85', b'')
line_clean = byteline.decode(encoding='utf-8')
# do stuff with clean line.
Knowing that the byte sequence that breaks the string is b'\xc2\x85' (it is interpreted as an ... ellipsis character.
First encode the string to an array of bytes with bytearray, then use replace method of the bytearray class, finally, decode the clean line using the decode method, which will return the string without the weird character from before the transformation.

Lua string.format and use of newline or control characters

I'm trying to string.format for raw output to the uart using NodeMCU.
I'm trying the function
uart.write(0,string.format("loop %03d local: %02d | gmt %02d:%02d:%02d local %02d/%02d/%04d\n",loops,timezonetime,gmthours,gmtmins,gmtsecs,Nmonth,Nday,Nyear))
but the \n is ignored, and text is concatenated.
print(string.format("loop %03d local: %02d | gmt %02d:%02d:%02d local %02d/%02d/%04d",loops,timezonetime,gmthours,gmtmins,gmtsecs,Nmonth,Nday,Nyear))
works as expected, but I can't control the newline always added by print()
How can I use uart.write and string.format to control the output including the placement and use of newline and other control characters?
The issue a result of newline handling in the LuaLoader that was used for accessing the NodeMCU board. When used with PUTTY, the output is as expected.
Here are the results of more detailed testing. It appears that \r does not work in the string parameter passed to uart.write()
-- uart.write Test
print("______first test____________") -- prime the output with a line and newline
uart.write(0,"asdfasdfasdfasdfasdf") -- no newline
print("______should be at end of same line as asdf...______")
uart.write(0,"asdfasdfasdfasdfasdf(newline)\n") -- with newline
print("______should be on line following asdf...____________")
uart.write(0,"asdfasdfasdfasdfasdf(CR)\r") -- with return only
uart.write(0,"OVERWRITE\n") -- overwrite the first part of asdf line, then newline
print("______should be on newline below OVERWRITE line ____________")
Output results:
dofile("uwtest.lua")
______first test____________
asdfasdfasdfasdfasdf______should be at end of same line as asdf...______
asdfasdfasdfasdfasdf(newline)
______should be on line following asdf...____________
asdfasdfasdfasdfasdf(CR)
OVERWRITE
______should be on newline below OVERWRITE line ____________
>
The expected result is the string "asdfasdfasdfasdfasdf(CR)\r" will be followed by a CR but not LF, causing the terminal cursor to move to the left
This appears to be an issue with the terminal emulation in LuaLoader.
When I connect to the NodeMCU with Putty, I get this output:
> dofile("uwtest.lua")
______first test____________
asdfasdfasdfasdfasdf______should be at end of same line as asdf...______
asdfasdfasdfasdfasdf(newline)
______should be on line following asdf...____________
OVERWRITEsdfasdfasdf(CR)
______should be on newline below OVERWRITE line ____________
>
The Putty output is as expected.

Emacs ESS keybinding "less than" dash

How to efficiently program in Emacs ESS-mode the key
"<" "[less than]"
to
"<- " "[less than][dash][space]"
Just like the MacOS version of R utilizes.
Looks like there is an existing function for it in the file ess-s-l.el. It would appear that you can use the variable ess-S-assign-key for this:
;; This is by Seth Falcon, modeled after ess-toggle-underscore (see below).
(defun ess-toggle-S-assign-key (force)
"Possibly bind the key in `ess-S-assign-key' to inserting `ess-S-assign'.
If `ess-S-assign-key' is \"_\", simply use \\[ess-toggle-underscore].
Otherwise, unless the prefix argument FORCE is set,
toggle between the new and the previous assignment."
(interactive "P")
(require 'ess-mode)
(require 'ess-inf)
(let ((current-action (lookup-key ess-mode-map ess-S-assign-key))
(insert-S-assign (lambda() (interactive)
(delete-horizontal-space) (insert ess-S-assign))))
(if (and (stringp ess-S-assign-key)
(string= ess-S-assign-key "_"))
(ess-toggle-underscore force)
;; else "do things here"
(let* ((current-is-S-assign (eq current-action insert-S-assign))
(new-action (if force insert-S-assign
;; else "not force" (default):
(if (or current-is-S-assign
(eq ess-S-assign-key-last insert-S-assign))
ess-S-assign-key-last
insert-S-assign))))
(message "[ess-toggle-S-assign-key:] current: '%s', new: '%s'"
current-action new-action)
(define-key ess-mode-map ess-S-assign-key new-action)
(define-key inferior-ess-mode-map ess-S-assign-key new-action)
(if (not (and force current-is-S-assign))
(setq ess-S-assign-key-last current-action))))))
Perhaps this is ESS version dependent.
In my version of ESS (12.03), it seems that you can bind ">" to 'ess-insert-S-assign to get what you like.
Look at the ess- commands available to you (M-x ess-<TAB><TAB> and search in the *Completions* buffer that just popped up for assign) to see which command will be the likely culprit that you should bind to ">".
If that does not work for you -- perhaps you might need to upgrade.

Reading the binary output of an external program in Common Lisp

I'm trying to run an external program in SBCL and capture its output.
The output is binary data (a png image), while SBCL insists on interpreting it as strings.
I tried a number of ways, like
(trivial-shell:shell-command "/path/to/png-generator" :input "some input")
(with-input-from-string (input "some input")
(with-output-to-string (output)
(run-program "/path/to/png-generator" () :input input :output output))
(with-input-from-string (input "some input")
(flexi-streams:with-output-to-sequence (output)
(run-program "/path/to/png-generator" () :input input :output output))
But I get errors like
Illegal :UTF-8 character starting at byte position 0.
It seems to me that SBCL is trying to interpret the binary data as a text and decode it. How do I change this behaviour ? I'm interested only in obtaining a vector of octets.
Edit: Since it is not clear from the text above, I'd like to add that at least in the case of flexi-stream, the element-type of the stream is a flexi-streams:octect (which is a (unsigned-byte 8)).
I would expect at least in this case run-program to read the raw bytes without many issues. Instead I get a message like Don't know how to copy to stream of element-type (UNSIGNED-BYTE 8)
Edit: I got angry at not being able to do this very simple task and solved the problem.
Functionally, the ability to send a stream of type UNSIGNED-BYTE into run-program and have it work correctly is severely limited, for reasons I don't understand. I tried gray streams, flexi-streams, fd streams, and a few other mechanisms, like you.
However, perusing run-program's source (for the fifth or sixth time), I noticed that there's an option :STREAM you can pass to output. Given that, I wondered if read-byte would work... and it did. For more performant work, one could determine how to get the length of a non-file stream and run READ-SEQUENCE on it.
(let*
;; Get random bytes
((proc-var (sb-ext:run-program "head" '("-c" "10" "/dev/urandom")
:search t
;; let SBCL figure out the storage type. This is what solved the problem.
:output :stream))
;; Obtain the streams from the process object.
(output (process-output proc-var))
(err (process-error proc-var)))
(values
;;return both stdout and stderr, just for polish.
;; do a byte read and turn it into a vector.
(concatenate 'vector
;; A byte with value 0 is *not* value nil. Yay for Lisp!
(loop for byte = (read-byte output nil)
while byte
collect byte))
;; repeat for stderr
(concatenate 'vector
(loop for byte = (read-byte err nil)
while byte
collect byte))))
If you're willing to use some external libraries, this can be done with babel-streams. This is a function I use to safely get content from a program. I use :latin-1 because it maps the first 256 bytes just to the characters. You could remove the octets-to-string and have the vector.
If you wanted stderr as well, you could use nested 'with-output-to-sequence' to get both.
(defun safe-shell (command &rest args)
(octets-to-string
(with-output-to-sequence (stream :external-format :latin-1)
(let ((proc (sb-ext:run-program command args :search t :wait t :output stream)))
(case (sb-ext:process-status proc)
(:exited (unless (zerop (sb-ext:process-exit-code proc))
(error "Error in command")))
(t (error "Unable to terminate process")))))
:encoding :latin-1))
Paul Nathan already gave a pretty complete answer as to how to read I/O from a program as binary, so I'll just add why your code didn't work: because you explicitely asked SBCL to interpret the I/O as a string of UTF-8 characters, using with-{in,out}put-to-string.
Also, I'd like to point that you don't need to go as far as run-program's source code to get to the solution. It's clearly documented in SBCL's manual.

Resources