How do you parse 4-bit chunks from binary? - parsing

I'm trying to understand how I might parse binary per 4 bits if it is possible.
For example:
I have 2-byte codes that need to be parsed to determine which instruction to use
#{1NNN} where the first 4 bits tell where which instruction, and NNN represents a memory location (i.e. #{1033} says jump to memory address #{0033}
It seems to be easy to do this with full bytes, but not with half bytes:
parse #{1022} [#{10} {#22}]
because #{1} isn't valid binary!
So far, I've used giant switch statements with: #{1033} AND #{F000} = #{1000} in order to process these, but wondering how a more mature reboler might do this.

This is a rather big entry, but it addresses your needs and shows off PARSE a bit.
This is basically a working, albeit simple VM which uses the memory layout you describe above.
I set up a simple block of RAM which is an actual program that it executes when I use PARSE with the emulator grammar rule... basically, it increments an address and then jumps to that address, skipping over an NOP.
it then hits some illegal op and dies.
REBOL [
title: "simple VM using Parse, from scratch, using no external libraries"
author: "Maxim Olivier-Adlhoch"
date: 2013-11-15
]
;----
; builds a bitset with all low-order bits of a byte set,
; so only the high bits have any weight
;----
quaternary: func [value][
bs: make bitset!
reduce [
to-char (value * 16)
'-
to-char ((value * 16) + 15)
]
]
;------
; get the 12 least significant bits of a 16 bit value
LSB-12: func [address [string! binary!] ][
as-binary (address AND #{0FFF})
]
;------
i32-to-binary: func [
n [integer!]
/rev
][
n: load join "#{" [form to-hex to-integer n "}"]
either rev [head reverse n][n]
]
;------
; load value at given address. (doesn't clear the opcode).
LVAL: func [addr [binary!]][
to-integer copy/part at RAM ( (to-integer addr) + 1) 2
]
;------
; implement the opcodes which are executed by the CPU
JMP: func [addr][
print ["jumping to " addr]
continue: at RAM ((to-integer addr) + 1) ; 0 based address but 1 based indexing ;-)
]
INC: func [addr][
print ["increment value at address: " addr]
new-val: 1 + LVAL addr
addr: 1 + to-integer addr
bin-val: at (i32-to-binary new-val) 3
change at RAM addr bin-val
]
DEC: func [addr][
print ["decrement value at address: " addr]
]
NOP: func [addr][
print "skipping Nop opcode"
]
;------
; build the bitsets to match op codes
op1: quaternary 1
op2: quaternary 2
op3: quaternary 3
op4: quaternary 4
;------
; build up our CPU emulator grammar
emulator: [
some [
[
here:
[ op1 (op: 'JMP) | op2 (op: 'INC) | op3 (op: 'DEC) | op4 (op: 'NOP)] ; choose op code
:here
copy addr 2 skip (addr: LSB-12 addr) ; get unary op data
continue:
(do reduce [op addr])
:continue
]
| 2 skip (
print ["^/^/^/ERROR: illegal opcode AT: " to-binary here " offset[" -1 + index? here "]"] ; graceful crash!
)
]
]
;------
; generate a bit of binary RAM for our emulator/VM to run...
0 2 4 6 8 ; note ... don't need comments, Rebol just skips them.
RAM: #{2002100540FF30015FFF}
RAM-blowup: { 2 002 1 005 4 0FF 3 001 5 FFF } ; just to make it easier to trace op & data
parse/all RAM emulator
print "^/^/Yes that error is on purpose, I added the 5FFF bytes^/in the 'RAM' just to trigger it :-)^/"
print "notice that it doesn't run the NOP (at address #0006), ^/since we used the JMP opcode to jump over it.^/"
print "also notice that the first instruction is an increment ^/for the address which is jumped (which is misaligned on 'boot')^/"
ask "press enter to continue"
the output is as follows:
increment value at address: #{0002}
jumping to #{0006}
decrement value at address: #{0001}
ERROR: illegal opcode AT: #{5FFF} offset[ 8 ]
Yes that error is on purpose, I added the 5FFF bytes
in the 'RAM' just to trigger it :-)
notice that it doesn't run the NOP (at address #0006),
since we used the JMP opcode to jump over it.
also notice that the first instruction is an increment
for the address which is jumped (which is misaligned on 'boot')
press enter to continue

Related

Forth: How to create a word that compiles other words until certain delimiter is found?

I use Forth (namely Swapforth) to configure certain hardware via I2C. I have a word:
i2c1-send ( reg-address byte -- )
that writes a byte to the specific internal register of a certain chip.
The initialization sequence is quite long, and therefore implementing it as below is not vialable due to memory consumption.
: i2c1-init
$1201 $10 i2c1-send
$2130 $43 i2c1-send
[...]
$0231 $43 i2c1-send
;
I have created an implementation that creates a structure holding the length of the sequence in the first cell and triple bytes in the next cells. (Please note that i2c1-send is just a placeholder allowing you to test it without my hardware).
: i2c1-send ( reg_addr byte -- )
\ It is just a placeholder to show what will be written in HW
swap
." addr=" hex . ." val=" . decimal CR
;
: i2c1: ( "<spaces>name" -- )
create here $326e9 0 ,
does> dup cell+ swap
# 0 do
dup c# >r 1+
dup c# 8 lshift swap 1+
dup c# rot or r> i2c1-send
1+
loop
drop
;
: i2c1-def ( addr val -- )
c, ( adr )
dup 8 rshift c,
255 and c,
;
: i2c1; ( -- )
\ Make sure that i2c1: was used before
$326e9 <> abort" i2c1; without i2c1:"
dup cell+ here swap - ( first_cell length )
\ Verify that the length is a multiple of 3
3 /mod swap 0<> abort" illegal length - not a multiple of 3"
swap !
;
With the above code you define the initialization list similarly:
i2c1: set1
$1234 $11 i2c1-def
$1521 $18 i2c1-def
[...]
$2313 $10 i2c1-def
i2c1;
But the memory consumption is significantly reduced (by factor of 2 in case of J1B Forth CPU).
However I dislike the syntax. I'd prefere to have something that would allow to define the initialization list just by numbers, until certain delimiter is found, like below:
i2c1-x: i2c1-init
$1234 $11
$1521 $18
[...]
$2313 $10
i2c1-x;
I have created the word shown below:
: i2c-delim s" i2c1-x;" ;
: i2c1-x: create here 0 ,
begin
parse-name
2dup i2c-delim compare 0<> while
evaluate \ We store the address later
parse-name
evaluate
c,
\ Now store the address
dup 8 rshift c,
255 and c,
repeat
2drop
dup cell+ here swap - ( first_cell length )
\ Verify that the length is a multiple of 3
3 /mod swap 0<> abort" length not a multiple of 3"
swap !
does> dup cell+ swap
# 0 do
dup c# >r 1+
dup c# 8 lshift swap 1+
dup c# rot or r> i2c1-send
1+
loop
drop
;
It works perfectly for short definitions:
i2c1-x: set2 $1234 $ac $6543 $78 $9871 $01 $3440 $02 i2c1-x;
But fails for longer ones that use multiple lines:
i2c1-x: set2
$1234 $ac
$6543 $78
$9871 $01
$3440 $02
i2c1-x;
Is it possible to define i2c1-x so that it handles multiple lines, or do I have to use solution based on separate i2c1:, i2c1-def and i2c1;?
There is REFILL word to parse multiple lines.
\ Get the next name (lexeme) possibly from the next lines
\ NB: Use the result of parse-name-sure immediate
\ since it may be garbled after the next refill
\ (the buffer may be be overwritten by the next line).
: parse-name-sure ( -- c-addr u|0 )
begin parse-name dup 0= while refill 0= if exit then 2drop repeat
;
\ Check if the first string equals to the second
: equals ( c-addr2 u2 c-addr1 u1 -- flag )
dup 3 pick <> if 2drop 2drop false exit then
compare 0=
;
It is a common approach to translate the input until some delimiter. A general function to perform this approach:
\ Translate the input till a delimiter
\ using xt as translator for a lexeme
2variable _delimiter
: translate-input-till-with ( i*x c-addr u xt -- j*x )
>r _delimiter 2!
begin parse-name-sure dup while
2dup _delimiter 2# equals 0= while
r# execute
repeat then 2drop rdrop
;
There is a sense to also factor out the manipulation of 16-bits units into a library:
[undefined] w# [if]
\ NB: little-endian endianness variant
: w! ( x addr -- ) dup 1+ >r >r dup 8 rshift r> c! r> c! ;
: w# ( addr -- x ) dup c# 8 lshift swap 1+ c# or ;
: w, ( x -- ) here 2 allot w! ;
[then]
Also, a function to converting text into number should be in a library. Using evaluate for that is not hygienic. See example of StoN definition in "How to enter numbers in Forth" question. A helper to convert the "$"-prefixed numbers may be found in your Forth-system.
\ dummy definitions for test only
: s-to-n ( addr u -- x ) evaluate ;
: send-i2c1 ( addr x -- ) ." send: " . . CR ;
The application code:
\ Translate the input numbers till the delimiter into the special format
\ (the code could be simplified using the quotations)
: i2c-delim s" i2c1-x;" ;
: translate-i2c-pair ( c-addr u -- )
s-to-n
parse-name-sure
2dup i2c-delim equals abort" translate-i2c: unexpected delimiter"
s-to-n c, w,
;
: translate-i2c-input ( -- )
i2c-delim ['] translate-i2c-pair translate-input-till-with
;
\ Send data from the special format
: send-i2c1-bulk ( addr u -- )
3 / 0 ?do
dup c# swap 1+
dup w# swap 2+ >r send-i2c1 r>
loop drop
;
\ The defining word
: i2c1-x:
create here >r 0 , here >r translate-i2c-input here r> - r> !
does> dup cell+ swap # send-i2c1-bulk
;
A testcase
i2c1-x: test
1 2
3 4
5
6
i2c1-x;
test

How to count locals in ANS-Forth?

While developing BigZ, mostly used for number theoretical experiments, I've discovered the need of orthogonality in the word-set that create, filter or transform sets. I want a few words that logically combinated cover a wide range of commands, without the need to memorize a large number of words and ways to combinate them.
1 100 condition isprime create-set
put the set of all prime numbers between 1 and 100 on a set stack, while
function 1+ transform-set
transform this set to the set of all numbers p+1, where p is a prime less than 100.
Further,
condition sqr filter-set
leaves the set of all perfect squares on the form p+1 on the stack.
This works rather nice for sets of natural numbers, but to be able to create, filter and transform sets of n-tuples I need to be able to count locals in unnamed words. I have redesigned words to shortly denote compound conditions and functions:
: ~ :noname ;
: :| postpone locals| ; immediate
1 100 ~ :| p | p is prime p 2 + isprime p 2 - isprime or and ;
1 100 ~ :| a b | a dup * b dup * + isprime ;
Executing this two examples gives the parameter stack ( 1 100 xt ) but to be able to handle this right, in the first case a set of numbers and in the second case a set of pairs should be produced, I'll have to complement the word :| to get ( 1 100 xt n ) where n is the numbet of locals used. I think one could use >IN and PARSE to do this, but it was a long time ago I did such things, so I doubt I can do it properly nowadays.
I didn't understand (LOCALS) but with patience and luck I managed to do it with my original idea:
: bl# \ ad n -- m
over + swap 0 -rot
do i c# bl = +
loop negate ;
\ count the number of blanks in the string ad n
variable loc#
: locals# \ --
>in # >r
[char] | parse bl# loc# !
r> >in ! ; immediate
\ count the number of locals while loading
: -| \ --
postpone locals#
postpone locals| ; immediate
\ replace LOCALS|
Now
: test -| a b | a b + ;
works as LOCALS| but leave the number of locals in the global variable loc#.
Maybe you should drop LOCALS| and parse the local variables yourself. For each one, call (LOCAL) with its name, and end with passing an empty string.
See http://lars.nocrew.org/dpans/dpans13.htm#13.6.1.0086 for details.

gforth base-execute syntax

Some online gforth docs provide a seemingly complete description of base-execute's effects:
base-execute i*x xt u – j*x gforth “base-execute”
execute xt with the content of BASE being u, and restoring the
original BASE afterwards.
But the syntax for the effects seems like a lock without a key -- the page links to nothing that describes what i*x xt u – j*x signifies. Some hunting turned up a partial description of the syntax notation, (which tells us that u is an unsigned number and xt is an execution token), but that's still not enough to understand i*x xt u – j*x.
How is base-execute used, and what does it do?
To understand what base-execute does you need to understand execute and BASE. I'll also explain how to read i*x and j*x in the stack effect.
execute works by taking an execution token xt and executing it. ' 1+ execute is the same as 1+ on its own. The reason to use execute, though, is because you can pass xt on the stack, instead of having to choose it ahead of time. For instance:
: exec-twice dup >r execute r> execute ;
2 ' 1+ exec-twice . ( this outputs 4 )
BASE is a variable that controls what numeric base to use for input and output.
BASE is initially 10. So 5 2 BASE ! . outputs 101 (which is 5 in base 2).
base-execute puts them together: it changes BASE to u, executes xt, then restores BASE to its previous value. Its implementation might look like this:
: base-execute BASE # >r BASE ! execute r> BASE ! ;
Here's an example usage:
: squared ( n1 -- n2 ) dup * ;
: squares ( n -- ) 0 do i squared . loop ;
10 squares ( 0 1 4 9 16 25 36 49 64 81 )
: hex-execute ( i*x xt -- j*x ) 16 base-execute ;
10 ' squares hex-execute ( 0 1 4 9 10 19 24 31 40 51 )
10 squares ( 0 1 ... 81 we're back to decimal )
Now for i*x xt u -- j*x:
The stack notation documentation you linked to has most of the information you need to read the effect. i*x -- j*x means that something might happen to the stack, but it doesn't specify what. In this case, the exact stack effect depends on what xt is.
To know the stack effect with a given xt, replace i*x and j*x with the two sides of xt's stack effect.
For example, if xt is ' . you would look at .'s stack effect, which is n --. In that case you could think of base-execute's stack effect as n xt-of-. u --.

How much memory used in Erlang VM?

i want to konw this data struct will use how much memory in Erlang VM?
[{"3GPP-UTRAN-FDD", [{"utran-cell-id-3gpp","CID1000"}, "1996-12-19t16%3a39%3a57-08%3a00", "1996-12-19t15%3a39%3a27%2e20-08%3a00"]}]
In my application, every process will store this data in self's loop data, and the numbert of this proces will be 120000.
The result which i test:
don't store this data, the memory will be:
memory[kB]: proc 1922806, atom 2138, bin 24890, code 72757, ets 459321
store this data, the momory will be:
memory[kB]: proc 1684032, atom 2138, bin 24102, code 72757, ets 459080
So the big difference is the memoery used by proc: (1922806 - 1684032) / 1024 = 233M.
After research, i find an insterting thing:
L = [{"3GPP-UTRAN-FDD", [{"utran-cell-id-3gpp","CID1000"}, "1996-12-19t16%3a39%3a57-08%3a00", "1996-12-19t15%3a39%3a27%2e20-08%3a00"]}].
B = list_to_binary(io_lib:format("~p", L)).
erts_debug:size(B). % The output is 6
The memory just use 6 words after using binary? How to explain this?
There are two useful functions for measuring the size of an Erlang term: erts_debug:size/1 and erts_debug:flat_size/1. Both of these functions return the size of the term in words.
erts_debug:flat_size/1 gives you the total size of a message without term-sharing optimization. This is guaranteed to be the size of the term if it is copied to a new heap, as with message passing and ets tables.
erts_debug:size/1 gives you the size of the term as it is in the current process' heap, allowing for memory usage optimization by sharing repeated terms.
Here is a demonstration of the differences:
1> MyTerm = {atom, <<"binary">>, 1}.
{atom,<<"binary">>,1}
2> MyList = [ MyTerm || _ <- lists:seq(1, 100) ].
[{atom,<<"binary">>,1}|...]
3> erts_debug:size(MyList).
210
4> erts_debug:flat_size(MyList).
1200
As you can see, there is a significant difference in the sizes due to term sharing.
As for your specific term, I used the Erlang shell (R16B03) and measured the term with flat_size. According to this, the memory usage of your term is: 226 words (1808B, 1.77KB).
This is a lot of memory to use for what appears to be a simple term, but that is outside of the scope of this question.
the size of the whole binary is 135 bytes when you do it list_to_binary(io_lib:format("~p", L))., if you are on a 64 bit system it represents 4.375 words so 6 words should be the correct size, but you have lost the direct access to the internal structure.
Strange but can be understood:
19> erts_debug:flat_size(list_to_binary([random:uniform(26) + $a - 1 || _ <- lists:seq(1,1000)])).
6
20> erts_debug:flat_size(list_to_binary([random:uniform(26) + $a - 1 || _ <- lists:seq(1,10000)])).
6
21> size(list_to_binary([random:uniform(26) + $a - 1 || _ <- lists:seq(1,10000)])).
10000
22> (list_to_binary([random:uniform(26) + $a - 1 || _ <- lists:seq(1,10000)])).
<<"myeyrltgyfnytajecrgtonkdcxlnaoqcsswdnepnmdxfrwnnlbzdaxknqarfyiwewlugrtgjgklblpdkvgpecglxmfrertdfanzukfolpphqvkkwrpmb"...>>
23> erts_debug:display({list_to_binary([random:uniform(26) + $a - 1 || _ <- lists:seq(1,10000)])}).
{<<10000 bytes>>}
"{<<10000 bytes>>}\n"
24>
This means that the erts_debug:flat_size return the size of the variable (which is roughly a type information, a pointer to the data and its size), but not the size of the binary data itself. The binary data is stored elsewhere and can be shared by different variables.

Storing data in RAM without avr-libc. How can I configure the proper memory sections using a custom linker script and initialization code?

Here's my linker script:
MEMORY {
text (rx) : ORIGIN = 0x000000, LENGTH = 64K
data (rw!x) : ORIGIN = 0x800100, LENGTH = 0xFFA0
}
SECTIONS {
.vectors : AT (0x0000) { entry.o (.vectors); }
.text : AT (ADDR (.vectors) + SIZEOF(.vectors)) { * (.text.startup); * (.text); * (.progmem.data); _etext = .; }
.data : AT (ADDR (.text) + SIZEOF (.text)) { PROVIDE (__data_start = .); * (.data); * (.rodata); * (.rodata.str1.1); PROVIDE (__data_end = .); } > data
.bss : AT (ADDR (.bss)) { PROVIDE (__bss_start = .); * (.bss); PROVIDE (__bss_end = .); } > data
__data_load_start = LOADADDR(.data);
__data_load_end = __data_load_start + SIZEOF(.data);
}
And this is my initialization code. init is called at reset.
.section .text,"ax",#progbits
/* Handle low level hardware initialization. */
.global init
init: eor r1, r1
out 0x3f, r1
ldi r28, 0xFF
ldi r29, 0x02
out 0x3e, r29
out 0x3d, r28
rjmp __do_copy_data
rjmp __do_clear_bss
jmp main
/* Handle copying data into RAM. */
.global __do_copy_data
__do_copy_data: ldi r17, hi8(__data_end)
ldi r26, lo8(__data_start)
ldi r27, hi8(__data_start)
ldi r30, lo8(__data_load_start)
ldi r31, hi8(__data_load_start)
rjmp .L__do_copy_data_start
.L__do_copy_data_loop: lpm r0, Z+
st X+, r0
.L__do_copy_data_start: cpi r26, lo8(__data_end)
cpc r27, r17
brne .L__do_copy_data_loop
rjmp main
/* Handle clearing the BSS. */
.global __do_clear_bss
__do_clear_bss: ldi r17, hi8(__bss_end)
ldi r26, lo8(__bss_start)
ldi r27, hi8(__bss_start)
rjmp .L__do_clear_bss_start
.L__do_clear_bss_loop: st X+, r1
.L__do_clear_bss_start: cpi r26, lo8(__bss_end)
cpc r27, r17
brne .L__do_clear_bss_loop
The problem is that the initialization code hangs sometime during the copying process. Here's an edited dump of my symbol table, if it's helpful to anyone.
00000000 a __tmp_reg__
...
00000000 t reset
...
00000001 a __zero_reg__
...
0000003d a __SP_L__
...
00000074 T main
0000009a T init
000000ae T __do_copy_data
000000c6 T __do_clear_bss
...
00000446 A __data_load_start
00000446 T _etext
0000045b A __data_load_end
00800100 D __data_start
00800100 D myint
00800115 B __bss_start
00800115 D __data_end
00800115 b foobar.1671
00800135 B ticks
00800139 B __bss_end
C is designed to work on von Neumann architectures. AVR is Harvard based. This means that C expects strings to be in RAM. As a consequence, if you ever take a look at the disassembly for any elf binary that would eventually be copied as a hex to your AVR chip, you will see two sections: __do_copy_data and __do_clear_bss [when required]. These routines, which are added in the linking stages, take care of the basic needs of the C language. As a consequence, what you are seeing here with your pointers is likely that they are pointing to the wrong addresses. In other words, they are either pointing to an address in program space but you are reading from data space [different address bus]. Or you are purposely reading from data space but have not copied the strings over.
See:
avr/pgmspace.h
FAQ: ROM Array and scroll down for strings
and of course, the AVR instruction set as provided by Atmel, especifically, the instruction to copy program memory over to data memory
Edited to reflect new question and comment: Your assembly for both sections look ok to me. I will have to take a look at your linker scripts with closer scrutinity to check if there any funny businesses going on there. Since you are writing a bootloader, do you mind if I ask if you have taken a look at bootloader support on AVR-libc?
http://deans-avr-tutorials.googlecode.com/svn/trunk/Progmem/Output/Progmem.pdf
This document may help to clarify the use of flash/ram memories of AVR.
I actually got it to work. All I had to do was enable reading from and writing to external RAM in the SREG. It was mindblowingly obvious and simple, I know, but it was buried in the data sheet and it was poorly documented.
This resource was helpful, but didn't have any documentation for my chip. If you're having this problem, look in the data sheet for your AVR and see how it is related. The process is not the same for all variations of AVRs.
How to use external RAM.

Resources