I trying to determine the number of coding tree blocks contained in each slice of a hevc picture without actually doing any decoding - aside from sps pps and slice header parsing.
Can the end slice address be calculated from the slice header?
You can always make a few changes to the encoder so it gives you the partitioning information in a log file.
Can that be an option for you?
as far as I know neither parameters sets nor the slices headers contains this information, I can assure you that because I am working on HM codec in the decoder side.
a simple way to find nub of CTU in the current frame is to add simply this line in the TDecGop::filterPicture (inside TLibDecoder/TDecGop.cpp) :
std::cout << "*Slice Cur End Ctu Ts Addr :\t" << pcSlice->getSliceCurEndCtuTsAddr() << "\n"; after recovering the slice pointer.
or better :
std::cout << "numCtusInFrame :\t" << pcPic->getNumberOfCtusInFrame() << "\n";
in 1st line of the function. you will get in the log screen (file) :
*Slice Cur End Ctu Ts Addr : 30
POC 0 TId: 0 ( I-SLICE, QP 21 ) [DT 0.087] [L0 ] [L1 ] [:,(unk)]
or
numCtusInFrame : 30
POC 0 TId: 0 ( I-SLICE, QP 21 ) [DT 0.087] [L0 ] [L1 ] [:,(unk)]
Related
I am trying to decode Can frames with micropython. I miss a lot of functions from python 3.x. I get 8 hex long bytes from the Rx SPI buffer of an MCP2515. For information, I'm on openMV IDE. I would like to extract the 1st and 2nd to make an integer, the 3rd and 4th to make another one, etc...
Along my researches, I found a function that convert a byte to signed integer (as the sign=True argument is not implemented in the from_byte function).
The main problem is that the solutions provided on others python's forum are only concerning python 3.x for pc but not micropython and a lot of solutions end with "function doesn't exist".
byte = b'\x12\x34\x56\x78\x9A\xBC\xDE\xFF'
def byte2int(bList, signed:bool=False):
if signed :
r = 0
for i in range(2):
d = 32 - ((i + 1) * 8)
r += bList[i] << d
return r
else : return int.from_bytes(byte, 'big')
Moreover, I don't think that I can trust the print function as print(b) return :
> print(b)
b'\x124Vx\x9a\xbc\xde\xff'
I don't understand why does the 3 vanished and was replaced by a 'V' after the 4 ! Sometime, my MCP2515 return some '|', '?', '.', '>', '<' or '=' when I print the can frames which are not hex characters !?
My hypothesis is that these are corrupted frames or SPI transmission.
Please, can someone help me to understand why micropython give me these messy results ?
Thank you very for your help.
I wrote something. Maybe not the best optimization possible, but I have to deal with micropython which is not as advanced than Python 3.x
This is a solution (surely not the most optimal). But it seems to work.
byte = b'\xFF\xC8\x56\x78\x9A\xBC\xDE\xFF'
byte2 = b'\x00\x00\x00#\x00\x00\x00\x00'
def extract(trame, signed:bool=False):
r = [0,0,0,0]
trame = [int(x) for x in bytearray(byte)]
for i in range(len(trame)/2):
val = (trame[i*2]<<8)+trame[i*2+1]
if signed and val > 0x8000 :
r[i] = -(65535-int(val)+1)
else :
r[i] = int(val)
return r
print(extract(byte, True))
I have an binary file with shows glibberish infos if i open it in Notepad.
I am working on an plugin to use with wireshark.
So my problem is that I need help. I am reading in an File and need to find 'V' '0' '0' '1' (0x56 0x30 0x30 0x31) in the File, because its the start of an Header, with means there is an packet inside. And I need to do this for the whole file, like parsing. Also should start the Frame with V 0 0 1 and not end with it.
I currently have an Code where I am searching for 0x7E and parse it. What I need is the length of the frame. For example V 0 0 1 is found, so the Length from V to the Position before the next V 0 0 1 in the File. So that I can work with the length and add it to an captured length to get the positions, that wireshark can work with.
For example my unperfect Code for working with 0x7E:
local line = file:read()
local len = 0
for c in (line or ''):gmatch ('.') do
len = len + 1
if c:byte() == 0x7E then
break
end
end
if not line then
return false
end
frame.captured_length = len
Here is also the Problem that the Frame ends with 7E which is wrong. I need something that works perfectly for 'V' '0' '0' '1'. Maybe I need to use string.find?
Please help me!
Thats an example how my file looks like if i use the HEX-Editor in Visual Studio Code.
Lua has some neat pattern tools. Here's a summary:
(...) Exports all captured text within () and gives it to us.
-, +, *, ?, "Optional match as little as possible", "Mandatory match as much as possible", "optional match as much as possible", "Optional match only once", respectively.
^ and $: Root to start or end of file, respectively.
We'll be using this universal input and output to test with:
local output = {}
local input = "V001Packet1V001Packet2oooV001aaandweredonehere"
The easiest way to do this is probably to recursively split the string, with one ending at the character before "V", and the other starting at the character after "1". We'll use a pattern which exports the part before and after V001:
local this, next = string.match(input, "(.-)V001(.*)")
print(this,next) --> "", "Packet1V001Packet2..."
Simple enough. Now we need to do it again, and we also need to eliminate the first empty packet, because it's a quirk of the pattern. We can probably just say that any empty this string should not be added:
if this ~= "" then
table.insert(output, this)
end
Now, the last packet will return nil for both this and next, because there will not be another V001 at the end. We can prepare for that by simply adding the last part of the string when the pattern does not match.
All put together:
local function doStep(str)
local this, next = string.match(str, "(.-)V001(.*)")
print(this,next)
if this then
-- There is still more packets left
if this ~= "" then
-- This is an empty packet
table.insert(output, this)
end
if next ~= "" then
-- There is more out there!
doStep(next)
end
else
-- We are the last survivor.
table.insert(output, str)
end
end
Of course, this can be improved, but it should be a good starting point. To prove it works, this script:
doStep(input)
print(table.concat(output, "; "))
prints this:
Packet1; Packet2ooo; aaandweredonehere
I'm attempting to create a Lua program to monitor periodic status pings of a slave device. The slave device sends its status in 16-bit hexadecimal words, which I need to convert to a binary string since each bit pertains to a property of the device. I can receive the input string, and I have a table containing 16 keys for each parameter. But I am having a difficult time understanding how to convert the hexadecimal word into a string of 16-bits so I can monitor it.
Here is a basic function of what I'm starting to work on.
function slave_Status(IP,Port,Name)
status = path:read(IP,Port)
sTable = {}
if status then
sTable.ready=bit32.rshift(status:byte(1), 0)
sTable.paused=bit32.rshift(status:byte(1), 1)
sTable.emergency=bit32.rshift(status:byte(1), 2)
sTable.started=bit32.rshift(status:byte(1), 3)
sTable.busy=bit32.rshift(status:byte(1), 4)
sTable.reserved1=bit32.rshift(status:byte(1), 5)
sTable.reserved2=bit32.rshift(status:byte(1), 6)
sTable.reserved3=bit32.rshift(status:byte(1), 7)
sTable.reserved4=bit32.rshift(status:byte(2), 0)
sTable.delay1=bit32.rshift(status:byte(2), 1)
sTable.delay2=bit32.rshift(status:byte(2), 2)
sTable.armoff=bit32.rshift(status:byte(2), 3)
sTable.shieldoff=bit32.rshift(status:byte(2), 4)
sTable.diskerror=bit32.rshift(status:byte(2), 5)
sTable.conoff=bit32.rshift(status:byte(2), 6)
sTable.envoff=bit32.rshift(status:byte(2), 7)
end
end
I hope this approach is understandable? I'd like to receive the Hex strings, for example 0x18C2 and turn that to 0001 1000 1100 0010, shifting the right-most bit to the right and placing that into the proper key. Then later in the function I would monitor if that bit had changed for the better or worse.
If I run a similar function in Terminator in Linux, and print out the pairs I get the following return:
49
24
12
6
3
1
0
0
56
28
14
7
3
1
0
0
This is where I am not understanding how to take each value and set it to bits
I'm pretty new to this so I do not doubt that there is an easier way to do this. If I need to explain further I will try.
tonumber(s, 16) will convert hex representation to decimal and string.char will return a symbol/byte representation of a number. Check this recent SO answer for an example of how they can be used; the solution in the answer may work for you.
I'd approach this in a different fashion than the one suggested by Paul.
First, create a table storing the properties of devices:
local tProperty = {
"ready",
"paused",
"emergency",
"started",
"busy",
"reserved1",
"reserved2",
"reserved3",
"reserved4",
"delay1",
"delay2",
"armoff",
"shieldoff",
"diskerror",
"conoff",
"envoff",
}
Then, since your device sends the data as 0xYYYY, you can call tonumber directly (if not a string). Use a function to store each bit in a table:
function BitConvert( sInput )
local tReturn, iNum = {}, tonumber( sInput ) -- optionally pass 16 as second argument to tonumber
while iNum > 0 do
table.insert( tReturn, 1, iNum % 2 )
iNum = math.floor( iNum / 2 )
end
for i = #tProperty - #tReturn, 1, -1 do
table.insert( tReturn, 1, 0 )
end
return tReturn
end
And then, map both the tables together:
function Map( tKeys, tValues )
local tReturn = {}
for i = 1, #tKeys do
tReturn[ tKeys[i] ] = tValues[i]
end
return tReturn
end
In the end, you would have:
function slave_Status( IP, Port, Name )
local status = path:read( IP, Port )
local sTable = Map( tProperty, BitConvert(status) )
end
I'm trying to understand how I might parse binary per 4 bits if it is possible.
For example:
I have 2-byte codes that need to be parsed to determine which instruction to use
#{1NNN} where the first 4 bits tell where which instruction, and NNN represents a memory location (i.e. #{1033} says jump to memory address #{0033}
It seems to be easy to do this with full bytes, but not with half bytes:
parse #{1022} [#{10} {#22}]
because #{1} isn't valid binary!
So far, I've used giant switch statements with: #{1033} AND #{F000} = #{1000} in order to process these, but wondering how a more mature reboler might do this.
This is a rather big entry, but it addresses your needs and shows off PARSE a bit.
This is basically a working, albeit simple VM which uses the memory layout you describe above.
I set up a simple block of RAM which is an actual program that it executes when I use PARSE with the emulator grammar rule... basically, it increments an address and then jumps to that address, skipping over an NOP.
it then hits some illegal op and dies.
REBOL [
title: "simple VM using Parse, from scratch, using no external libraries"
author: "Maxim Olivier-Adlhoch"
date: 2013-11-15
]
;----
; builds a bitset with all low-order bits of a byte set,
; so only the high bits have any weight
;----
quaternary: func [value][
bs: make bitset!
reduce [
to-char (value * 16)
'-
to-char ((value * 16) + 15)
]
]
;------
; get the 12 least significant bits of a 16 bit value
LSB-12: func [address [string! binary!] ][
as-binary (address AND #{0FFF})
]
;------
i32-to-binary: func [
n [integer!]
/rev
][
n: load join "#{" [form to-hex to-integer n "}"]
either rev [head reverse n][n]
]
;------
; load value at given address. (doesn't clear the opcode).
LVAL: func [addr [binary!]][
to-integer copy/part at RAM ( (to-integer addr) + 1) 2
]
;------
; implement the opcodes which are executed by the CPU
JMP: func [addr][
print ["jumping to " addr]
continue: at RAM ((to-integer addr) + 1) ; 0 based address but 1 based indexing ;-)
]
INC: func [addr][
print ["increment value at address: " addr]
new-val: 1 + LVAL addr
addr: 1 + to-integer addr
bin-val: at (i32-to-binary new-val) 3
change at RAM addr bin-val
]
DEC: func [addr][
print ["decrement value at address: " addr]
]
NOP: func [addr][
print "skipping Nop opcode"
]
;------
; build the bitsets to match op codes
op1: quaternary 1
op2: quaternary 2
op3: quaternary 3
op4: quaternary 4
;------
; build up our CPU emulator grammar
emulator: [
some [
[
here:
[ op1 (op: 'JMP) | op2 (op: 'INC) | op3 (op: 'DEC) | op4 (op: 'NOP)] ; choose op code
:here
copy addr 2 skip (addr: LSB-12 addr) ; get unary op data
continue:
(do reduce [op addr])
:continue
]
| 2 skip (
print ["^/^/^/ERROR: illegal opcode AT: " to-binary here " offset[" -1 + index? here "]"] ; graceful crash!
)
]
]
;------
; generate a bit of binary RAM for our emulator/VM to run...
0 2 4 6 8 ; note ... don't need comments, Rebol just skips them.
RAM: #{2002100540FF30015FFF}
RAM-blowup: { 2 002 1 005 4 0FF 3 001 5 FFF } ; just to make it easier to trace op & data
parse/all RAM emulator
print "^/^/Yes that error is on purpose, I added the 5FFF bytes^/in the 'RAM' just to trigger it :-)^/"
print "notice that it doesn't run the NOP (at address #0006), ^/since we used the JMP opcode to jump over it.^/"
print "also notice that the first instruction is an increment ^/for the address which is jumped (which is misaligned on 'boot')^/"
ask "press enter to continue"
the output is as follows:
increment value at address: #{0002}
jumping to #{0006}
decrement value at address: #{0001}
ERROR: illegal opcode AT: #{5FFF} offset[ 8 ]
Yes that error is on purpose, I added the 5FFF bytes
in the 'RAM' just to trigger it :-)
notice that it doesn't run the NOP (at address #0006),
since we used the JMP opcode to jump over it.
also notice that the first instruction is an increment
for the address which is jumped (which is misaligned on 'boot')
press enter to continue
I've been working on a log viewer for a Rails app and have found that I need to read around 200 lines of a log file from bottom to top instead of the default top to bottom.
Log files can get quite large, so I've already tried and ruled out the IO.readlines("log_file.log")[-200..-1] method.
Are there any other ways to go about reading a file backwards in Ruby without the need for a plugin or gem?
The only correct way to do this that also works on enormous files is to read n bytes at a time from the end until you have the number of lines that you want. This is essentially how Unix tail works.
An example implementation of IO#tail(n), which returns the last n lines as an Array:
class IO
TAIL_BUF_LENGTH = 1 << 16
def tail(n)
return [] if n < 1
seek -TAIL_BUF_LENGTH, SEEK_END
buf = ""
while buf.count("\n") <= n
buf = read(TAIL_BUF_LENGTH) + buf
seek 2 * -TAIL_BUF_LENGTH, SEEK_CUR
end
buf.split("\n")[-n..-1]
end
end
The implementation is a little naive, but a quick benchmark shows what a ridiculous difference this simple implementation can already make (tested with a ~25MB file generated with yes > yes.txt):
user system total real
f.readlines[-200..-1] 7.150000 1.150000 8.300000 ( 8.297671)
f.tail(200) 0.000000 0.000000 0.000000 ( 0.000367)
The benchmark code:
require "benchmark"
FILE = "yes.txt"
Benchmark.bmbm do |b|
b.report "f.readlines[-200..-1]" do
File.open(FILE) do |f|
f.readlines[-200..-1]
end
end
b.report "f.tail(200)" do
File.open(FILE) do |f|
f.tail(200)
end
end
end
Of course, other implementations already exist. I haven't tried any, so I cannot tell you which is best.
There's a module Elif available (a port of Perl's File::ReadBackwards) which does efficient line-by-line backwards reading of files.
Since I'm too new to comment on molf awesome answer I have to post it as a separate answer.
I needed this feature to read log files while they're written , and the last portion of the logs contain the string I need to know it's done and I can start parsing it.
Hence handling small sized files is crucial for me (I might ping the log while it's tiny).
So I enhanced molf code:
class IO
def tail(n)
return [] if n < 1
if File.size(self) < ( 1 << 16 )
tail_buf_length = File.size(self)
return self.readlines.reverse[0..n-1]
else
tail_buf_length = 1 << 16
end
self.seek(-tail_buf_length,IO::SEEK_END)
out = ""
count = 0
while count <= n
buf = self.read( tail_buf_length )
count += buf.count("\n")
out += buf
# 2 * since the pointer is a the end , of the previous iteration
self.seek(2 * -tail_buf_length,IO::SEEK_CUR)
end
return out.split("\n")[-n..-1]
end
end