Convert floating point values to hexadecimal - lua

Question

I'm struggling with a floating point to hex conversion in Lua. My application communicates with an old Akai S2000 sampler. The sampler codes two byte messages into four nibble values. The nibbles are in reverse order, hence the most significant nibble is last. There is one parameter that uses a binary encoding of the fraction part of the value. The two MS nibbles are used to encode the integral part of the value and the LS nibbles are used to encode the binary fraction.
Based on this discussion https://bytes.com/topic/c/answers/219928-how-convert-float-hex I have started to implement a Lua algorithm to generate these nibble values from a given parameter value. As I am not that strong with bit calculations I think I am doing many things the wrong way. There should be an easier way to compute these values and avoid a lot of my silly if/else hacks.
My code (pasted below) works for the positive numbers but it is a lot harder with the negative fractions.
In my tests I have added to tables with expected values. Each table works like this:
key = give value * 100
value = expected outcome from my algorithm
I.e. the first entry in the positiveNumbers table represents an input value of 0.01 and the expected output for that value is a four byte MemoryBlock containing 02 00 00 00 (The first two bytes represent the fraction and the last two the integral part).
Currently my algorithm fails at -0.94 and I can't hack around it without breaking for some other value.
Is there anybody who is string with bit calculations and that easily can see any noob mistake I have made especially with converting the negative values?
Any help or pointers would be appreciated!
Lua Code:
function float2nibbles(value)
local nibbles = MemoryBlock(4, true)
-- Retreive integral and fraction parts of the given value to be converted
local integ, fract = math.modf(math.abs(value))
-- Calculate the values of the integral part (last two nibbles)
local bi = BigInteger(integ)
if value < 0 then
-- This variable is sometimes added in the negative conversion of the MS nibbles
local lsAdd = 1
if integ == 0 then
lsAdd = 0
end
nibbles:setByte(2, bit.band(bit.bnot(bi:getBitRangeAsInt(0,4)) + lsAdd, 0xF))
nibbles:setByte(3, bit.band(bit.bnot(bi:getBitRangeAsInt(4,4)), 0xF))
else
nibbles:setByte(2, bit.band(bi:getBitRangeAsInt(0,4), 0xF))
nibbles:setByte(3, bit.band(bi:getBitRangeAsInt(4,4), 0xF))
end
-- Calculate the values of the fraction (first two nibbles)
local remainder = fract
local prevRemain = 0
for i = 1,2 do
remainder = remainder * 16
-- Integral part of the remainder
local d = math.modf(remainder)
if value < 0 and fract ~= 0 then
local lsAdd = 1
if fract == 0 or i == 1 then
lsAdd = 0
end
console(string.format("lsAdd %d", lsAdd))
nibbles:setByte(2 - i, bit.band(bit.bnot(d) + lsAdd, 0xF))
else
nibbles:setByte(2 - i, bit.band(d, 0xF))
end
console(string.format("fract %d = %d, %.2f", i, d, remainder))
prevRemain = remainder
remainder = remainder - d
end
-- For some reason this increment helps when the LS nibble should increment the value of the second nibble
if nibbles:getByte(0) == 0 and nibbles:getByte(1) ~= 0 and value < 0 then
console(string.format("weird increment { %d %d }", nibbles:getByte(0), nibbles:getByte(1)))
nibbles:setByte(1, nibbles:getByte(1) + 1)
end
-- The precision of this data is one byte but apparently they seem to use a third increment to check for rounding
remainder = remainder * 16
console(string.format("final remainder %.2f", remainder))
if math.abs(remainder - prevRemain) > 0.001 and remainder > 14 then
console(string.format("overflow -> %.2f (%.2f)", remainder, prevRemain))
if value < 0 then
nibbles:setByte(0, nibbles:getByte(0) - 1)
else
nibbles:setByte(0, nibbles:getByte(0) + 1)
end
end
console(string.format("%.2f : integral part %s (%s), fract %.2f", value, bit.tohex(integ, 2), nibbles:toHexString(1), fract))
return nibbles
end
local positiveNumbers = {
"02 00 00 00",
"05 00 00 00",
"07 00 00 00",
"0A 00 00 00",
"0C 00 00 00",
"0F 00 00 00",
"02 01 00 00",
"04 01 00 00",
"07 01 00 00",
"09 01 00 00",
"0C 01 00 00",
"0E 01 00 00",
"01 02 00 00",
"03 02 00 00",
"06 02 00 00",
"09 02 00 00",
"0B 02 00 00",
"0E 02 00 00",
"00 03 00 00",
"03 03 00 00",
"05 03 00 00",
"08 03 00 00",
"0B 03 00 00",
"0D 03 00 00",
"00 04 00 00",
"02 04 00 00",
"05 04 00 00",
"07 04 00 00",
"0A 04 00 00",
"0C 04 00 00",
"0F 04 00 00",
"02 05 00 00",
"04 05 00 00",
"07 05 00 00",
"09 05 00 00",
"0C 05 00 00",
"0E 05 00 00",
"01 06 00 00",
"03 06 00 00",
"06 06 00 00",
"09 06 00 00",
"0B 06 00 00",
"0E 06 00 00",
"00 07 00 00",
"03 07 00 00",
"05 07 00 00",
"08 07 00 00",
"0B 07 00 00",
"0D 07 00 00",
"00 08 00 00",
"02 08 00 00",
"05 08 00 00",
"07 08 00 00",
"0A 08 00 00",
"0C 08 00 00",
"0F 08 00 00",
"02 09 00 00",
"04 09 00 00",
"07 09 00 00",
"09 09 00 00",
"0C 09 00 00",
"0E 09 00 00",
"01 0A 00 00",
"03 0A 00 00",
"06 0A 00 00",
"09 0A 00 00",
"0B 0A 00 00",
"0E 0A 00 00",
"00 0B 00 00",
"03 0B 00 00",
"05 0B 00 00",
"08 0B 00 00",
"0B 0B 00 00",
"0D 0B 00 00",
"00 0C 00 00",
"02 0C 00 00",
"05 0C 00 00",
"07 0C 00 00",
"0A 0C 00 00",
"0C 0C 00 00",
"0F 0C 00 00",
"02 0D 00 00",
"04 0D 00 00",
"07 0D 00 00",
"09 0D 00 00",
"0C 0D 00 00",
"0E 0D 00 00",
"01 0E 00 00",
"03 0E 00 00",
"06 0E 00 00",
"09 0E 00 00",
"0B 0E 00 00",
"0E 0E 00 00",
"00 0F 00 00",
"03 0F 00 00",
"05 0F 00 00",
"08 0F 00 00",
"0B 0F 00 00",
"0D 0F 00 00",
"00 00 01 00"
}
local negativeNumbers = {
"0E 0F 0F 0F",
"0B 0F 0F 0F",
"09 0F 0F 0F",
"06 0F 0F 0F",
"04 0F 0F 0F",
"01 0F 0F 0F",
"0E 0E 0F 0F",
"0C 0E 0F 0F",
"09 0E 0F 0F",
"07 0E 0F 0F",
"04 0E 0F 0F",
"02 0E 0F 0F",
"0F 0D 0F 0F",
"0D 0D 0F 0F",
"0A 0D 0F 0F",
"07 0D 0F 0F",
"05 0D 0F 0F",
"02 0D 0F 0F",
"00 0D 0F 0F",
"0D 0C 0F 0F",
"0B 0C 0F 0F",
"08 0C 0F 0F",
"05 0C 0F 0F",
"03 0C 0F 0F",
"00 0C 0F 0F",
"0E 0B 0F 0F",
"0B 0B 0F 0F",
"09 0B 0F 0F",
"06 0B 0F 0F",
"04 0B 0F 0F",
"01 0B 0F 0F",
"0E 0A 0F 0F",
"0C 0A 0F 0F",
"09 0A 0F 0F",
"07 0A 0F 0F",
"04 0A 0F 0F",
"02 0A 0F 0F",
"0F 09 0F 0F",
"0D 09 0F 0F",
"0A 09 0F 0F",
"07 09 0F 0F",
"05 09 0F 0F",
"02 09 0F 0F",
"00 09 0F 0F",
"0D 08 0F 0F",
"0B 08 0F 0F",
"08 08 0F 0F",
"05 08 0F 0F",
"03 08 0F 0F",
"00 08 0F 0F",
"0E 07 0F 0F",
"0B 07 0F 0F",
"09 07 0F 0F",
"06 07 0F 0F",
"04 07 0F 0F",
"01 07 0F 0F",
"0E 06 0F 0F",
"0C 06 0F 0F",
"09 06 0F 0F",
"07 06 0F 0F",
"04 06 0F 0F",
"02 06 0F 0F",
"0F 05 0F 0F",
"0D 05 0F 0F",
"0A 05 0F 0F",
"07 05 0F 0F",
"05 05 0F 0F",
"02 05 0F 0F",
"00 05 0F 0F",
"0D 04 0F 0F",
"0B 04 0F 0F",
"08 04 0F 0F",
"05 04 0F 0F",
"03 04 0F 0F",
"00 04 0F 0F",
"0E 03 0F 0F",
"0B 03 0F 0F",
"09 03 0F 0F",
"06 03 0F 0F",
"04 03 0F 0F",
"01 03 0F 0F",
"0E 02 0F 0F",
"0C 02 0F 0F",
"09 02 0F 0F",
"07 02 0F 0F",
"04 02 0F 0F",
"02 02 0F 0F",
"0F 01 0F 0F",
"0D 01 0F 0F",
"0A 01 0F 0F",
"07 01 0F 0F",
"05 01 0F 0F",
"02 01 0F 0F",
"00 01 0F 0F",
"0D 00 0F 0F",
"0B 00 0F 0F",
"08 00 0F 0F",
"05 00 0F 0F",
"03 00 0F 0F",
"00 00 0F 0F"
}
function verifyFloat2Nibbles(value, expectedMemBlock)
local temp = string.upper(float2nibbles(value):toHexString(1))
assert(expectedMemBlock == temp,
string.format("Incorrect result for %.2f, expected %s, got %s", value, expectedMemBlock, temp))
end
for k,v in pairs(positiveNumbers) do
verifyFloat2Nibbles(k / 100, v)
end
for k,v in pairs(negativeNumbers) do
verifyFloat2Nibbles((k / 100) * -1, v)
end

Answer 1

function float2nibbles(value)
local nibbles = MemoryBlock(4, true)
local n = math.floor(math.abs(value)*256 + 0.13)
n = value < 0 and 0x10000 - n or n
for pos = 0, 3 do
nibbles:setByte(pos, n%16)
n = math.floor(n/16)
end
return nibbles
end

Answer 2

One problem you have is that when you negate, you are not propagating the carry from one nibble to the next. The code is way too complicated to review and fix. Here's a redo:
n2a = {[0]='0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f'}
function f2h (f)
-- TODO should range check f
local scaledf = f * 256 -- move binary point 8 bits right
-- TODO optionally you could also round scaledf here
local scaledi = math.modf(scaledf) -- integer part
-- convert to nibbles
local scalediparts = { (0xf & scaledi),
(0xf & (scaledi >> 4)),
(0xf & (scaledi >> 8)),
(0xf & (scaledi >> 12)) }
-- output in hex
print(n2a[scalediparts[1]], n2a[scalediparts[2]], n2a[scalediparts[3]], n2a[scalediparts[4]])
-- return the nibbles
return scalediparts
end
and a console test:
> f2h(-0.01)
e f f f
table: 0x7feaf9d06390
> f2h(-0.94)
0 1 f f
table: 0x7feaf9e009a0
> f2h(0.01)
2 0 0 0
table: 0x7feaf9d06410
>
This is using Lua 5.3 bit operators; conversion to the bit library is straightforward. You'll also need to modify the print statement and return value to match your preference.