luajit ffi string garbage-collected - lua

I'm trying to find a way to prevent my strings from being collected by the gc. I'm using a small struct and it is populated using the snippet below. Ideally the table itself would move to an ffi struct but I not there yet (and I suspect that then all my structs will be gc-ed). I can go-around using static sizes but considering the amount of records and other similar structures I may move to ffi, this would be a large waste of RAM).
local ffi = require("ffi")
ffi.cdef[[
typedef struct {
uint8_t ctype;
uint16_t freq;
char* name;
char* icao;
} s_frq;
]]
local s_frq = ffi.typeof("s_frq")
[...]
local frq = s_frq()
frq.ctype = tonumber(fields[1])
frq.freq = freq
frq.icao = ffi.new("char[?]", #icao+1, icao)
frq.name = ffi.new("char[?]", #name+1, name)
db_apt[icao].freqs[#db_apt[icao].freqs+1] = frq
I've read somewhere that:
The latter. Pointers are not followed by the GC. Only references from
local variables, upvalues, Lua tables etc. are considered. This also
means you must not store the only reference to an object allocated
with ffi.new() as a pointer in a struct field.
but then I cannot figure out where I should store my pointers. Creating a lua table to store them side by side with my ffi struc would defeat the purpose of not using large lua tables (because the host application - X-Plane - will crash if receiving an alloc request bigger than 32MB).

Related

Read of memory allocation returns spurious results if, following read, free() is called - why does this happen? (embedded)

Programming on a stm32f4 some strange behaviour is observed:
Data is allocated using realloc, which is called every second or so, as such; ptr = realloc(ptr, sizeof)
Values are read into the data - it has been confirmed that: A) The indexing of the array is correct and B) Immediately following each read of values into memory the array holds the correct values.
Upon reading the array the code fails to produce proper output (outputs 0s the vast majority of the time) if free(ptr) is called in any code following the read. When free(ptr) is not called the code functions properly. It seems that the sequential nature of C breaks down in this instance?
Immediately following each read of values into memory the array holds the correct values regardless of any 'free' calls. Realloc is used because this interrupt is called repeatedly. The 'random pointer' has been set to NULL when initialised, before the pointer is realloced. This is an embedded program on a stm32f4.
Being inexperienced with embedded c I can only speculate, but imagine the cause may be faulty optimisation?
Is this behaviour known? I am aware that it is best practice to avoid malloc ect but due to the large variances in amounts of data potentially being held in this application the flexibility is required.
The code mallocs using pointers contained within a global struct. The following code is the offending material:
structContainingMemoryPointer storedData;
numberOfInts = 0;
// ***********Getdata if interrupt conditions state to do so - contained within interrupt***********
interrupt {
if (SpecificInterrupt) {
numberOfInts++;
storedData.Arrayptr =
realloc(storedData.Arrayptr,
sizeof(int) * storedData.numberOfInts * 2);
// Store the value of actualTemp
storedData.Arrayptr[storedData.numberOfInts - 1] = actualTemp;
// Step through the temperature values array and send to USART
for (arrayStep = 0; arrayStep < storedData.numberOfTempAllocations;
arrayStep++) {
// Convert to string and send
sprintf(valueString, ":%d", storedData.temperature[arrayStep]);
USART_puts(USART2, valueString);
}
}
// ***********free memory*************
free(storedDataStruct.Arrayptr);
storedDataStruct.Arrayptr = NULL;
// End of program, no return from this point to previous points.

mlpack sparse coding solution not found

I am trying to learn how to use the Sparse Coding algorithm with the mlpack library. When I call Encode() on my instance of mlpack::sparse_coding:SparseCoding, I get the error
[WARN] There are 63 inactive atoms. They will be reinitialized randomly.
error: solve(): solution not found
Is it simply that the algorithm cannot learn a latent representation of the data. Or perhaps it is my usage? The relevant section follows
EDIT: One line was modified to fix an unrelated error, but the original error remains.
double* Application::GetSparseCodes(arma::mat* trainingExample, int atomCount)
{
double* latentRep = new double[atomCount];
mlpack::sparse_coding::SparseCoding<mlpack::sparse_coding::DataDependentRandomInitializer> sc(*trainingExample, Utils::ATOM_COUNT, 1.0);
sc.Encode(Utils::MAX_ITERATIONS);
arma::mat& latentRepMat = sc.Codes();
for (int i = 0; i < atomCount; i++)
latentRep[i] = latentRepMat.at(i, 0);
return latentRep;
}
Some relevant parameters
const static int IMAGE_WIDTH = 20;
const static int IMAGE_HEIGHT = 20;
const static int PIXEL_COUNT = IMAGE_WIDTH * IMAGE_HEIGHT;
const static int ATOM_COUNT = 64;
const static int MAX_ITERATIONS = 100000;
This could be one of a handful of issues but given the description it's a little difficult to tell which of these it is (or if it is something else entirely). However, these three ideas should provide a good place to start:
Matrices in mlpack are column-major. That means each observation should represent a column. If you use mlpack::data::Load() to load, e.g., a CSV file (which are generally one row per observation), it will automatically transpose the dataset. SparseCoding will act oddly if you pass it transposed data. See also http://www.mlpack.org/doxygen.php?doc=matrices.html.
If there are 63 inactive atoms, then only one atom is actually active (given that ATOM_COUNT is 64). This means that the algorithm has found that the best way to represent the dictionary (at a given step) uses only one atom. This could happen if the matrix you are passing consists of all zeros.
mlpack will provide verbose output, which may also be helpful for debugging. Usually this is used by using mlpack's CLI class to parse command-line input, but you can enable verbose output with mlpack::Log::Info.ignoreInput = false. You may obtain a lot of output that way, but it will give a better look at what is going on...
The mlpack project has its own mailing list where you may be likely to get a quicker or more comprehensive response, by the way.

Reading and Writing Structs to and from Arduino's EEPROM

I'm trying to write data structures defines in C to my Arduino Uno board's non-volatile memory, so the values of the struct will be retained after the power goes off or it is reset.
To my understanding, the only way to do this (while the sketch is running) would be to write to arduino's EEPROM. Although I can write individual bytes (sets a byte with value 1 at address 0):
eeprom_write_byte(0,1);
I am stuck trying to write a whole struct:
typedef struct NewProject_Sequence {
NewProject_SequenceId sequenceId;
NewProject_SequenceLength maxRange;
NewProject_SequenceLength minRange;
NewProject_SequenceLength seqLength;
NewProject_SceneId sceneList[5];
} NewProject_Sequence;
Because of the EEPROM's limit of 100,000 writes, I don't want to write to the Arduino in a loop going through each byte, for this will probably use it up pretty fast. Does anyone know a more efficient way of doing this, either with EEPROM or if there's a way to write to PROGMEM while the sketch is running? (without using the Arduino Library, just C).
RESOLVED
I ended up writing two custom functions -- eepromWrite and eepromRead. They are listed below:
void eepromRead(uint16_t addr, void* output, uint16_t length) {
uint8_t* src;
uint8_t* dst;
src = (uint8_t*)addr;
dst = (uint8_t*)output;
for (uint16_t i = 0; i < length; i++) {
*dst++ = eeprom_read_byte(src++);
}
}
void eepromWrite(uint16_t addr, void* input, uint16_t length) {
uint8_t* src;
uint8_t* dst;
src = (uint8_t*)input;
dst = (uint8_t*)addr;
for (uint16_t i = 0; i < length; i++) {
eeprom_write_byte(dst++, *src++);
}
}
The would be implemented like this:
uint16_t currentAddress;
struct {
uint16_t x;
uint16_t y;
} data;
struct {
} output;
uint16_t input
eepromWrite(currentAddress, data, sizeof(data);
eepromRead(currentAddress, output, sizeof(data));
Several solutions and or combinations.
setup a timer event to store the values periodically, rather then
back to back.
use a checksum, then increment the initial offset,
when writing. Where when reading you attempt each increment until
you have a valid checksum. this spreads your data across the entire
range increasing your life. modern flash drives do this.
Catch the unit turning off, by using an external Brown Out Detector to trigger an INT to then quickly write the EEPROM. Where you can then also use the internal BOD to prevent corruption, before it falls below safe writing voltages. By having the external significantly higher than the internal thresholds. The time to write before complete shutdown can be increased by increasing the VCC capacitance. Where the external BOD is compared before the VCC and not directly the VCC itself.
Here is a video explaining how to enable the internal BOD, for a ATtiny, where it is nearly identical for the other ATmega's. Video
The Arduino EEPROM library provides get/put functions that are able to read and write structs...
Link to EEPROM.put(...)
The write is made only when a byte has changed.
So, using put/get is the solution to your problem.
I'm using these in a wide (25k) project without any problem.
And as already said I've used a timer to write not each time but some time to times.
Turning off detection is also a very good way to do this.

Is there a built in routine to convert lightuserdata to int?

Lightuserdata is different from userdata so what can I do with it? I mean the operations of lightuserdata in lua. Looks like I cannot convert it to any other data type.
One of my case:
My C library returns a C pointer named 'c_pointer', AKA lightuserdata to Lua, and then I want:
my_pointer = c_pointer +4
and then pass 'my_pointer' back to C library. Since I cannot do anything with 'c_pointer', so the expression 'c_pointer + 4' is invalid.
I am wondering are there some practical solutions to this?
Lightuserdata are created by C libraries. They are simply C pointers.
For example, you can use them to refer to data you allocate with malloc, or statically allocate in your module. Your C library can transfer these pointers to the Lua side as a lightuserdata using lua_pushlightuserdata, and later Lua can give it back to your library (or other C code) on the stack. Lua code can use the lightuserdata as any other value, storing it in a table, for example, even as a table key.
ADDENDUM
To answer your revised question, if you want to add an offset to the pointer, do it on the C side. Pass the lightuserdata and the integer offset to C, and let C do the offset using ptr[n]
void * ptr = lua_touserdata(L, idx1);
lua_Integer n = lua_tointeger(L. idx2);
// do something with
((char *)ptr)[n];
Plain Lua has no pointer arithmetic, so as Doug Currie stated you would need to do the pointer arithmetic on the C side.
LuaJIT on the other hand can do pointer arithmetic (via the FFI library), so consider using that instead.

How to write a double value byte by byte

I have to communicate with a dll and it lua and this is the function I use to write strings by bytes:
writeString = function(pid, process, address, value)
local i = 1
while i <= String.Length(value) do
local byte = string.byte(value, i, i)
DLL.CallFunction("hook.dll", "writeMemByte", pid..','..process..','..address + (i-1)..','..byte, DLL_RETURN_TYPE_INTEGER, DLL_CALL_CDECL)
i = i + 1
end
DLL.CallFunction("hook.dll", "writeMemByte", pid..','..process..','..address + (i-1)..',0', DLL_RETURN_TYPE_INTEGER, DLL_CALL_CDECL)
end
I basically need to adapt this to write a double value byte by byte.
I just can't think how to make the memory.writeDouble function.
EDIT: this is my readString function:
readString = function(pid, process, address)
local i, str = 0, ""
repeat
local curByte = DLL.CallFunction("hook.dll", "readMemByte", pid..','..process..','..(address + i), DLL_RETURN_TYPE_INTEGER, DLL_CALL_CDECL)
if curByte == "" then curByte = 0 end
curByte = tonumber(curByte)
str = str .. string.char(curByte)
i = i + 1
until (curByte == 0)
return str
end,
My first recommendation would be: try to find a function that accepts strings representing doubles instead of doubles. Implementing the lua side of that would be incredibly easy, since you already have a writeString - it could be something very similar to this:
writeDouble = function(pid, process, address, value)
writeString(pid, process, address, tostring(value))
end
If you don't have that function, but you have access to the dll source, you can try to add that function yourself; it shouldn't be much more complicated than getting the string and then calling atof on it.
If you really can't modify the dll, then you need to figure out the exact double format that the lib is expecting - there are lots of factors that can change that format. The language and compiler used, the operative systems, and the compiler flags, to cite some.
If the dll uses a standard format, like IEE-754, the format will usually have well documented "translations" from/two bites. Otherwise, it's possible that you'll have to develop them yourself.
Regards and good luck!
There are many libraries available for Lua that do just this.
If you need the resulting byte array (string), string.pack should do it; you can find precompiled binaries for Windows included with Lua for Windows.
If you are more interested in using the double to interface with foreign code, I would recommend taking a different approach using alien, a Foreign Function Interface library that lets you directly call C functions.
If you able to, I even more highly recommend switching to LuaJIT, a Just-In-Time compiler for Lua that provides the power, speed and reach of C and assembly, but with the comfort an flexibility of Lua.
If none of these solutions are viable, I can supply some code to serialise doubles (not accessible at the moment).

Resources