I'm currently trying to add c14n support for crystal-lang using a c-binding with libxml2. I've successfully been able to use xmlC14NDocSave to save the canonical xml to a file. The problem I'm having is with the xmlOutputBufferPtr for both xmlC14NDocSaveTo and xmlC14NExecute.
The error I receive is (Mac and Linux)
xmlC14NExecute: output buffer encoder != NULL but C14N requires UTF8 output
The documentation states
this buffer MUST have encoder==NULL because C14N requires UTF-8 output
in src/C14N/lib_C14N.cr I have the following code
type CharEncodingHandler = Void*
type Buff = Void*
#type OutputBuffer = Void*
struct OutputBuffer
context : Void*
writecallback : OutputWriteCallback
closecallback : OutputCloseCallback
encoder : CharEncodingHandler
buffer : Buff
conv : Buff
written : Int32
error : Int32
end
....
fun xmlC14NDocSaveTo(doc : LibXML::Node*, nodes : LibXML::NodeSet*, mode : Mode, inclusive_ns_prefixes : UInt8**, with_comments : Int32, buf : OutputBuffer*) : Int32
fun xmlC14NExecute(doc : LibXML::Node*, is_visible_callback : IsVisibleCallback, user_data : Void*, mode : Mode, inclusive_ns_prefixes : UInt8**, with_comments : Int32, buf : OutputBuffer*) : Int32
In src/C14N.cr
output = XML::LibC14N::OutputBuffer.new
p output.encoder
XML::LibC14N.xmlC14NDocSaveTo(#xml, nil, #mode, nil, 0, out output)
The results of the p ouput.encoder are Pointer(Void).null so It appears the value is null.
The c14n.c function is just checking null on the buf->encoder struct
if (buf->encoder != NULL) {
xmlGenericError(xmlGenericErrorContext,
"xmlC14NExecute: output buffer encoder != NULL but C14N requires UTF8 output\n");
return (-1);
}
Any help would be appreciated, the code can be found on my github account. clone and run crystal spec
Don't specify out output, that simply reserves a block of memory of the size of the struct on the stack and passes a pointer to it. As of Crystal 0.7.6 it's not zeroed it out, so you pass garbage.
Using output = XML::LibC14N::OutputBuffer.new is already the first right step, since that does zero out the memory. Now to pass it, simply replace out output with pointerof(output).
Related
I need some help converting a luajit pointer to a string and back.
First I define the ctype:
ffi.cdef[[
typedef struct {
unsigned char Bytes[16];
} EncryptionKeys[100000000];
void* malloc(size_t);
void free(void*);
]]
Then use malloc to allocate some memory and then create the 'EncryptionKeys' variable.
local EncryptionKeyMemoryAddress = ffi.C.malloc(ffi.sizeof("EncryptionKeys"))
local EncryptionKeys = ffi.cast("EncryptionKeys(&)", EncryptionKeyMemoryAddress)
I first convert the variable into a lua string using:
ffi.string(EncryptionKeyMemoryAddress)
But I can't figure out how to convert it back!
Can someone please help me?
FYI: I am passing the 'EncryptionKeyMemoryAddress' variable to one of the function parameters for a lua lane (https://lualanes.github.io/lanes/).
Edit:
Here is the section of code that I am working on:
This is for the client managers module of my server that manages a list of lua states that all have access to any clients connected to the server. They all use a shared section of memory that I want them to have access to using a pointer.
local ClientFFIString = [[
typedef struct {
unsigned char Bytes[16];
} EncryptionKeys[100000000];
void* malloc(size_t);
void free(void*);
]]
ffi.cdef(Matchpools.FFIString)
local EncryptionKeyMemoryAddress = ffi.C.malloc(ffi.sizeof("EncryptionKeys"))
--------------------------------------------
function ClientManagers.CreateNewClientManager()
local EncryptionKeys = ffi.cast("EncryptionKeys(&)", EncryptionKeyMemoryAddress)
EncryptionKeys[0].Bytes[0] = 24
print("___a", EncryptionKeys[0].Bytes[0])
local NewIndex = #ClientManagers.List+1
ClientManagers.List[NewIndex] = ClientManagerFunc(
ClientFFIString,
ffi.string(EncryptionKeysMemoryAddress)
)
end
--------------------------------------------
local ClientManagerFunc = Lanes.gen("*", function(ClientFFIString, EncryptionKeysMemoryAddress)
ffi = require("ffi")
ffi.cdef(ClientFFIString)
local EncryptionKeys = ffi.cast("EncryptionKeys(&)", EncryptionKeyMemoryAddress)
print("___a", EncryptionKeys[0].Bytes[0])
-- I want this to be 24 just like it is in the function that created this lua state
local ClientManagerRunning = true
while ClientManagerRunning do
--local dt = GetDt()
--UpdateClientData(dt)
--UpdateMatchmaking(dt)
end
end)
You can convert Lua string to your structure pointer (and later use it as an array):
ffi.cdef"typedef struct {unsigned char Bytes[16];} Key;"
ptr=ffi.cast("Key *", your_string)
print("First Byte of the First Key in your Array:", ptr[0].Bytes[0])
UPDATE:
Let's test how it works for an array containing three keys:
local ffi = require'ffi'
ffi.cdef"typedef struct {unsigned char Bytes[16];} Key;"
local your_string = string.char(11):rep(16)..string.char(22):rep(16)..string.char(33):rep(16)
local ptr=ffi.cast("Key *", your_string)
print("First Byte of the First Key in your Array:", ptr[0].Bytes[0])
print("First Byte of the Second Key in your Array:", ptr[1].Bytes[0])
print("First Byte of the Third Key in your Array:", ptr[2].Bytes[0])
It prints 11, 22, 33
UPDATE 2:
Pass the address of buffer instead of the content of buffer
In the main thread
-- allocate the buffer
local Keys = ffi.cast("EncryptionKeys&", ffi.C.malloc(ffi.sizeof("EncryptionKeys")))
-- write to the buffer
Keys[2].Bytes[5] = 42
-- create string containing 64-bit address
local string_to_send = tostring(ffi.cast("uint64_t", Keys))
-- Send string_to_send to a lane
Inside the lane
-- receive the string
local received_string = .....
-- restore the buffer pointer
local Keys = ffi.cast("EncryptionKeys&", loadstring("return "..received_string)())
-- read the data from the buffer
print(Keys[2].Bytes[5])
In F# i'm using an external DLL (in this case SDL Graphics library) I'm importing the method I require as follows...
[<DllImport("SDL2.dll", CallingConvention = CallingConvention.Cdecl)>]
extern int SDL_QueryTexture(nativeint texture, uint32& format, int& access, int& w, int& h)
This works fine and I can successfully call the method using the following...
let result = SDLDefs.SDL_QueryTexture(textTexture, &format, &access, &w, &h)
The problem is that the native SDL methods accept null values for many pointer arguments. This is required in some scenarios (which function like overloaded methods). I can't find any way to call these methods from F# passing nulls.
For example, this fails with "does not have null as proper value"
let result = SDLDefs.SDL_QueryTexture(textTexture, &format, null, &w, &h)
I read about the attribute [AllowNullLiteral] but it seems like I can only apply it to types I define, and not pre-defined types which are used in my imported DLL.
Is there any way I can do this?
If you want to specify nulls, you need to use "raw pointers", which are represented by types nativeint and nativeptr<T>.
[<DllImport("SDL2.dll", CallingConvention = CallingConvention.Cdecl)>]
extern int SDL_QueryTexture(nativeint texture, uint32& format, nativeint access, int& w, int& h)
// Call without null
let access = 42
let pAccess = NativePtr.stackalloc<int> 1
NativePtr.write pAccess access
SQL_QueryTexture( textTexture, &format, NativePtr.toNativeInt pAccess, &w, &h )
let returnedAccess = NativePtr.read pAccess
// Call with null
SQL_QueryTexture( textTexture, &format, null, &w, &h )
NOTE: be careful with stackalloc. Allocating memory on the stack is quite handy, because you don't need to explicitly release it, but pointers to it will become invalid once you exit the current function. So you can only pass such pointers to an external function if you're sure that the function won't store the pointer and try to use it later.
If you need to pass a pointer to real heap memory that's not going anywhere, you'll need Marshal.AllocHGlobal. But don't forget to release! (or else :-)
let access = 42
let pAccess = Marshal.AllocHGlobal( sizeof<int> )
NativePtr.write (NativePtr.ofNativeInt pAccess) access
SQL_QueryTexture( textTexture, &format, pAccess, &w, &h )
Marshal.FreeHGlobal( pAccess )
It always return a String which is (at least I guess) the table identifier someone can help in anyway?
Thats my function:
function listFiles(dir)
local ffi = require("ffi")
ffi.cdef[[char ** PHYSFS_enumerateFiles ( const char * dir );]]
local liblove = ffi.os == "Windows" and ffi.load("love") or ffi.C
local tb={}
tb=liblove.PHYSFS_enumerateFiles(dir)
return tb
end
It should return me a String with filecontents of the "Dir" I pass to it, but it doesnt. Can't figure out why.
You should read the reference properly. The enumeration function returns a pointer to string pointers, after the last string follows a NULL pointer. Conversion from char* to Lua string can be done with ffi.string.
I have the following code:
MagickWand *wand = NewMagickWand();
char* cmdargs[] = {
"compare",
"receipt-expected.png",
"-metric",
"psnr",
"difference.png",
"difference2.png",
NULL
};
int argcount = 6;
// Allocate memory for MagickCommand
ImageInfo * info = AcquireImageInfo();
ExceptionInfo* e = AcquireExceptionInfo();
// Execute command
char *metadata = NULL;
MagickBooleanType status = MagickCommandGenesis(info, CompareImageCommand, argcount, cmdargs, &metadata, e);
status is 0, which I assume it working because it has no error and the command works correctly in the CLI.
How do I get the metric it has produced? meta is NULL.
$ compare receipt-expected.png -metric psnr difference.png difference2.png
15.4169
Ideally you would access the API directly, rather than attempting to call a new ImageMagick process as a subprocess.
MagickWand * alpha, * beta, * result;
// ... Allocated & Init `alpha' & `beta'
double metric;
result = MagickCompareImages(alpha,
beta,
PeakSignalToNoiseRatioMetric,
&metric);
printf("psnr = %f\n", metric);
How do I get the metric it has produced?
You can not, as metadata is intended to hold additional IO information in a heap. In this instance, any information written to char ** metadata will be destroyed immediately after the internal command writes to standard output. See here for reference.
This function returns a ctypes.unsigned_char.array() and I do read string on it. It is getting the titles of windows. The problem is sometimes it throws TypeError.
try {
console.error('straight readString on XWindowGetProperty data:', rez_GP.data.readString());
} catch (ex) {
console.error('ex on straight readString:', ex);
}
Please notice the rez_GP.data.readString()
For example this instance: TypeError: malformed UTF-8 character sequence at offset 48. In this situation the window title is Editing js-macosx/bootstrap.js at base-template · Noitidart/js-macosx - Mozilla Firefox The 48th offset is that dot chracter you see, it's chracter code is 183. How to do readString() on this buffer without getting this error?
Thanks
readString expects a utf-8 encoded string. This is true for strings returned by _NET_WM_NAME, but not for WM_NAME.
I found a way to read the string propertly even ifs not utf-8, but im not sure if its he best way or the recommended way. This works though, i have to cast it to unsigned_char (must be this, so not char or jschar) then do fromCharCode:
function readAsChar8ThenAsChar16(stringPtr, known_len, jschar) {
// when reading as jschar it assumes max length of 500
// stringPtr is either char or jschar, if you know its jschar for sure, pass 2nd arg as true
// if known_len is passed, then assumption is not made, at the known_len position in array we will see a null char
// i tried getting known_len from stringPtr but its not possible, it has be known, i tried this:
//"stringPtr.contents.toString()" "95"
//"stringPtr.toString()" "ctypes.unsigned_char.ptr(ctypes.UInt64("0x7f73d5c87650"))"
// so as we see neither of these is 77, this is for the example of "_scratchpad/EnTeHandle.js at master · Noitidart/_scratchpad - Mozilla Firefox"
// tries to do read string on stringPtr, if it fails then it falls to read as jschar
var readJSCharString = function() {
var assumption_max_len = known_len ? known_len : 500;
var ptrAsArr = ctypes.cast(stringPtr, ctypes.unsigned_char.array(assumption_max_len).ptr).contents; // MUST cast to unsigned char (not ctypes.jschar, or ctypes.char) as otherwise i dont get foreign characters, as they are got as negative values, and i should read till i find a 0 which is null terminator which will have unsigned_char code of 0 // can test this by reading a string like this: "_scratchpad/EnTeHandle.js at master · Noitidart/_scratchpad - Mozilla Firefox" at js array position 36 (so 37 if count from 1), we see 183, and at 77 we see char code of 0 IF casted to unsigned_char, if casted to char we see -73 at pos 36 but pos 77 still 0, if casted to jschar we see chineese characters in all spots expect spaces even null terminator is a chineese character
console.info('ptrAsArr.length:', ptrAsArr.length);
//console.log('debug-msg :: dataCasted:', dataCasted, uneval(dataCasted), dataCasted.toString());
var charCode = [];
var fromCharCode = []
for (var i=0; i<ptrAsArr.length; i++) { //if known_len is correct, then will not hit null terminator so like in example of "_scratchpad/EnTeHandle.js at master · Noitidart/_scratchpad - Mozilla Firefox" if you pass length of 77, then null term will not get hit by this loop as null term is at pos 77 and we go till `< known_len`
var thisUnsignedCharCode = ptrAsArr.addressOfElement(i).contents;
if (thisUnsignedCharCode == 0) {
// reached null terminator, break
console.log('reached null terminator, at pos: ', i);
break;
}
charCode.push(thisUnsignedCharCode);
fromCharCode.push(String.fromCharCode(thisUnsignedCharCode));
}
console.info('charCode:', charCode);
console.info('fromCharCode:', fromCharCode);
var char16_val = fromCharCode.join('');
console.info('char16_val:', char16_val);
return char16_val;
}
if (!jschar) {
try {
var char8_val = stringPtr.readString();
console.info('stringPtr.readString():', char8_val);
return char8_val;
} catch (ex if ex.message.indexOf('malformed UTF-8 character sequence at offset ') == 0) {
console.warn('ex of offset utf8 read error when trying to do readString so using alternative method, ex:', ex);
return readJSCharString();
}
} else {
return readJSCharString();
}
}