I'm trying to parse *.bin file of Late Data. I found out some meanings of some group of bytes (look at the picture), but some group of bytes I could't veryfied (in red circles). Does anyone knows what are they mean?
Related
So i'm working on a OBJ/GLTF2 converter, and for simplicity i've decided to use one file for every kind of buffer, i have positions.bin (vertex) indices.bin Normals.bin and Uvs.bin the exported files open with windows 10 visualizer but the GLTF validator prints a bunch of MESH_PRIMITIVE_ACCESSOR_WITHOUT_BYTESTRIDE errors.
The file is structured so every buffer binary file have just one view and many accessor with offset (one for each face)
I'm doing something wrong ? or the validator isn't working as expected ? my data is tightly packed so i see no reason to have a ByteStride ...
I haven't an hosting so i'm using we transfer here, sorry for that
Example file
This question has been answered here: https://github.com/KhronosGroup/glTF/issues/1198
To sum up the explainations is that the bytestride can be deducted by the software that reads the GLTF as long as the bufferview isn't shared among accessors, tightly packed data still have the bytestride, it just happen to be equal to the data length and MUST be specified when it can't be deducted.
I'm writing a parser for the SQLi-protocol ("turbo") used by Informix. I have most opcodes covered by now, yet SQ_FETCHBLOB I don't have a clue yet. Reverse engineering the driver is difficult since it copies values from its internal state machine, which itself is hard to track. All I know is that SQ_FETCHBLOB is followed by 56 bytes of data, some of which seem to be the BLOB's total size and fetch-offset.
Does anyone have some information on how to decode SQ_FETCHBLOB as used by Informix SQLi ?
I can't comment on the specifics of the SQ_FETCHBLOB SQLI packet type but you might want to look at the file $INFORMIXDIR/incl/esql/blob.h which is shipped with Client SDK. This describes the tblob_t data structure which is 56 bytes.
A Linux/GFortran question.
I know exactly what my problem is but I can't figure out how to solve it...
I want to import the MNIST dataset images and labels into Fortran arrays to play around with Machine Learning algorithms using Fortran. I've done this with Python but I can't replicate reading the data files with Fortran.
The dataset files and file layout descriptions are at:
http://yann.lecun.com/exdb/mnist/
The 2 problems I'm struggling with are...
1) The data in the files is stored in unsigned bytes. I can't find a similar datatype in Fortran. I'm using integer(kind=1) to read the first 4 bytes successfully, which constitutes the file magic number, but I'm worried about incorrectly reading the value of one of these bytes into the signed integer(kind=1) datatype.
2) The data is stored in Big-Endian format. So when I read the number of images, rows and columns, which are stored in 4 byte integers, into my Little-Endian machine, I receive the obvious gobbledegook. Ideally, what I would like to be able to do is specify the Endiness of a variable to read from a file in an edit descriptor. Is this possible?
Any assistance would be much appreciated.
Kind regards
I have a binary file that I want to parse. The file is broken up into records that are 1024 bytes each. The high level steps needed are:
Read 1024 bytes at a time from the file.
Parse each 1024-byte "record" (chunk) and place the parsed data into a map or struct.
Return the parsed data to the user and any error(s).
I'm not looking for code, just design/approach help.
Due to I/O constraints, I don't think it makes sense to attempt concurrent reads from the file. However, I see no reason why the 1024-byte records can't be parsed using goroutines so that multiple 1024-byte records are being parsed concurrently. I'm new to Go, so I wanted to see if this makes sense or if there is a better (faster) way:
A main function opens the file and reads 1024 bytes at a time into byte arrays (records).
The records are passed to a function that parses the data into a map or struct. The parser function would be called as a goroutine on each record.
The parsed maps/structs are appended to a slice via a channel. I would preallocate the underlying array managed by the slice as the file size (in bytes) divided by 1024 as this should be the exact number of elements (assuming no errors).
I'd have to make sure I don't run out of memory as well, as the file can be anywhere from a few hundred MB up to 256 TB (rare, but possible). Does this make sense or am I thinking about this problem incorrectly? Will this be slower than simply parsing the file in a linear fashion as I read it 1024 bytes at a time, or will parsing these records concurrently as byte arrays perform better? Or am I thinking about the problem all wrong?
I'm not looking for code, just design/approach help.
Cross-posted on Software Engineering
This is an instance of the producer-consumer problem, where the producer is your main function that generates 1024-byte records and the consumers should process these records and send them to a channel so they are added to the final slice. There are a few questions tagged producer-consumer and Go, they should get you started. As for what is fastest in your case, it depends on so many things that it is really not possible to answer. The best solution may be anywhere from a completely sequential implementation to a cluster of servers in which the records are moved around by RabbitMQ or something similar.
I need to read and interpret a binary file containing a TIFF image. I know there exist readers for doing this but I want to go the hard way. I found the TIFF format description and need to parse the binary file in small chunks. Assume I was able to read in memory the complete binary file. This means that I have a variable containing one long list of bytes.
I know via the format definition what the meaning is of the different groups of n bytes.
How can one define character variables with different lengths (sometimes 2, sometimes 3, sometimes 4 etc.) so that the variable address points to the right position in the image variable array?
With other words, assume my image is loaded into an array Image containing all bytes of the file.
The first 2 bytes I want to load in a string with length 2 bytes so that I can just link the address pointer to the first position in the Image array and automatically the first 2 bytes are associated with the first character string. A second string of 4 bytes would have another meaning and so I make the address for the second string of 4 bytes point to the 3rd position of the Image array.
Is this feasible in C++? I remember that this was a normal way of working for dynamical memory allocation in Fortran 77 in a simulation code I analysed a long time ago.
Thanks in advance for the hints!
Regards,
Stefan
The C++ language is easily capable of processing TIFF files from a byte array. The idea you have in mind is basically correct, but there a few problems with it. C strings are zero-terminated and the strings which appear in TIFF files are not necessarily zero terminated since their length is specified explicitly. It really is simpler to create a dedicated data structure to hold the TIFF-specific data fields and then parse the binary data into the structure. Your method will immediately run into trouble with the Motorola/Intel byte issue if your machine has the opposite endian-ness.