Libfuzzer offers two APIs to develop custom mutators.
size_t LLVMFuzzerCustomMutator(uint8_t *Data, size_t Size, size_t MaxSize, unsigned int Seed)
size_t LLVMFuzzerCustomCrossOver(const uint8_t *Data1, size_t Size1, const uint8_t *Data2, size_t Size2, uint8_t *Out, size_t MaxOutSize, unsigned int Seed)
How are these APIs supposed to be used?
The fuzzer is required to be deterministic. How do I ensure that with the custom mutators?
You just need to implement those functions alongside your LLVMFuzzerTestOneInput.
The google/fuzzing repository has a tutorial on how to implement structure-aware fuzzing.
Also, you could take inspiration from CustomMutatorTest.cpp, and CustomCrossOverTest.cpp, from the LLVM repository.
The fuzzer is required to be deterministic.
Right, but here you will be writing the mutation functions, which are different; Mutations will happen before your LLVMFuzzerTestOneInput is called.
However, they have an analogous requirement.
As outlined in the source code, alongside LLVMFuzzerCustomMutator, and LLVMFuzzerCustomCrossOver respectively:
Optional user-provided custom mutator.
Mutates raw data in [Data, Data+Size) inplace.
Returns the new size, which is not greater than MaxSize.
Given the same Seed produces the same mutation.
Optional user-provided custom cross-over function.
Combines pieces of Data1 & Data2 together into Out.
Returns the new size, which is not greater than MaxOutSize.
Should produce the same mutation given the same Seed.
i.e. Two calls to a mutation function with the same Data and seed should produce the same result.
One last thing: you don't need to implement both functions; LLVMFuzzerCustomMutator should be enough in most cases.
Related
libFuzzer has functions that can be implemented by the end-user like this:
size_t LLVMFuzzerCustomMutator(
uint8_t* data, size_t size, size_t max_size, unsigned int seed)
Am I free to sometimes insert some bytes in data thereby making it larger; I assume max_size may not be exceeded? If I needed more bytes than max_bytes to perform the necessary insertion how would I do that? Do I return the new size?
I have this loop
void f1(unsigned char *data, unsigned int size) {
unsigned int A[256] = {0u};
for (register unsigned int i = 0u; i < size; i++) {
++A[data[i]];
}
...
Is there any way to vectorize it manually?
Since multiple entries in data[i] might contain the same value, I don't see how this could be vectorized simply since there can be race conditions. The point of vectorization is that each element is independent of the other elements, and so can be computed in parallel. But your algorithm doesn't allow that. "Vectorize" is not the same thing as "make go faster."
What you seem to be building here is a histogram, and iOS has built-in, optimized support for that. You can create a single-channel, single-row image and use vImageHistogramCalculation_Planar8 like this:
void f1(unsigned char *data, unsigned int size) {
unsigned long A[256] = {0u};
vImage_Buffer src = { data, 1, size, size };
vImage_Error err = vImageHistogramCalculation_Planar8(&src, A, kvImageDoNotTile);
if (err != kvImageNoError) {
// error
}
...
}
Be careful about assuming this is always a win, though. It depends on the size of your data. Making a function call is very expensive, so it can take several million bytes of data to make it worth it. If you're computing this on smaller sets than that, then a simple, compiler-optimized loop is often the best approach. You need to profile this on real devices to see which is faster for your purposes.
Just make sure to allow the compiler to apply all vectorizing optimizations by turning on -Ofast (Fastest, Aggressive). That won't matter in this case because your loop can't be simply vectorized. But in general, -Ofast allows the compiler to apply vectorizing optimizations in cases that it might slightly grow code size (which isn't allowed under the default -Os). -Ofast also allows a little sloppiness in how floating point math is performed, so should not be used in cases where strict IEEE floating point conformance is required (but this is almost never the case for iOS apps, so -Ofast is almost always the correct setting).
The optimisation the compiler would attempt to do here is to parallelize ++A[data[i]]
It cannot do so because the contents of A depend on the previous iteration of the loop.
You could break this dependancy by using one frequency array (A) per way of parallelism, and then computing the sum of these at the end. I assume here you've got two ways of parallelism and that the size is even.
void f1(const unsigned char * const data, unsigned int size) {
unsigned int A0[256] = {0u};
unsigned int A1[256] = {0u};
for (unsigned int i = 0u; i < size /2u; i++) {
++A0[data[2*i]];
++A1[data[2*i+1]];
}
for (unsigned i=0u; i < 256; ++i){
A0[i] = A0[i] + A1[i];
}
}
Does this win you much? There only one way to find out - try it and measure the results. I suspect that the Accelerate framework will do much better than this, even for relatively small values on size. It's also optimised at run-time for the target architecture.
Compilers are pretty smart, but there are things you can do in C or C++ to help the compiler:
Apply const wherever possible: It's then obvious which data is invariant.
Identify pointers to non-overlapping memory regions with the restrict (__restrict in C++) qualifier. Without knowing this, the compiler must assume a write through one pointer potentially alters data that could be read with another. clang will in fact generate run-time checks and code-paths for both the overlapping and non-overlapping case, but there will be limits to this, and you can probably reduce code-size by being explicit.
I doubt the register qualifier for i makes any difference.
I'm trying to use OpenCV from Go. OpenCV defines a struct CvMat that has a data field:
typedef struct CvMat
{
...
union
{
uchar* ptr;
short* s;
} data;
}
I'm using the go bindings for opencv found here. This has a type alias for CvMat:
type Mat C.CvMat
Now I have a Mat object and I want to access the data field on it. How can I do this? If I try to access _data, it doesn't work. I printed out the fields on the Mat object with the reflect package and got this:
...
{data github.com/lazywei/go-opencv/opencv [8]uint8 24 [5] false}
...
So there is a data field on it, but it's not even the same type. It's an array of 8 uint8s! I'm looking for a uchar* that is much longer than 8 characters. How do I get to this uchar?
The short answer is that you can't do this without modifying go-opencv. There are a few impediments here:
When you import a package, you can only use identifiers that have been exported. In this case, data does not start with an upper case letter, so is not exported.
Even if it was an exported identifier, you would have trouble because Go does not support unions. So instead the field has been represented by a byte array that matches the size of the underlying C union (8 bytes in this case, which matches the size of a 64-bit pointer).
Lastly, it is strongly recommended not to expose cgo types from packages. So even in cases like this where it may be possible to directly access the underlying C structure, I would recommend against it.
Ideally go-opencv would provide an accessor for the information you are after (presumably one that could check which branch of the union is in use, rather than silently returning bad data. I would suggest you either file a bug report on the package (possibly with a patch), or create a private copy with the required modifications if you need the feature right away.
I am studying FLANN, a library for approximate nearest neighbors search.
For the LSH method they represent an object (point in search space), as
an array of unsigned int. I am not sure why they do this, and not
represent a point simply as a double array (which would represent a point
in multi-dimensional vector space). Maybe because LSH is used for binary
features? Can someone share more about the possible use of unsigned int in
this case? Why unsigned int if you only need a 0 and 1 for each feature?
Thanks
Please note that I will refer to the latest FLANN release, i.e. flann-1.8.3 at the time of writing.
For the LSH method they represent an object (point in search space),
as an array of unsigned int
No: this is wrong. The LshIndex class includes a buildIndexImpl method that implements the LSH indexing. Since the LSH is basically a collection of hash tables, the effective indexing occurs on the LshTable class.
The elementary indexing method, i.e. the method that indexes one feature vector (aka descriptor, or point) at a time is:
/** Add a feature to the table
* #param value the value to store for that feature
* #param feature the feature itself
*/
void add(unsigned int value, const ElementType* feature) {...}
Note: the buildIndexImpl method uses the alternative version that simply iterates over the features, and call the above method on each.
As you can see this method has 2 arguments which is a pair (ID, descriptor):
value which is unsigned int represents the feature vector unique numerical identifier (aka feature index)
feature represents the feature vector itself
If you look at the implementation you can see that the first step consists in hashing the descriptor value to obtain the related bucket key (= the identifier of the slot pointing to the bucket in which this descriptor ID will be stored):
BucketKey key = getKey(feature);
In practice the getKey hashing function is only implemented for binary descriptors, i.e. descriptors that can be represented as an array of unsigned char:
// Specialization for unsigned char
template<>
inline size_t LshTable<unsigned char>::getKey(const unsigned char* feature) const {...}
Maybe because LSH is used for binary features?
Yes: as stated above, the FLANN LSH implementation works in the Hamming space for binary descriptors.
If you were to use descriptors with real values (in R**d) you should refer to the original paper that includes details about how to convert the feature vectors into binary strings so as to use the Hamming space and hash functions.
Can someone share more about the possible use of unsigned int in this
case? Why unsigned int if you only need a 0 and 1 for each feature?
See above: the unsigned int value is only used to store the related ID of each feature vector.
I've read that the signed char and unsigned char types are not guaranteed to be 8 bits on every platform, but sometimes they have more than 8 bits.
If so, using OpenCv how can we be sure that CV_8U is always 8bit?
I've written a short function which takes a 8 bit Mat and happens to convert, if needed, CV_8SC1 Mat elements into uchars and CV_8UC1 into schar.
Now I'm afraid it is not platform independent an I should fix the code in some way (but don't know how).
P.S.: Similarly, how can CV_32S always be int, also on machine with no 32bit ints?
Can you give a reference of this (I've never heard of that)? Probably you mean the padding that may be added at the end of a row in a cv::Mat. That is of no problem, since the padding is usually not used, and especially no problem if you use the interface functions, e.g. the iterators (c.f.). If you would post some code, we could see, if your implementation actually had such problems.
// template methods for iteration over matrix elements.
// the iterators take care of skipping gaps in the end of rows (if any)
template<typename _Tp> MatIterator_<_Tp> begin();
template<typename _Tp> MatIterator_<_Tp> end();
the CV_32S will be always 32-bit integer because they use types like those defined in inttypes.h (e.g. int32_t, uint32_t) and not the platform specific int, long, whatever.