Efficiently count how many transparent pixels are in UIImage/CIImage with Metal - ios

Question

What is the fastest way we can count how many transparent pixels exist in CIImage/UIImage?
For example:
My first thought, if we speak about efficiency, is to use Metal Kernel using either CIColorKernel or so, but I can't understand how to use it to output "count".
Also other ideas I had in mind:
Use some kind of average color to calculate it, the "redder" the more filled with pixels? Maybe some kind of linear calculation depends on the image size (using CIAreaAverage CIFilter?
Count pixels one by one and check the RGB values?
Using Metal parallel capabilities, similar to this post: Counting coloured pixels on the GPU - Theory?
Scale down the image and then count? Or do all the other processes suggested above just with scaled than version, and multiple it back depends on the scale down proportions after calculating?
What is the fastest way to achieve this count?

Answer 1

To answer your question how to do it metal, you would use device atomic_int.
Essentially you create an Int MTLBuffer and pass it to your kernel and increment it with atomic_fetch_add_explicit.
Create buffer once:
var bristleCounter = 0
counterBuffer = device.makeBuffer(bytes: &bristleCounter, length: MemoryLayout<Int>.size, options: [.storageModeShared])
Reset counter to 0 and binding counter buffer:
var z = 0
counterBuffer.contents().copyMemory(from: &z, byteCount: MemoryLayout<Int>.size)
kernelEncoder.setBuffer(counterBuffer, offset: 0, index: 0)
Kernel:
kernel void myKernel (device atomic_int *counter [[buffer(0)]]) {}
Increment counter in Kernel (and get the value):
int newCounterValue = atomic_fetch_add_explicit(counter, 1, memory_order_relaxed);
Get the counter on the CPU side:
kernelEncoder.endEncoding()
kernelBuffer.commit()
kernelBuffer.waitUntilCompleted()
//Counter from kernel now in counterBuffer
let bufPointer = counterBuffer.contents().load(as: Int.self)
print("Counter: \(bufPointer)")

Answer 2

What you want to perform is a reduction operation, which is not necessarily well-suited for the GPU due to its massively parallel nature. I'd recommend not writing a reduction operation for the GPU yourself, but rather use some highly optimized built-in APIs that Apple provides (like CIAreaAverage or the corresponding Metal Performance Shaders).
The most efficient way depends a bit on your use case, specifically where the image comes from (loaded via UIImage/CGImage or the result of a Core Image pipeline?) and where you'd need the resulting count (on the CPU/Swift side or as an input for another Core Image filter?).
It also depends on if the pixels could also be semi-transparent (alpha not 0.0 or 1.0).
If the image is on the GPU and/or the count should be used on the GPU, I'd recommend using CIAreaAverage. The alpha value of the result should reflect the percentage of transparent pixels. Note that this only works if there are now semi-transparent pixels.
The next best solution is probably just iterating the pixel data on the CPU. It might be a few million pixels, but the operation itself is very fast so this should take almost no time. You could even use multi-threading by splitting the image up in chunks and use concurrentPerform(...) of DispatchQueue.
A last, but probably overkill solution would be to use Accelerate (this would make #FlexMonkey happy): Load the image's pixel data into a vDSP buffer and use the sum or average methods to calculate the percentage using the CPU's vector units.
Clarification
When I was saying that a reduction operation is "not necessarily well-suited for the GPU", I meant to say that it's rather complicated to implement in an efficient way and by far not as straightforward as a sequential algorithm.
The check whether a pixel is transparent or not can be done in parallel, sure, but the results need to be gathered into a single value, which requires multiple GPU cores reading and writing values into the same memory. This usually requires some synchronization (and thereby hinders parallel execution) and incurs latency cost due to access to the shared or global memory space. That's why efficient gather algorithms for the GPU usually follow a multi-step tree-based approach. I can highly recommend reading NVIDIA's publications on the topic (e.g. here and here). That's also why I recommended using built-in APIs when possible since Apple's Metal team knows how to best optimize these algorithms for their hardware.
There is also an example reduction implementation in Apple's Metal Shading Language Specification (pp. 158) that uses simd_shuffle intrinsics for efficiently communicating intermediate values down the tree. The general principle is the same as described by NVIDIA's publications linked above, though.

Answer 3

If the image contains semitransparent pixels, it can be easily preprocessed to make all pixels with alpha below certain threshold fully transparent, or fully opaque otherwise. Then the CIAreaAverage could be applied, as was originally suggested in the question, and finally the approximate number of the fully opaque pixels can be calculated by multiplying alpha component of the result by the image size.
For pre-processing we could use a trivial CIColorKernel like this:
half4 clampAlpha(coreimage::sample_t color) {
half4 out = half4(color);
out.a = step(half(0.99), out.a);
return out;
}
(Choose whatever threshold you like instead of 0.99)
To get the alpha component out of the output of CIAreaAverage we could do something like this:
let context = CIContext(options: [.workingColorSpace: NSNull(), .outputColorSpace: NSNull()])
var color: [Float] = [0, 0, 0, 0]
context.render(output,
toBitmap: &color,
rowBytes: MemoryLayout<Float>.size * 4,
bounds: CGRect(origin: .zero, size: CGSize(width: 1, height: 1)),
format: .RGBAf,
colorSpace: nil)
// color[3] contains alpha component of the result
With that approach everything is done on GPU while taking advantage of its inherent parallelism.
BTW, check this app out https://apps.apple.com/us/app/filter-magic/id1594986951. It lets you play with every single CoreImage filter out there.