I wrote the following Metal Core Image Kernel to produce constant red color.
extern "C" float4 redKernel(coreimage::sampler inputImage, coreimage::destination dest)
{
return float4(1.0, 0.0, 0.0, 1.0);
}
And then I have this in Swift code:
class CIMetalRedColorKernel: CIFilter {
var inputImage:CIImage?
static var kernel:CIKernel = { () -> CIKernel in
let bundle = Bundle.main
let url = bundle.url(forResource: "Kernels", withExtension: "ci.metallib")!
let data = try! Data(contentsOf: url)
return try! CIKernel(functionName: "redKernel", fromMetalLibraryData: data)
}()
override var outputImage: CIImage? {
guard let inputImage = inputImage else {
return nil
}
let dod = inputImage.extent
return CIMetalRedColorKernel.kernel.apply(extent: dod, roiCallback: { index, rect in
return rect
}, arguments: [inputImage])
}
}
As you can see, the dod is given to be the extent of the input image. But when I run the filter, I get a whole red image beyond the extent of the input image (DOD), why? I have multiple filters chained together and the overall size is 1920x1080. Isn't the red filter supposed to run only for DOD rectangle passed in it and produce clear pixels for anything outside the DOD?
With the extent parameter of the kernel call you signal the region for which the kernel produces meaningful results—or, as you correctly named it, the domain of definition.
However, this also means that whatever it produces outside this region is basically undefined and up to you as the kernel developer to decide.
A generator kernel like the one you wrote usually has an infinite domain of definition since it just produces a red color, regardless of the input. To restrict the output to a specific area, you can apply a crop to it:
let dod = inputImage.extent
let result = CIMetalTestRenderer.kernel.apply(extent: .infinite, roiCallback: { index, rect in
return rect
}, arguments: [inputImage])
return result.cropped(to: dod)
After the cropping, everything outside of dod will be transparent.
Update:
It turns out you have to set the extent parameter of the kernel call to .infinite to make this work. I suspect that cropped(to:) checks if the image already has the given extent and will do nothing in this case. So to make CI really apply the cropping, you have to specify the domain of definition your kernel actually produces.
I think the counter-intuitive thing here is that CI does not apply your kernel to just the pixels of the extent you specify. It seems there is some automatic clamp-to-extent going on when the result is not cropped properly, but honestly, I'm also rather confused by this...
Related
I'm trying to port some CIFilter from this source by using metal shading language for Core Image.
I have a palette of color composed by an array of RGB struct and I want to pass them as an argument to a custom CI color image kernel.
The RGB struct is converted into an array of SIMD3<Float>.
static func SIMD3Palette(_ palette: [RGB]) -> [SIMD3<Float>] {
return palette.map{$0.toFloat3()}
}
The kernel should take and array of simd_float3 values, the problem is the when I launch the filter it tells me that the argument at index 1 is expecting an NSData.
override var outputImage: CIImage? {
guard let inputImage = inputImage else
{
return nil
}
let palette = EightBitColorFilter.palettes[Int(inputPaletteIndex)]
let extent = inputImage.extent
let arguments = [inputImage, palette, Float(palette.count)] as [Any]
let final = colorKernel.apply(extent: extent, arguments: arguments)
return final
}
This is the kernel:
float4 eight_bit(sample_t image, simd_float3 palette[], float paletteSize, destination dest) {
float dist = distance(image.rgb, palette[0]);
float3 returnColor = palette[0];
for (int i = 1; i < floor(paletteSize); ++i) {
float tempDist = distance(image.rgb, palette[i]);
if (tempDist < dist) {
dist = tempDist;
returnColor = palette[i];
}
}
return float4(returnColor, 1);
}
I'm wondering how can I pass a data buffer to the kernel since converting it into an NSData seems not enough. I saw some example but they are using "full" shading language that is not available for Core Image that is a sort of subset for dealing only with fragments.
Update
We have now figured out how to pass data buffers directly into Core Image kernels. Using a CIImage as described below is not needed, but still possible.
Assuming that you have your raw data as an NSData, you can just pass it to the kernel on invocation:
kernel.apply(..., arguments: [data, ...])
Note: Data might also work, but I know that NSData is an argument type that allows Core Image to cache filter results based on input arguments. So when in doubt, better cast to NSData.
Then in the kernel function, you only need to declare the parameter with an appropriate constant type:
extern "C" float4 myKernel(constant float3 data[], ...) {
float3 data0 = data[0];
// ...
}
Previous Answer
Core Image kernels don't seem to support pointer or array parameter types. Though there seem to be something coming with iOS 13. From the Release Notes:
Metal CIKernel instances support arguments with arbitrarily structured data.
But, as so often with Core Image, there seem to be no further documentation for that…
However, you can still use the "old way" of passing buffer data by wrapping it in a CIImage and sampling it in the kernel. For example:
let array: [Float] = [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0]
let data = array.withUnsafeBufferPointer { Data(buffer: $0) }
let dataImage = CIImage(bitmapData: data, bytesPerRow: data.count, size: CGSize(width: array.count/4, height: 1), format: .RGBAf, colorSpace: nil)
Note that there is no CIFormat for 3-channel images since the GPU doesn't support those. So you either have to use single-channel .Rf and re-pack the values inside your kernel to float3 again, or add some strides to your data and use .RGBAf and float4 respectively (which I'd recommend since it reduces texture fetches).
When you pass that image into your kernel, you probably want to set the sampling mode to nearest, otherwise you might get interpolated values when sampling between two pixels:
kernel.apply(..., arguments: [dataImage.samplingNearest(), ...])
In your (Metal) kernel, you can assess the data as you would with a normal input image via a sampler:
extern "C" float4 myKernel(coreimage::sampler data, ...) {
float4 data0 = data.sample(data.transform(float2(0.5, 0.5))); // data[0]
float4 data1 = data.sample(data.transform(float2(1.5, 0.5))); // data[1]
// ...
}
Note that I added 0.5 to the coordinates so that they point in the middle of a pixel in the data image to avoid ambiguity and interpolation.
Also note that pixel values you get from a sampler always have 4 channels. So even when you are creating your data image with formate .Rf, you'll get a float4 when sampling it (the other values are filled with 0.0 for G and B and 1.0 for alpha). In this case, you can just do
float data0 = data.sample(data.transform(float2(0.5, 0.5))).x;
Edit
I previously forgot to transform the sample coordinate from absolute pixel space (where (0.5, 0.5) would be the middle of the first pixel) to relative sampler space (where (0.5, 0.5) would be the middle of the whole buffer). It's fixed now.
I made it, event if the answer was good and also deploys to lower target the result wasn't exactly what I was expecting. The difference between the original kernel written as a string and the above method to create an image to be used as a source of data were kind of big.
Didn't get exactly the reason, but the image I was passing as a source of the palette was kind of different from the created one in size and color(probably due to color spaces).
Since there was no documentation about this statement:
Metal CIKernel instances support arguments with arbitrarily structured
data.
I tried a lot in my spare time and came up to this.
First the shader:
float4 eight_bit_buffer(sampler image, constant simd_float3 palette[], float paletteSize, destination dest) {
float4 color = image.sample(image.transform(dest.coord()));
float dist = distance(color.rgb, palette[0]);
float3 returnColor = palette[0];
for (int i = 1; i < floor(paletteSize); ++i) {
float tempDist = distance(color.rgb, palette[i]);
if (tempDist < dist) {
dist = tempDist;
returnColor = palette[i];
}
}
return float4(returnColor, 1);
}
Second the palette transformation into SIMD3<Float>:
static func toSIMD3Buffer(from palette: [RGB]) -> Data {
var simd3Palette = SIMD3Palette(palette)
let size = MemoryLayout<SIMD3<Float>>.size
let count = palette.count * size
let palettePointer = UnsafeMutableRawPointer.allocate(
byteCount: simd3Palette.count * MemoryLayout<SIMD3<Float>>.stride,
alignment: MemoryLayout<SIMD3<Float>>.alignment)
let simd3Pointer = simd3Palette.withUnsafeMutableBufferPointer { (buffer) -> UnsafeMutablePointer<SIMD3<Float>> in
let p = palettePointer.initializeMemory(as: SIMD3<Float>.self,
from: buffer.baseAddress!,
count: buffer.count)
return p
}
let data = Data(bytesNoCopy: simd3Pointer, count: count * MemoryLayout<SIMD3<Float>>.stride, deallocator: .free)
return data
}
The first time I tried by appending SIMD3 to the Data object but wasn't working probably due to memory alignment.
Remember to dealloc the memory created after you used it.
Hope to help someone else.
I'm writing CIFilter, but result pixel colors are different than returned values from metal function.
kernel.metal
#include <CoreImage/CoreImage.h>
extern "C" { namespace coreimage {
float4 foo(sample_t rgb){
return float4(0.3f, 0.5f, 0.7f, 1.0f);
}
}
MetalFilter.swift
import CoreImage
class MetalFilter: CIFilter {
private let kernel: CIColorKernel
var inputImage: CIImage?
override init() {
let url = Bundle.main.url(forResource: "default", withExtension: "metallib")!
let data = try! Data(contentsOf: url)
kernel = try! CIColorKernel(functionName: "foo", fromMetalLibraryData: data)
super.init()
}
required init?(coder aDecoder: NSCoder) {
fatalError("init(coder:) has not been implemented")
}
func outputImage() -> CIImage? {
guard let inputImage = inputImage else {return nil}
return kernel.apply(extent: inputImage.extent, arguments: [inputImage])
}
}
When I get outputImage I have these values:
R = 0.58431372549019611
G = 0.73725490196078436
B = 0.85490196078431369
It's some kind of post processing (like pow(x, 1/2.373) after metal function returns values.
Core Image performs color matching two times when you process an image:
From the color space if the input image to the working color space of the CIContext and, in the final rendering step after all filters were applied, from the working color space to the output color space of the context.
Those color spaces are configured with default values that, in my experience, depend on the device (and its display) you are running on. However, you can define both color spaces using the kCIContextWorkingColorSpace and kCIContextOutputColorSpace options when creating your CIContext.
If you set both values to NSNull(), Core Image won't perform any color matching, treating all color values as they are in the image buffers. However, your filter probably has some assumptions on the color space of the input samples. So keep that in mind when you are dealing with inputs from sources like the camera that might have different color spaces depending on the device and camera configuration.
Another way to ensure the input samples are always in the color space you need is to set the kCISamplerColorSpace option when creating a CISampler that serves as input to your custom kernel.
After applying CIRadialGradient to my image it gets reduced in width by about 20%.
guard let image = bgImage.image, let cgimg = image.cgImage else {
print("imageView doesn't have an image!")
return
}
let coreImage = CIImage(cgImage:cgimg)
guard let radialMask = CIFilter(name:"CIRadialGradient") else {
return
}
guard let maskedVariableBlur = CIFilter(name:"CIMaskedVariableBlur") else {
print("CIMaskedVariableBlur does not exist")
return
}
maskedVariableBlur.setValue(coreImage, forKey: kCIInputImageKey)
maskedVariableBlur.setValue(radialMask.outputImage, forKey: "inputMask")
guard let selectivelyFocusedCIImage = maskedVariableBlur.outputImage else {
print("Setting maskedVariableBlur failed")
return
}
bgImage.image = UIImage(ciImage: selectivelyFocusedCIImage)
To clarify, bgImage is a UIImageView.
Why does this happen and how do I fix it?
Without RadialMask:
With RadialMask:
With the difference that on my physical iPhone the smaller image is aligned to the left.
I tend to explicitly state how big the image is by using a CIContext and creating a specifically sized CGImage instead of simply using UIImage(ciImage:). Try this, assuming your inputImage is called coreGraphics:
let ciCtx = CIContext()
let cgiig = ctx.createCGImage(selectivelyFocusedCIImage, from: coreImage.extent)
let uiImage = UIImage(cgImage: cgIMG!)
A few notes....
(1) I pulled this code out from an app I'm wrapping up. This is untested code (including the forced-unwrap), but the concept of what I'm doing is solid.
(2) You don't explain a lot of what you are trying to do, but when I see a variable named selectivelyFocusedCIImage I get concerned that you may be trying to use CoreImage in a more interactive way than "just" creating one image. If you want "near real-time" performance, render the CIImage in either a (deprecated as of iOS 12) GLKView or an MTKView instead of a UIImageView. The latter only uses the CPU where the two former use the GPU.
(3) Finally, a word of warning on CIContexts - they are expensive to create! Usually you can code it such that there's only one context that can be shared by everything n your app.
Look up the documentation, it's a mask that being applied to the image:
Docs: CIRadialGradient
The different sizes are caused by the kernel size of the blur filter:
The blur filter needs to sample a region around each pixel. Since there are no pixels beyond the image bounds, Core Image reduces the extend of the result image by half the kernel size (blur radius) to signal that for those pixels there is not enough information for a proper blur.
However, you can tell Core Image to treat the border pixels as extending infinitely in all directions so that the blur filter gets enough information even on the edges of the image. Afterwards you can crop the result back to the original dimension.
In your code, just change the following two lines:
maskedVariableBlur.setValue(coreImage.clampedToExtent(), forKey: kCIInputImageKey)
bgImage.image = UIImage(ciImage: selectivelyFocusedCIImage.cropped(to:coreImage.extend))
I also found this question on Apple Dev Forum.
Is it possible for a CIDetector set with CIDetectorTypeRectangle to return more than just one rectangle?
At the moment, this code always return a feature.count of 0 or 1, even if the picture is full of rectangles.
let context = CIContext()
let opts = [CIDetectorAccuracy : CIDetectorAccuracyHigh]
let detector = CIDetector(ofType: CIDetectorTypeRectangle, context: context, options: opts)
let image = CIImage(image: self.photoTaken)
let features = detector.features(in: image)
print(features.count) // never more than 1
According to this talk in WWDC (http://asciiwwdc.com/2014/sessions/514), it is limited to only one rectangle.
Here is a quote for that:
So we've created a generic rectangle detector object and it takes one
option parameter which is the aspect ratio that we want to search for.
And again, you can ask the detector to return the features array.
Now right now, it just returns one rectangle but that may change in
the future.
I'm writing a simple(at least I thought it would be simple) custom kernel that takes the difference of a specified pixel and an entire image.
Below is the code that I have, this just makes the filter. It's good to use in a playground play with.
import UIKit
import CoreImage
let Flower = CIImage( image: UIImage(named: "flower.png")!)!
class Test: CIFilter
{
var inputImage1 : CIImage?
var inputImage2 : CIImage?
var kernel = CIKernel(string:
"kernel vec4 colorRemap(sampler inputIm, sampler GaussIm) " +
"{ " +
"vec4 size = samplerExtent(inputIm); " +
"float row = 1.0; " +
"float column = 1.0; " +
"float pixelx = (column - 1.0)/(size.w - 1.0)+1.0/(2.0*size.z);" +
"float pixely = (size.z - row)/(size.z - 1.0)-1.0/(2.0*size.w);" +
"vec3 g0 =sample(GaussIm,vec2(pixelx,pixely)).rgb; " +
"vec3 current = sample(inputIm,samplerCoord(inputIm)).rgb; " +
"vec3 diff =(current - g0); " +
"return vec4(diff,1.0); " +
"} "
)
var extentFunction: (CGRect, CGRect) -> CGRect =
{ (a: CGRect, b: CGRect) in return CGRectZero }
override var outputImage: CIImage!
{
if let inputImage1 = inputImage1,
inputImage2 = inputImage2,
kernel = kernel
{
let extent = inputImage1.extent
let arguments = [inputImage1,inputImage2]
return kernel.applyWithExtent(extent,
roiCallback:
{ (index, rect) in
return rect
},
arguments: arguments)
}
return nil
}
}
To use the filter, you can do the following
let filter = Test()
filter.inputImage1 = Flower
filter.inputImage2 = Flower
let output = filter.outputImage
Now, in the above code, I've specified that we're taking the difference between the pixel located at (1,1) of GaussIm, as if we were treating the image as a matrix (in the usual sense), and the entire image of inputIm.
After playing around, I had come to realize that the Custom Kernel Language treats images a bit like OpenGL does. The bottom left corner is mapped to (0,0), and the top right being (1,1), so that pixel coordinates are numbers between 0 and 1. The issue with this is that I want to specify whatever pixel I want to use to take the difference.
The first 5 lines of the kernel code attempts to alleviate this by computing the center of each pixel location in the image. I'm not sure if this is right considering how OpenGL treats it's images, or maybe there's a better way.
If I run this code above, with the below image:
I get the following with XCode:
Further, if I do the same thing in MATLAB, I get the following output:
Why am I getting a different output than in MATLAB? It almost seems a tad darker than what I'm getting from my custom filter, and yet they are close to the same output at the same time. My thought was that it must be the way the custom kernel is taking the difference amongst pixels, but I'm not really sure what's going on.
I ended up figuring this out -- the reason for the clipping is due to the nature of how images are computed. This work was done in a playground, not on a context, so anything that was displayed was clipped to the range of [0,1]. The way to fix this was to make sure that your CIContext that you are doing calculations on support a floating point precision in its calculations, via the options.