I'd like decode a PNG image into an OpenCV Mat object using imdecode. I'm working on a function like
fn handle_frame(buf: &[u8]) -> Result<(), opencv::Error> {
original_image: Mat = imgcodecs::imdecode(buf, imgcodecs::IMREAD_COLOR)?;
let width = original_image.cols()?;
let height = original_image.rows()?;
println!("Success! Dimensions are {}x{}", width, height);
Ok(())
}
But I cannot pass by byte buffer to imdecode because I'd first need to convert it to something that has the ToInputArray trait. How to do this?
I found out that when I change the type of the input buffer to Vec<u8> I can do this:
let original_image: Mat = imgcodecs::imdecode(&VectorOfuchar :: from_iter(buf), imgcodecs::IMREAD_COLOR)?;
Here is a mostly complete example:
let filename = "somepicture.png";
let mut reader: Box<dyn BufRead> = Box::new(BufReader::new(File::open(filename)?));
let mut buffer : Vec<u8> = Vec::new();
let _read_count = reader.read_to_end(&mut buffer)?;
let result = cv::imgcodecs::imdecode(&cv::types::VectorOfu8::from_iter(buffer), cv::imgcodecs::IMREAD_COLOR)?;
cv::highgui::imshow(&filename, &result)?;
let _key = cv::highgui::wait_key(0)?;
Related
I am trying to use a CIColorKernel or CIBlendKernel with sampler arguments but the program crashes. Here is my shader code which compiles successfully.
extern "C" float4 wipeLinear(coreimage::sampler t1, coreimage::sampler t2, float time) {
float2 coord1 = t1.coord();
float2 coord2 = t2.coord();
float4 innerRect = t2.extent();
float minX = innerRect.x + time*innerRect.z;
float minY = innerRect.y + time*innerRect.w;
float cropWidth = (1 - time) * innerRect.w;
float cropHeight = (1 - time) * innerRect.z;
float4 s1 = t1.sample(coord1);
float4 s2 = t2.sample(coord2);
if ( coord1.x > minX && coord1.x < minX + cropWidth && coord1.y > minY && coord1.y <= minY + cropHeight) {
return s1;
} else {
return s2;
}
}
And it crashes on initialization.
class CIWipeRenderer: CIFilter {
var backgroundImage:CIImage?
var foregroundImage:CIImage?
var inputTime: Float = 0.0
static var kernel:CIColorKernel = { () -> CIColorKernel in
let url = Bundle.main.url(forResource: "AppCIKernels", withExtension: "ci.metallib")!
let data = try! Data(contentsOf: url)
return try! CIColorKernel(functionName: "wipeLinear", fromMetalLibraryData: data) //Crashes here!!!!
}()
override var outputImage: CIImage? {
guard let backgroundImage = backgroundImage else {
return nil
}
guard let foregroundImage = foregroundImage else {
return nil
}
return CIWipeRenderer.kernel.apply(extent: backgroundImage.extent, arguments: [backgroundImage, foregroundImage, inputTime])
}
}
It crashes in the try line with the following error:
Fatal error: 'try!' expression unexpectedly raised an error: Foundation._GenericObjCError.nilError
If I replace the kernel code with the following, it works like a charm:
extern "C" float4 wipeLinear(coreimage::sample_t s1, coreimage::sample_t s2, float time)
{
return mix(s1, s2, time);
}
So there are no obvious errors in the code, such as passing incorrect function name or so.
For your use case, you actually can use a CIColorKernel. You just have to pass the extent of your render destination to the kernel as well, then you don't need the sampler to access it.
The kernel would look like this:
extern "C" float4 wipeLinear(coreimage::sample_t t1, coreimage::sample_t t2, float4 destinationExtent, float time, coreimage::destination destination) {
float minX = destinationExtent.x + time * destinationExtent.z;
float minY = destinationExtent.y + time * destinationExtent.w;
float cropWidth = (1.0 - time) * destinationExtent.w;
float cropHeight = (1.0 - time) * destinationExtent.z;
float2 destCoord = destination.coord();
if ( destCoord.x > minX && destCoord.x < minX + cropWidth && destCoord.y > minY && destCoord.y <= minY + cropHeight) {
return t1;
} else {
return t2;
}
}
And you call it like this:
let destinationExtent = CIVector(cgRect: backgroundImage.extent)
return CIWipeRenderer.kernel.apply(extent: backgroundImage.extent, arguments: [backgroundImage, foregroundImage, destinationExtent, inputTime])
Note that the last destination parameter in the kernel is passed automatically by Core Image. You don't need to pass it with the arguments.
Yes, you can't use samplers in CIColorKernel or CIBlendKernel. Those kernels are optimized for the use case where you have a 1:1 mapping from input pixel to output pixel. This allows Core Image to execute multiple of these kernels in one command buffer since they don't require any intermediate buffer writes.
A sampler would allow you to sample the input at arbitrary coordinates, which is not allowed in this case.
You can simply use a CIKernel instead. It's meant to be used when you need to sample the input more freely.
To initialize the kernel, you need to adapt the code like this:
static var kernel: CIKernel = {
let url = Bundle.main.url(forResource: "AppCIKernels", withExtension: "ci.metallib")!
let data = try! Data(contentsOf: URL)
return try! CIKernel(functionName: "wipeLinear", fromMetalLibraryData: data)
}()
When calling the kernel, you now need to also provide a ROI callback, like this:
let roiCallback: CIKernelROICallback = { index, rect -> CGRect in
return rect // you need the same region from the input as the output
}
// or even shorter
let roiCallback: CIKernelROICallback = { $1 }
return CIWipeRenderer.kernel.apply(extent: backgroundImage.extent, roiCallback: roiCallback, arguments: [backgroundImage, foregroundImage, inputTime])
Bonus answer:
For this blending effect, you actually don't need any kernel at all. You can achieve all that with simple cropping and compositing:
class CIWipeRenderer: CIFilter {
var backgroundImage:CIImage?
var foregroundImage:CIImage?
var inputTime: CGFloat = 0.0
override var outputImage: CIImage? {
guard let backgroundImage = backgroundImage else { return nil }
guard let foregroundImage = foregroundImage else { return nil }
// crop the foreground based on time
var foregroundCrop = foregroundImage.extent
foregroundCrop.size.width *= inputTime
foregroundCrop.size.height *= inputTime
return foregroundImage.cropped(to: foregroundCrop).composited(over: backgroundImage)
}
}
I can convert a Base64 string to a DynamicImage using the image and base64 crates but I can not convert the image back to a Base64 string.
What I am doing wrong here? How to achieve this?
extern crate base64;
extern crate image;
fn main() {
/*
Base64 to image
let img_buffer = base64::decode("qwerty...").unwrap();
let mut base_img = image::load_from_memory(img_buffer.as_slice()).unwrap();
*/
let mut base_img = image::open("player.png").unwrap();
base_img.invert();
// base_img.save("player1.png").unwrap();
// image to Base64
let res_base64 = base64::encode(base_img.raw_pixels().as_slice());
println!("{}", res_base64)
}
Base64 value of original image
iVBORw0KGgoAAAANSUhEUgAAAgAAAAIACAYAAAD0eNT6AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAgAElEQVR42uydd5hlRZn/v1Un3dzdt/P0pJ6enANxGKISFEQkg4ur6667uquuCiKCIJKUpAj7U3cN66orWVERVxBmmDxMYHLOM90znfvGk6rq98e9ICoo0Kf73u5+P8+DjPBwZrpO1Xm...
Base64 value of Gimp inverted image
iVBORw0KGgoAAAANSUhEUgAAAgAAAAIACAYAAAD0eNT6AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAB3RJTUUH4wgMCCwxQF4N2QAAIABJREFUeNrsvXmc3FWZ7...
The response I am getting not the proper Base64 value:
////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///8A////AP///wD///...
The Base64 values of the original and Gimp images represent PNG-encoded images. The response you're getting represents raw pixel data. You will need to convert the data to PNG before encoding as Base64. This should do it:
let mut buf = vec![[]];
base_img.write_to(&mut buf, image::ImageOutputFormat::PNG);
let res_base64 = base64::encode(&buf);
My 2022 solution is:
#Cargo.toml
[dependencies]
base64 = "0.13"
image = "0.24"
use image::{DynamicImage, ImageOutputFormat};
use std::io::Cursor;
fn image_to_base64(img: &DynamicImage) -> String {
let mut image_data: Vec<u8> = Vec::new();
img.write_to(&mut Cursor::new(&mut image_data), ImageOutputFormat::Png)
.unwrap();
let res_base64 = base64::encode(image_data);
format!("data:image/png;base64,{}", res_base64)
}
let img = image::load_from_memory(&data.clone()).unwrap();
println!("{}", image_to_base64(&img));
You can use this string for html tag <img src="..." />.
Thanks to this great article(http://machinethink.net/blog/coreml-custom-layers/), I understood how to write converting using coremltools and Lambda with Keras custom layer.
But, I cannot understand on the situation, function with two parameters.
#python
def scaling(x, scale):
return x * scale
Keras layer is here.
#python
up = conv2d_bn(mixed,
K.int_shape(x)[channel_axis],
1,
activation=None,
use_bias=True,
name=name_fmt('Conv2d_1x1'))
x = Lambda(scaling, # HERE !!
output_shape=K.int_shape(up)[1:],
arguments={'scale': scale})(up)
x = add([x, up])
On this situation, how can I write func evaluate(inputs: [MLMultiArray], outputs: [MLMultiArray]) in custom MLCustomLayer class on Swift? I understand just in one parameter function situation, like this,
#swift
func evaluate(inputs: [MLMultiArray], outputs: [MLMultiArray]) throws {
for i in 0..<inputs.count {
let input = inputs[i]
let output = outputs[i]
for j in 0..<input.count {
let x = input[j].floatValue
let y = x / (1 + exp(-x))
output[j] = NSNumber(value: y)
}
}
}
How about two parameters function, like x * scale?
Full code is here.
Converting to Core ML model with custom layer
https://github.com/osmszk/dla_team14/blob/master/facenet/coreml/CoremlTest.ipynb
Network model by Keras
https://github.com/osmszk/dla_team14/blob/master/facenet/code/facenet_keras_v2.py
Thank you.
It looks like scale is a hyperparameter, not a learnable parameter, is that correct?
In that case, you need to add scale to the parameters dictionary for the custom layer. Then in your Swift class, scale will also be inside the parameters dictionary that is passed into your init(parameters) function. Store it inside a property and then in evaluate(inputs, outputs) read from that property again.
My blog post actually shows how to do this. ;-)
I solved this problem on this way thanks to hollance's blog. On converting func, in this case, in convert_lambda, I should have added a scale parameter for the custom layer.
python code(converting Core ML)
def convert_lambda(layer):
if layer.function == scaling:
params = NeuralNetwork_pb2.CustomLayerParams()
params.className = "scaling"
params.description = "scaling input"
# HERE!! This is important.
params.parameters["scale"].doubleValue = layer.arguments['scale']
return params
else:
return None
coreml_model = coremltools.converters.keras.convert(
model,
input_names="image",
image_input_names="image",
output_names="output",
add_custom_layers=True,
custom_conversion_functions={ "Lambda": convert_lambda })
swift code(Custom layer)
//custom MLCustomLayer `scaling` class
let scale: Float
required init(parameters: [String : Any]) throws {
if let scale = parameters["scale"] as? Float {
self.scale = scale
} else {
self.scale = 1.0
}
print(#function, parameters, self.scale)
super.init()
}
func evaluate(inputs: [MLMultiArray], outputs: [MLMultiArray]) throws {
for i in 0..<inputs.count {
let input = inputs[i]
let output = outputs[i]
for j in 0..<input.count {
let x = input[j].floatValue
let y = x * self.scale
output[j] = NSNumber(value: y)
}
//faster
/*
let count = input.count
let inputPointer = UnsafeMutablePointer<Float>(OpaquePointer(input.dataPointer))
let outputPointer = UnsafeMutablePointer<Float>(OpaquePointer(output.dataPointer))
var scale = self.scale
vDSP_vsmul(inputPointer, 1, &scale, outputPointer, 1, vDSP_Length(count))
*/
}
}
Thank you.
I am trying to use openCL with FSCL on F# but I am obtaining some errors that I don't understand
open FSCL.Compiler
open FSCL.Language
open FSCL.Runtime
open Microsoft.FSharp.Linq.RuntimeHelpers
open System.Runtime.InteropServices
[<StructLayout(LayoutKind.Sequential)>]
type gpu_point2 =
struct
val mutable x: float32
val mutable y: float32
new ( q ,w) = {x=q; y=w}
end
[<ReflectedDefinition>]
let PointSum(a:gpu_point2,b:gpu_point2) =
let sx =(a.x+b.x)
let sy =(a.y+b.y)
gpu_point2(sx,sy)
[<ReflectedDefinition;Kernel>]
let Modgpu(b:float32[], c:float32[],wi:WorkItemInfo) =
let gid = wi.GlobalID(0)
let arp = Array.zeroCreate<gpu_point2> b.Length
let newpoint = gpu_point2(b.[gid],c.[gid])
arp.[gid] <- newpoint
arp
[<ReflectedDefinition;Kernel>]
let ModSum(a:gpu_point2[],b:gpu_point2[],wi:WorkItemInfo) =
let gid = wi.GlobalID(0)
let cadd = Array.zeroCreate<gpu_point2> a.Length
let newsum = PointSum(a.[gid],b.[gid])
cadd.[gid] <- newsum
cadd
[<ReflectedDefinition;Kernel>]
let ModSum2(a:gpu_point2[],b:gpu_point2[],wi:WorkItemInfo) =
let gid = wi.GlobalID(0)
let cadd = Array.zeroCreate<gpu_point2> a.Length
let newsum = gpu_point2(a.[gid].x+b.[gid].x,a.[gid].y+b.[gid].y)
cadd.[gid] <- newsum
cadd
let ws = WorkSize(64L)
let arr_s1= <# Modgpu([|0.f..63.f|],[|63.f..(-1.f)..0.f|],ws)#>.Run()
let arr_s2 = <# Modgpu([|63.f..(-1.f)..0.f|],[|0.f..63.f|],ws)#>.Run()
With this code when I try to use ModSum as
let rsum = <# ModSum(arr_s1,arr_s2,ws)#>.Run()
doesn't work, but instead when I use ModSum2 works perfectly
let rsum = <# ModSum2(arr_s1,arr_s2,ws)#>.Run()
The error I obtain the first time I run it is
FSCL.Compiler.CompilerException: Unrecognized construct in kernel body NewObject (gpu_point2, sx, sy)
and if I re-run the fsi console says
System.NullReferenceException: Object reference not set to an instance of an object.
The only thing I know is that the error doesn't comes from the use of another function since I can define a dot product function that works.
[<ReflectedDefinition>]
let PointProd(a:gpu_point2,b:gpu_point2) =
let f = (a.x*b.x)
let s = (a.y*b.y)
f+s
Thus, I guess the problem comes from the return type of PointSum, but is there a way to create such a function to sum two points and return the point type? And Why is not working?
Edit/Update:
Also with a record happens the same if I define the type as :
[<StructLayout(LayoutKind.Sequential)>]
type gpu_point_2 = {x:float32; y:float32}
If I try to create a function that directly sums two gpu_point_2 on a function works, but if I call a second function it raises the same error as using a struct.
Try to add [<ReflectedDefinition>] on the constructor of gpu_point2:
[<StructLayout(LayoutKind.Sequential)>]
type gpu_point2 =
struct
val mutable x: float32
val mutable y: float32
[<ReflectedDefinition>] new (q, w) = {x=q; y=w}
end
Normally each code that is called from the device need this attribute, constructors included.
Currently I am using this method to loop through every pixel, and insert a value into a 3D array based upon RGB values. I need this array for other parts of my program, however it is extraordinarily slow. When run on a 50 x 50 picture, it is almost instant, but as soon as you start getting into the hundreds x hundreds it takes a long time to the point where the app is useless. Anyone have any ideas on how to speed up my method?
#IBAction func convertImage(sender: AnyObject) {
if let image = myImageView.image {
var pixelData = CGDataProviderCopyData(CGImageGetDataProvider(image.CGImage))
var data: UnsafePointer<UInt8> = CFDataGetBytePtr(pixelData)
let height = Int(image.size.height)
let width = Int(image.size.width)
var zArry = [Int](count:3, repeatedValue: 0)
var yArry = [[Int]](count:width, repeatedValue: zArry)
var xArry = [[[Int]]](count:height, repeatedValue: yArry)
for (var h = 0; h < height; h++) {
for (var w = 0; w < width; w++) {
var pixelInfo: Int = ((Int(image.size.width) * Int(h)) + Int(w)) * 4
var rgb = 0
xArry[h][w][rgb] = Int(data[pixelInfo])
rgb++
xArry[h][w][rgb] = Int(data[pixelInfo+1])
rgb++
xArry[h][w][rgb] = Int(data[pixelInfo+2])
}
}
println(xArry[20][20][1])
}
}
Maybe there is a way to convert the UIImage to a different type of image and create an array of pixels. I am open to all suggestions. Thanks!
GOAL: The goal is to use the array to modify the RGB values of all pixels, and create a new image with the modified pixels. I tried simply looping through all of the pixels without storing them, and modifying them into a new array to create an image, but got the same performance issues.
Update:
After countless tries I realized I was making my tests on debug configuration.
Switched to release, and now it's so much faster.
Swift seems to be many times slower on the debug configuration.
The difference now between your code and my optimized version is several times faster.
It seems as you have a big slowdown from using image.size.width instead of the local variable width.
Original
I tried to optimize it a bit and come up with this:
#IBAction func convertImage () {
if let image = UIImage(named: "test") {
let pixelData = CGDataProviderCopyData(CGImageGetDataProvider(image.CGImage))
let data: UnsafePointer<UInt8> = CFDataGetBytePtr(pixelData)
let height = Int(image.size.height)
let width = Int(image.size.width)
let zArry = [Int](count:3, repeatedValue: 0)
let yArry = [[Int]](count:width, repeatedValue: zArry)
let xArry = [[[Int]]](count:height, repeatedValue: yArry)
for (index, value) in xArry.enumerate() {
for (index1, value1) in value.enumerate() {
for (index2, var value2) in value1.enumerate() {
let pixelInfo: Int = ((width * index) + index1) * 4 + index2
value2 = Int(data[pixelInfo])
}
}
}
}
}
However in my tests this is barely 15% faster. What you need is orders of magnitude faster.
Another ideea is use the data object directly when you need it without creating the array like this:
let image = UIImage(named: "test")!
let pixelData = CGDataProviderCopyData(CGImageGetDataProvider(image.CGImage))
let data: UnsafePointer<UInt8> = CFDataGetBytePtr(pixelData)
let width = Int(image.size.width)
// value for [x][y][z]
let value = Int(data[((width * x) + y) * 4 + z])
You didn't say how you use this array in your app, but I feel that even if you find a way to get this array created much faster, you would get another problem when you try to use it, as it would take a long time too..