In the app I'm working on, we're capturing photos which need to have 4:3 aspect ratio in order to maximize the field of view we capture. Up untill now we were using AVCaptureSessionPreset640x480 preset, but now we're in need of larger resolution.
As far as I've figured, the only other two 4:3 formats are 2592x1936 and 3264x2448. Since these are too large for our use case, I need a way to downsize them. I looked into a bunch of options but did not find a way (prefereably without copying the data) to do this in an efficient manner without losing the exif data.
vImage was one of the things I looked into but as far as I've figured the data would need to be coppied and the exif data would be lost. Another option was creating an UIImage from data provided by jpegStillImageNSDataRepresentation, scaling it and getting the data back. This approach also seems to strip the exif data.
The ideal approach here would be resizing the buffer contents directly and resizing the photo. Does anyone have an idea how I would go about doing this?
I ended up using ImageIO for resizing purposes. Leaving this piece of code here in case someone runs into the same problem, as I've spent way too much time on this.
This code will preserve the exif data, but will create a copy of the image data. I ran some benchmarks - the execution time for this method is ~0.05sec on iPhone6, using AVCaptureSessionPresetPhoto as the preset for the original photo.
If someone does have a more optimal solution, please leave a comment.
- (NSData *)resizeJpgData:(NSData *)jpgData
{
CGImageSourceRef source = CGImageSourceCreateWithData((CFDataRef)jpgData, NULL);
// Create a copy of the metadata that we'll attach to the resized image
NSDictionary *metadata = (NSDictionary *)CFBridgingRelease(CGImageSourceCopyPropertiesAtIndex(source, 0, NULL));
NSMutableDictionary *metadataAsMutable = [metadata mutableCopy];
// Type of the image (e.g. public.jpeg)
CFStringRef UTI = CGImageSourceGetType(source);
NSDictionary *options = #{ (id)kCGImageSourceCreateThumbnailFromImageIfAbsent: (id)kCFBooleanTrue,
(id)kCGImageSourceThumbnailMaxPixelSize: #(MAX(FORMAT_WIDTH, FORMAT_HEIGHT)),
(id)kCGImageSourceTypeIdentifierHint: (__bridge NSString *)UTI };
CGImageRef resizedImage = CGImageSourceCreateThumbnailAtIndex(source, 0, (CFDictionaryRef)options);
NSMutableData *destData = [NSMutableData data];
CGImageDestinationRef destination = CGImageDestinationCreateWithData((CFMutableDataRef)destData, UTI, 1, NULL);
if (!destination) {
NSLog(#"Could not create image destination");
}
CGImageDestinationAddImage(destination, resizedImage, (__bridge CFDictionaryRef) metadataAsMutable);
// Tell the destination to write the image data and metadata into our data object
BOOL success = CGImageDestinationFinalize(destination);
if (!success) {
NSLog(#"Could not create data from image destination");
}
if (destination) {
CFRelease(destination);
}
CGImageRelease(resizedImage);
CFRelease(source);
return destData;
}
Related
I am using the following code to extract depth map (by following Apple's own example):
- (nullable AVDepthData *)depthDataFromImageData:(nonnull NSData *)imageData orientation:(CGImagePropertyOrientation)orientation {
AVDepthData *depthData = nil;
CGImageSourceRef imageSource = CGImageSourceCreateWithData((CFDataRef)imageData, NULL);
if (imageSource) {
NSDictionary *auxDataDictionary = (__bridge NSDictionary *)CGImageSourceCopyAuxiliaryDataInfoAtIndex(imageSource, 0, kCGImageAuxiliaryDataTypeDisparity);
if (auxDataDictionary) {
depthData = [[AVDepthData depthDataFromDictionaryRepresentation:auxDataDictionary error:NULL] depthDataByApplyingExifOrientation:orientation];
}
CFRelease(imageSource);
}
return depthData;
}
And I call this from:
[[PHAssetResourceManager defaultManager] requestDataForAssetResource:[PHAssetResource assetResourcesForAsset:asset].firstObject options:nil dataReceivedHandler:^(NSData * _Nonnull data) {
AVDepthData *depthData = [self depthDataFromImageData:data orientation:[self CGImagePropertyOrientationForUIImageOrientation:pickedUiImageOrientation]];
CIImage *image = [CIImage imageWithDepthData:depthData];
UIImage *uiImage = [UIImage imageWithCIImage:image];
UIGraphicsBeginImageContext(uiImage.size);
[uiImage drawInRect:CGRectMake(0, 0, uiImage.size.width, uiImage.size.height)];
UIImage *newImage = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
NSData *pngData = UIImagePNGRepresentation(newImage);
UIImage* pngImage = [UIImage imageWithData:pngData]; // rewrap
UIImageWriteToSavedPhotosAlbum(pngImage, nil, nil, nil);
} completionHandler:^(NSError * _Nullable error) {
}];
Here is the result: it's a low quality (and rotated but let's put orientation aside for now) image:
Then I've transferred the original HEIC file, opened in Photoshop, went to Channels, and selected depth map as below:
Here is the result:
It's a higher resolution/quality, correctly oriented depth map. Why is the code (actually Apple's own code at https://developer.apple.com/documentation/avfoundation/avdepthdata/2881221-depthdatafromdictionaryrepresent?language=objc) resulting in lower-quality result?
I've found the issue. Actually, it was hiding in plain sight. What is obtained from the +[AVDepthData depthDataFromDictionaryRepresentation:error:] method returns disparity data. I've converted it to depth using the following code:
if(depthData.depthDataType != kCVPixelFormatType_DepthFloat32){
depthData = [depthData depthDataByConvertingToDepthDataType:kCVPixelFormatType_DepthFloat32];
}
(Haven't tried but 16-bit Depth, kCVPixelFormatType_DepthFloat16, should also work well)
After converting disparity to depth, the image is exactly the same as in Photoshop. I should have woken up as I was using CGImageSourceCopyAuxiliaryDataInfoAtIndex(imageSource, 0, kCGImageAuxiliaryDataTypeDisparity); (note the "disparity" in the end) and Photoshop was clearly saying "depth map", converting disparity to depth (or just somehow reading as depth, I honestly don't know the physical encoding, maybe iOS was converting depth to disparity when I was copying the aux data in the first place) on the fly.
Side note: I've also solved the orientation issue by creating the image source directly from [PHAsset requestContentEditingInputWithOptions:completionHandler:] method and passing the contentEditingInput.fullSizeImageURL into CGImageSourceCreateWithURL method. It took care of the orientation.
I'm working on creating and storing OpenGL ES1 3D models, and want to include image files to be used as textures, within the same file as the 3D model data. I am having trouble loading the image data in a usable format. I'm using UIImageJPEGRepresentation to convert the image data and store it into an NSData. I then append it to a NSMutableData object, along with all the 3D data, and write it out to a file. The data seems to write and read without error, but I encounter problems when trying use the image data to create a "CGImageRef" which I use to generate the texture data for the 3D model. The image data seems to be in an unrecognized format after it is loaded from the file, because it generates the error "CGContextDrawImage: invalid context 0x0.” when I attempt to create the "CGImageRef". I suspect that the image data is gettin misaligned somehow, causing it to be rejected when attempting to create the "CGImageRef". I appreciate any help. I'm stumped at this point. All of the data sizes and offsets add up and look fine. Saves and loads happen without error. The image data just seems off a bit, but I don't know why.
Here's my code:
//======================================================
- (BOOL)save3DFile: (NSString *)filePath {
// load TEST IMAGE into UIIMAGE
UIImage *image = [UIImage imageNamed:#“testImage.jpg"];
// convert image to JPEG encoded NSDATA
NSData *imageData = UIImageJPEGRepresentation(image,1.0);
// Save length of imageData to global "imDataLen" to use later in “load3DFile”
imDataLen = [imageData length];
// TEST: this works fine for CGImageRef creation in “loadTexture”
// traceView.image=[UIImage imageWithData:[imageData subdataWithRange:NSMakeRange(0, imageDataLen)]];
// [self loadTexture];
// TEST: this also works fine for CGImageRef creation in “loadTexture”
// traceView.image=[UIImage imageWithData:txImData];
// [self loadTexture];
fvoh.fileVersion = FVO_VERSION;
fvoh.obVertDatLen = obVertDatLen;
fvoh.obFaceDatLen = obFaceDatLen;
fvoh.obNormDatLen = obNormDatLen;
fvoh.obTextDatLen = obTextDatLen;
fvoh.obCompCount = obCompCount;
fvoh.obVertCount = obVertCount;
fvoh.obElemCount = obElemCount;
fvoh.obElemSize = obElemSize;
fvoh.obElemType = obElemType;
NSMutableData *obSvData;
obSvData=[NSMutableData dataWithBytes:&fvoh length:(sizeof(fvoh))];
[obSvData appendBytes:obElem length:obFaceDatLen];
[obSvData appendBytes:mvElem length:obVertDatLen];
[obSvData appendBytes:mvNorm length:obNormDatLen];
[obSvData appendBytes:obText length:obTextDatLen];
[obSvData appendBytes:&ds length:(sizeof(ds))];
// next, we append image data, and write all data to a file
// seems to work fine, no errors, at this point
[obSvData appendBytes: imageData length:[imageData length]];
BOOL success=[obSvData writeToFile: filePath atomically:YES];
return success;
}
//======================================================
- (void) load3DFile:(NSString *)filePath {
NSData *fvoData;
NSUInteger offSet,fiLen,fhLen,dsLen;
[[FileList sharedFileList] setCurrFile:(NSString *)filePath];
fvoData=[NSData dataWithContentsOfFile:filePath];
fiLen=[fvoData length];
fhLen=sizeof(fvoh);
dsLen=sizeof(ds);
memcpy(&fvoh,[fvoData bytes],fhLen);offSet=fhLen;
//+++++++++++++++++++++++++++++++
obVertDatLen = fvoh.obVertDatLen;
obFaceDatLen = fvoh.obFaceDatLen;
obNormDatLen = fvoh.obNormDatLen;
obTextDatLen = fvoh.obTextDatLen;
obCompCount = fvoh.obCompCount;
obVertCount = fvoh.obVertCount;
obElemCount = fvoh.obElemCount;
obElemSize = fvoh.obElemSize;
obElemType = fvoh.obElemType;
//+++++++++++++++++++++++++++++++
memcpy(obElem, [fvoData bytes]+offSet,obFaceDatLen);offSet+=obFaceDatLen;
memcpy(mvElem, [fvoData bytes]+offSet,obVertDatLen);offSet+=obVertDatLen;
memcpy(mvNorm, [fvoData bytes]+offSet,obNormDatLen);offSet+=obNormDatLen;
memcpy(obText, [fvoData bytes]+offSet,obTextDatLen);offSet+=obTextDatLen;
memcpy(&ds, [fvoData bytes]+offSet,dsLen);offSet+=dsLen;
// the following seem to read the data into “imageData” just fine, no errors
// NSData *imageData = [fvoData subdataWithRange:NSMakeRange(offSet, imDataLen)];
// NSData *imageData = [fvoData subdataWithRange:NSMakeRange((fiLen-imDataLen), imDataLen)];
// NSData *imageData = [NSData dataWithBytes:[fvoData bytes]+offSet length: imDataLen];
NSData *imageData = [NSData dataWithBytes:[fvoData bytes]+(fiLen-imDataLen) length: imDataLen];
// but the contents of imageData seem to end up in an unexpected format, causing error:
// “CGContextDrawImage: invalid context 0x0.” during CGImageRef creation in “loadTexture”
traceView.image=[UIImage imageWithData:imageData];
[self loadTexture];
}
//======================================================
- (void)loadTexture {
CGImageRef image=[traceView.image].CGImage;
CGContextRef texContext;GLubyte* bytes=nil;GLsizei width,height;
if(image){
width=(GLsizei)CGImageGetWidth(image);
height=(GLsizei)CGImageGetHeight(image);
bytes=(GLubyte*) calloc(width*height*4,sizeof(GLubyte));
texContext=CGBitmapContextCreate(bytes,width,height,8,width*4,CGImageGetColorSpace(image),
kCGImageAlphaPremultipliedLast);
CGContextDrawImage(texContext,CGRectMake(0.0,0.0,(CGFloat)width,(CGFloat)height),image);
CGContextRelease(texContext);
}
if(bytes){
glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_MAG_FILTER,GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_MIN_FILTER,GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_WRAP_S,GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_WRAP_T,GL_CLAMP_TO_EDGE);
glTexImage2D(GL_TEXTURE_2D,0,GL_RGBA,width,height,0,GL_RGBA,GL_UNSIGNED_BYTE,bytes);
free(bytes);
}
}
//======================================================
I failed to receive any answers to this question. I finally stumbled across the answer myself. When I execute the save3DFile code, instead of adding the image data to NSMutableData *obSvData, using 'appendBytes' as illustrated below:
[obSvData appendBytes: imageData length:[imageData length]];
I instead use 'appendData' as shown here:
[obSvData appendData: imageData];
where imageData was previously filled with the contents of a UIImage and converted to JPEG format in the process as follows:
NSData *imageData = UIImageJPEGRepresentation(image,1.0);
See the complete code listing above for context. Anyway, the using 'appendData' instead of 'appendBytes' made all the difference, and allowed me to store the image data in the same file along with all the other 3D model data (vertices, indices, normals, et cetera), reloading all that data without problem, and successfully create 3D models with textures from a single file.
let image_data = UIImageJPEGRepresentation(self.imagetoadd.image!,0.0)
The image in ios, am using swift 3 to do this is being uploaded rotated.How can I solve such thing?
JPEG images usually contain an EXIF dictionary, here are stored a lot information about how the image was taken, image rotation is one of it.
UIImage instances keeps these information (if the original image has it) as well inside a specific property called imageOrientation.
As far as I remember this information is ripped of by using the method UIImageJPEGRepresentation.
To create a correct data instance with the above information you must use Core Graphics methods, or normalize the rotation before sending the image.
To normalize the image something like that should be enough:
CGImageRef cgRef = imageToSave.CGImage;
UIImage * fixImage = [[UIImage alloc] initWithCGImage:cgRef scale:imageToSave.scale orientation:UIImageOrientationUp];
To keep the rotation information:
CFURLRef url = (__bridge_retained CFURLRef)[NSURL fileURLWithPath:path];//Save data path
NSDictionary * metadataDictionary = [self imageMetadataForPath:pathToOriginalImage];
CFMutableDictionaryRef metadataImage = (__bridge_retained CFMutableDictionaryRef) metadata;
CGImageDestinationRef destination = CGImageDestinationCreateWithURL(url, kUTTypeJPEG, 1, NULL);
CGImageDestinationAddImage(destination, image, metadataImage);
if (!CGImageDestinationFinalize(destination)) {
DLog(#"Failed to write image to %#", path);
}
Where the -imageMetadataForPath:
- (NSDictionary*) imageMetadataForPath:(NSString*) imagePath{
NSURL *imageURL = [NSURL fileURLWithPath:imagePath];
CGImageSourceRef mySourceRef = CGImageSourceCreateWithURL((__bridge CFURLRef)imageURL, NULL);
NSDictionary * dict = (NSDictionary *) CFBridgingRelease(CGImageSourceCopyPropertiesAtIndex(mySourceRef,0,NULL));
CFRelease(mySourceRef);
return dict;
}
This is a copy and paste from a project of mine, you probably need to do a huge refactoring, also because it is using manual memory management in core foundation and you are using SWIFT. Of course by using this last set of instructions, the backend code must be prepared to deal with image orientation too.
If you want to know more about rotation, here is a link.
I want to generate a true color animated Gif from a couple of PNG files represented as base64 string. I found this post and did something similar. I have an array with the dataUrls:
NSArray* imageDataUrls; // array with the data urls without data:image/png;base64, prefix
Here is what I did:
NSDictionary *fileProperties = #{
(__bridge id)kCGImagePropertyGIFDictionary: #{
(__bridge id)kCGImagePropertyGIFLoopCount: #0, // 0 means loop forever
}
};
NSDictionary *frameProperties = #{
(__bridge id)kCGImagePropertyGIFDictionary: #{
(__bridge id)kCGImagePropertyGIFDelayTime: #0.4f, // a float (not double!) in seconds, rounded to centiseconds in the GIF data
}
};
NSURL *documentsDirectoryURL = [[NSFileManager defaultManager] URLForDirectory:NSDocumentDirectory inDomain:NSUserDomainMask appropriateForURL:nil create:YES error:nil];
NSURL *fileURL = [documentsDirectoryURL URLByAppendingPathComponent:#"animated.gif"];
CFMutableDataRef destinationData = CFDataCreateMutable(kCFAllocatorDefault, 0);
CGImageDestinationRef destination = CGImageDestinationCreateWithData(destinationData, kUTTypeGIF, kFrameCount, NULL);
CGImageDestinationSetProperties(destination, (__bridge CFDictionaryRef)fileProperties);
NSData* myImageData;
UIImage *myImage = [UIImage alloc];
for (NSUInteger i = 0; i < kFrameCount; i++) {
#autoreleasepool {
myImageData = [NSData dataFromBase64String:[imageDataUrls objectAtIndex:i]];
myImage = [myImage initWithData: myImageData];
CGImageDestinationAddImage(destination, myImage.CGImage, (__bridge CFDictionaryRef)frameProperties);
}
}
myImageData = nil;
myImage = nil;
CFRelease(destination);
NSData* data = nil;
data = (__bridge NSData *)destinationData;
Finally, I send the gif image as base64EncodedString back to the phonegap container.
// send back gif image
CDVPluginResult* pluginResult = [CDVPluginResult resultWithStatus:CDVCommandStatus_OK messageAsString: [data base64EncodedString]];
It works good but the quality of the resulting gif image is bad. This is because it has only 256 colors.
Here is the original png image:
Here is a screenshot of the generated gif image:
How do I get the same quality as I imported, i.e., how can I raise the quality level of the generated gif? How do I generate true color gifs on iOS?
GIFs are not designed to store true-color data, and they are also poorly suited for animations1. Since this is such an unusual use of GIFs, you will have to write a lot of your own code.
Break each frame into rectangular chunks, where each chunk contains at most 256 distinct colors. The easiest way to do this is to use 16x16 chunks.
Convert each chunk to an indexed image.
Add each chunk to the GIF. For the first chunk in a frame, use the frame delay. For other chunks in a frame, use a delay of 0.
Done. You will have to familiarize yourself with the GIF specification, which is freely available online (GIF89a specification at W3.org, see section 23). You will also need to find an LZW compressor, which is not too hard to find. The animation will also use an obscene amount of storage: including base64 conversion, I estimate about 43 bits/pixel, or about 1.2 Gbit/s for 720p video, which is about 400x as much storage as you would use for high-quality MPEG4 or WebM, and probably about 3x as much storage as the PNGs would require. The storage and bandwidth requirements will likely incur undesirable costs for hosts and clients, unless the animations are very short and small.
Note that this will not allow you to use alpha transparency, this is a hard limitation of the GIF format.
Opinion
The idea of putting high quality animations in a GIF is absurd in the extreme, even though it is possible. It is especially absurd given the available alternatives:
If you are targeting modern browsers or mobile devices, MPEG4 (support matrix) and WebM (support matrix) are the obvious choices. Between the two formats, only Opera Mini supports neither.
If you are targeting older browsers or less-capable devices, or if you cannot afford MPEG4 encoding, you can encode the frames as individual JPEG or PNG images. Bundle these with a JSON payload with the timing, and use JavaScript or other client-side scripting to switch between animation frames. This works surprisingly well.
Notes
1 From the GIF 89a specification:
Animation - The Graphics Interchange Format is not intended as a platform for
animation, even though it can be done in a limited way.
I have built a camera using AVFoundation.
Once my AVCaptureStillImageOutput has completed its captureStillImageAsynchronouslyFromConnection:completionHandler: method, I create a NSData object like this:
NSData *imageData = [AVCaptureStillImageOutput jpegStillImageNSDataRepresentation:imageDataSampleBuffer];
Once I have the NSData object, I would like to rotate the image -without- converting to a UIImage. I have found out that I can convert to a CGImage to do so.
After I have the imageData, I start the process of converting to CGImage, but I have found that the CGImageRef ends up being THIRTY times larger than the NSData object.
Here is the code I use to convert to CGImage from NSData:
CGDataProviderRef imgDataProvider = CGDataProviderCreateWithCFData((__bridge CFDataRef)(imageData));
CGImageRef imageRef = CGImageCreateWithJPEGDataProvider(imgDataProvider, NULL, true, kCGRenderingIntentDefault);
If I try to NSLog out the size of the image, it comes to 30 megabytes when the NSData was a 1.5-2 megabyte image!
size_t imageSize = CGImageGetBytesPerRow(imageRef) * CGImageGetHeight(imageRef);
NSLog(#"cgimage size = %zu",imageSize);
I thought that maybe when you go from NSData to CGImage, the image decompresses, and then maybe if I converted back to NSData, that it might go back to the right file size.
imageData = (NSData *) CFBridgingRelease(CGDataProviderCopyData(CGImageGetDataProvider(imageRef)));
The above NSData has the same length as the CGImageRef object.
If I try to save the image, the image is a 30mb image that cannot be opened.
I am totally new to using CGImage, so I am not sure if I am converting from NSData to CGImage and back incorrectly, or if I need to call some method to decompress again.
Thanks in advance,
Will
I was doing some image manipulation and came across your question on SO. Seems like no one else came up with an answer, so here's my theory.
While it's theoretically possible to convert a CGImageRef back to NSData in the manner that you described, the data itself is invalid and not a real JPEG or PNG, as you discovered by it not being readable. So I don't think that the NSData.length is correct. You have to actually jump through a number of steps to recreate an NSData representation of a CGImageRef:
// incoming image data
NSData *image;
// create the image ref
CGDataProviderRef imgDataProvider = CGDataProviderCreateWithCFData((__bridge CFDataRef) image);
CGImageRef imageRef = CGImageCreateWithJPEGDataProvider(imgDataProvider, NULL, true, kCGRenderingIntentDefault);
// image metadata properties (EXIF, GPS, TIFF, etc)
NSDictionary *properties;
// create the new output data
CFMutableDataRef newImageData = CFDataCreateMutable(NULL, 0);
// my code assumes JPEG type since the input is from the iOS device camera
CFStringRef type = UTTypeCreatePreferredIdentifierForTag(kUTTagClassMIMEType, (__bridge CFStringRef) #"image/jpg", kUTTypeImage);
// create the destination
CGImageDestinationRef destination = CGImageDestinationCreateWithData(newImageData, type, 1, NULL);
// add the image to the destination
CGImageDestinationAddImage(destination, imageRef, (__bridge CFDictionaryRef) properties);
// finalize the write
CGImageDestinationFinalize(destination);
// memory cleanup
CGDataProviderRelease(imgDataProvider);
CGImageRelease(imageRef);
CFRelease(type);
CFRelease(destination);
NSData *newImage = (__bridge_transfer NSData *)newImageData;
With these steps, the newImage.length should be the same as image.length. I haven't tested since I actually do cropping between the input and the output, but based on the crop, the size is roughly what I expected (the output is roughly half the pixels of the input and thus the output length roughly half the size of the input length).
if somebody is looking for swift version of "Covert CGImage to Data" , Here it is :
extension CGImage {
var jpegData: Data? {
guard let mutableData = CFDataCreateMutable(nil, 0),
let destination = CGImageDestinationCreateWithData(mutableData, kUTTypeJPEG, 1, nil)
else {
return nil
}
CGImageDestinationAddImage(destination, self, nil)
guard CGImageDestinationFinalize(destination) else { return nil }
return mutableData as Data
}
}