Unfortunately, there appears to be no way to using a built-in method on iOS to extract 32 bit RGBA data from a PNG file without losing the alpha channel reference. Therefore, some people have been using libpng to extract their OpenGL textures. However, all the examples have required the png file to be loaded from a file. Assuming these textures are imported over a network connection, they would have to be saved to files from NSData and then read. What is the best way to extract raw PNG data into raw OpenGL RGBA texture data?
Ended up writing a category which solves this problem using the customization capabilities of libpng. Posted a gist here: https://gist.github.com/joshcodes/5681512
Hopefully this helps someone else who needs to know how this is done. The essential part is creating a method
void user_read_data(png_structp png_ptr, png_bytep data, png_size_t length)
{
void *nsDataPtr = png_get_io_ptr(png_ptr);
ReadStream *readStream = (ReadStream*)nsDataPtr;
memcpy(data, readStream->source + readStream->index, length);
readStream->index += length;
}
and using
// init png reading
png_set_read_fn(png_ptr, &readStream, user_read_data);
as a custom read method.
Related
My server doesn't support the HEIF format. So I need to transform it to JPEG before uploading from my app.
I do this:
UIImage *image = [UIImage imageWithData:imageData];
NSData *data=UIImageJPEGRepresentation(image, 1.0);
But how can I know that the data is HEIF (or HEIC) ? I can look at a file:
([filePath hasSuffix:#".HEIC"] || [filePath hasSuffix:#".heic"])
But I don't think it's a good answer. Is there any other solution?
Both existing answers have good recommendations, but to attempt to tell the whole story...
UIImage doesn't represent an image file or even binary data in an image-file format. A UIImage is best thought of as an abstract representation of the displayable image encoded in that data — that is, a UIImage is the result of the decoding process. By the time you have a UIImage object, it doesn't care what file format it came from.
So, as #Ladislav's answer notes, if you have a UIImage already and you just want to get data in a particular image file format, call one of the convenience functions for getting a UIImage into a file-formatted data. As its name might suggest, UIImageJPEGRepresentation returns data appropriate for writing to a JPEG file.
If you already have a UIImage, UIImageJPEGRepresentation is probably your best bet, since you can use it regardless of the original image format.
As #ScottCorscadden implies, if you don't have a UIImage (yet) because you're working at a lower level such that you have access to the original file data, then you'll need to inspect that data to divine its format, or ask whatever source you got the data from for metadata describing its format.
If you want to inspect the data itself, you're best off reading up on the HIEF format standards. See nokiatech, MPEG group, or wikipedia.
There's a lot going on in the HEIF container format and the possible kinds of media that can be stored within, so deciding if you have not just a HEIF file, but an HEIF/HEVC file compatible with this-or-that viewer could be tricky. Since you're talking about excluding things your server doesn't support, it might be easier to code from the perspective of including only the things that your server does support. That is, if you have data with no metadata, look for something like the JPEG magic number 0xffd8ff, and use that to exclude anything that isn't JPEG.
Better, though, might be to look for metadata. If you're picking images from the Photos library with PHImageManager.requestImageData(for:options:resultHandler:), the second parameter to your result handler is the Uniform Type Identifier for the image data: for HEIF and HEIC files, public.heif, public.heif-standard, and public.heic have been spotted in the wild.
(Again, though, if you're looking for "images my sever doesn't support", you're better off checking for the formats your server does support and rejecting anything not on that list, rather than trying to identify all the possible unsupported formats.)
When you are sending to your server you are most likely decoding the UIImage and sending it as Data so just do
let data = UIImageJPEGRepresentation(image, 0.9)
Just decide what quality works best for you, here it is 0.9
A bit late to the party, but other than checking the extension (after the last dot), you can also check for the "magic number" aka file signature. Byte 5 to 8 should give you the constant "ftyp". The following 4 bytes would be the major brand, which I believe is one of "mif1", "heic" and "heix".
For example, the first 12 bytes of a .heic image would be:
00 00 00 18 66 74 79 70 6d 69 66 31
which, after removing 0s and trim the result, literally decoded to ftypmif1.
Well, you could look at magic bytes - JPEG and PNG certainly are known, and I seem to see some references that HEIF (.heic) starts with a NUL byte. If you're using any of the PHImageManager methods like requestImageDataForAsset:options:resultHandler, that resultHandler will be passed a NSString * _Nullable dataUTI reference. There's a decent WWDC video/slides on this (possibly here) that suggest if the UTI is not kUTTypeJPEG you convert it (and the slides have some lower-level sample code in swift to do it that preserve orientation too).
I should also mention, if you have control at your app layer and all uploads come from there, do all this there.
If you're using Photos framework and are importing images from photo library, there's a solution that was mentioned briefly during WWDC17. First, import core services:
import MobileCoreServices
Then, when you request the image, check the UTType that is returned as a second parameter to your block:
// asset: PHAsset
PHImageManager.default().requestImageData(for: asset, options: nil) { imageData, dataUTI, orientation, info in
guard let dataUTI = dataUTI else { return }
if !(UTTypeConformsTo(dataUTI as CFString, kUTTypeJPEG) || UTTypeConformsTo(dataUTI as CFString, kUTTypePNG)) {
// imageData is neither JPG not PNG, possibly subject for transcoding
}
}
Other UTTypes can be found here
Ok, I know that this question is almost the same as get-rtsp-stream-from-live555-and-decode-with-avfoundation, but now VideoToolbox for iOS8 became public for use and although I know that it can be done using this framework, I have no idea of how to do this.
My goals are:
Connect with a WiFiCamera using rtsp protocol and receive stream data (Done with live555)
Decode the data and convert to UIImages to display on the screen (motionJPEG like)
And save the streamed data on a .mov file
I reached all this goals using ffmpeg, but unfortunately I can't use it due to my company's policy.
I know that I can display on the screen using openGL too, but this time I have to convert to UIImages. I also tried to use the libraries below:
ffmpeg: can't use this time due to company's policy. (don't ask me why)
libVLC: display lags about 2secs and I don't have access to stream data to save into a .mov file...
gstreamer: same as above
I believe that live555 + VideoToolbox will do the job, just can't figure out how to do this happen ...
I did it. VideoToolbox is still poor documented and we have no much information about video programming (without using ffmpeg) so it cost me more time than I really expected.
For stream using live555, I got the SPS and PPS info to create the CMVideoFormatDescription like this:
const uint8_t *props[] = {[spsData bytes], [ppsData bytes]};
size_t sizes[] = {[spsData length], [ppsData length]};
OSStatus result = CMVideoFormatDescriptionCreateFromH264ParameterSets(NULL, 2, props, sizes, 4, &videoFormat);
Now, the difficult part (because I'm noob on video programming): Replace the NALunit header with a 4 byte length code as described here
int headerEnd = 23; //where the real data starts
uint32_t hSize = (uint32_t)([rawData length] - headerEnd - 4);
uint32_t bigEndianSize = CFSwapInt32HostToBig(hSize);
NSMutableData *videoData = [NSMutableData dataWithBytes:&bigEndianSize length:sizeof(bigEndianSize)];
[videoData appendData:[rawData subdataWithRange:NSMakeRange(headerEnd + 4, [rawData length] - headerEnd - 4)]];
Now I was able to create a CMBlockBuffer successfully using this raw data and pass the buffer to VTDecompressionSessionDecodeFrame. From here is easy to convert the response CVImageBufferRef to UIImage... I used this stack overflow thread as reference.
And finally, save the stream data converted on UIImage following the explanation described on How do I export UIImage array as a movie?
I just posted a little bit of my code because I believe this is the important part, or in other words, it is where I was having problems.
I'm using AVAssetWriter/AVAssetWriterInput to write H.264 raw data to an MP4 file. As I'm receiving the data from a remote server, I use the following CoreMedia APIs to get a sample buffer (CMSampleBufferRef) containing the H.264 data in AVCC format that is in turned appended to an MP4 file by sending to an AVAssetWriterInput the message (BOOL)appendSampleBuffer:(CMSampleBufferRef)sampleBuffer:
CMBlockBufferCreateWithMemoryBlock to create a memory block
CMBlockBufferReplaceDataBytes to write the H.264 in AVCC format to the memory block
CMSampleBufferCreate to create a sample buffer with the memory block and a format descriptor containing the H.264 "extradata"
Everything works as expected, the only problem with this approach is that I'm periodically calling the above APIs and what I would really like is instead to be able to reuse the resources allocated - in particular CMSampleBufferRef and CMBlockBufferRef. Basically, I would like to have a pool of CMSampleBuffer's and be able to update its memory content and format descriptor as I'm receiving new H.264 data from the remote server.
I know that exists AVAssetWriterInputPixelBufferAdaptorthat gives access to a CVPixelBufferPool but, I can't use it in my case because as far as I know, to properly instantiate a pixel buffer adaptor, at minimum I need to be able to pass the video frame dimensions which I would' know until I parse the stream. Further, I don't know how to write the H.264 "extradata" with a CVPixelBuffer. So, I'm thinking that I need to stick with CMSampleBuffer. Unfortunately, it seems that CoreMedia APIs don't offer the possibility to update the memory block nor the format descriptor of a sample buffer once created (as far as I can tell, I only have access to immutable references of those objects). Thus, the best I can do so far is to reuse the memory block CMBlockBufferRef but I'm still recreating the sample buffer. My code is below. Hopefully someone here will have some ideas on how to implement a pool of CMSampleBuffer's or perhaps a more efficient way to write H.264 AVCC stream to MP4?
- (CMSampleBufferRef)sampleBufferWithData:(NSData*)data formatDescriptor:(CMFormatDescriptionRef)formatDescription
{
OSStatus result;
CMSampleBufferRef sampleBuffer = NULL;
// _blockBuffer is a CMBlockBufferRef instance variable
if (!_blockBuffer)
{
size_t blockLength = MAX_LENGTH;
result = CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault,
NULL,
blockLength,
kCFAllocatorDefault,
NULL,
0,
blockLength,
kCMBlockBufferAssureMemoryNowFlag,
&_blockBuffer);
// check error
}
result = CMBlockBufferReplaceDataBytes([data bytes], _blockBuffer, 0, [data length]);
// check error
const size_t sampleSizes = [data length];
CMSampleTimingInfo timing = [self sampleTimingInfo];
result = CMSampleBufferCreate(kCFAllocatorDefault,
_blockBuffer,
YES,
NULL,
NULL,
formatDescription,
1,
1,
&timing,
1,
&sampleSizes,
&sampleBuffer);
// check error
return sampleBuffer;
}
If you are receiving raw H.264 data, then there is not much do to and no need to deal with CoreMedia at all.
Buffer all VCL NAL units until you get SPS/PPS NAL units. Create the extradata from them, then just append all buffered and new VCL NAL units to the file. In case you are received the NAL units in Annex B format you need to convert them to AVCC format (basically replacing the start code with a length code)
You only need to work with 'CMSampleBuffer' if you want to decode uncompressed pictures or if you want to decode compressed pictures. As you are already working with a raw H.264 stream and just want to write it into an MP4 file, just do so. No need to touch CoreMedia at all here.
Regarding CoreMedia: you wrap your video information in a CMBlockBuffer. This buffers together with a CMVideoFormatDescriptor (generated from SPS/PPS) plus CMTime make up a CMSampleBuffer. And multiple CMSampleBuffers make up a 'CMSampleBufferPool'.
'CVPixelBuffer' and 'CVPixelBufferPool' are not involved. These are either the input or output of a 'VTCompressionSession' or "VTDecompressionSession' when dealing with encoding/decoding h.264 video.
As said in your case, no need to touch any of the core framworks at all as you are just creating a file.
An overview about Annex B and AVCC stream format can be found here: Possible Locations for Sequence/Picture Parameter Set(s) for H.264 Stream
In my application I have to send images of different formats to the server (it must be all file formats that can be read by the UIImage class) https://developer.apple.com/library/ios/#documentation/uikit/reference/UIImage_Class/Reference/Reference.html
And the problem is: I don't know when I should use each of this methods. Of course it's obvious that for .png images I need to use UIImagePNGRepresentation and for .jpg/.jpeg UIImageJPEGRepresentation. But what about other formats (.tiff,.gif , etc.)? There are only two methods for image manipulations and so many formats.
You say:
Of course it's obvious that for .png images I need to use UIImagePNGRepresentation and for .jpg/.jpeg UIImageJPEGRepresentation.
No, that's not necessarily the case. If you have some original "digital asset", rather than creating a UIImage and then using one of those two functions to create the NSData that you'll upload, you will often just load the NSData from the original asset and bypass the round-trip to a UIImage at all. If you do this, you don't risk any loss of data that converting to a UIImage, and then back again, can cause.
There are some additional considerations, though:
Meta data:
These UIImageXXXRepresentation functions strip the image of its meta data. Sometimes that's a good thing (e.g. you don't want to upload photos of your children or expensive gadgets the include the GPS locations where malcontents could identify where the shot was taken). In other cases, you don't want the meta data to be thrown away (e.g. date of the original shot, which camera, etc.).
You should make an explicit decision as to whether you want meta data stripped or not. If not, don't round-trip your image through a UIImage, but rather use the original asset.
Image quality loss and/or file size considerations:
I'm particularly not crazy about UIImageJPEGRepresentation because it a lossy compression. Thus, if you use a compressionQuality value smaller than 1.0, you can lose some image quality (modest quality loss for values close to 1.0, more significant quality loss with lower compressionQuality values). And if you use a compressionQuality of 1.0, you mitigate much of the JPEG image quality loss, but the resulting NSData can often be bigger than the original asset (at least if the original was, itself, a compressed JPEG or PNG), resulting in slower uploads.
UIImagePNGRepresentation doesn't introduce compression-based data loss, but depending upon the image, you may still lose data (e.g. if the original file was a 48-bit TIFF or used a colorspace other than sRGB).
It's a question of whether you are ok with some image quality loss and/or larger file size during the upload process.
Image size:
Sometimes you don't want to upload the full resolution image. For example, you might be using a web service that wants images no bigger than 800px per side. Or if you're uploading a thumbnail, they might want something even smaller (e.g. 32px x 32px). By resizing images, you can make the upload much smaller and thus much faster (though with obvious quality loss). But if you use an image resizing algorithm, then creating a PNG or JPEG using these UIImageXXXRepresentation functions would be quite common.
In short, if I'm trying to minimize the data/quality loss, I would upload the original asset if it's in a format that the server accepts, and I'd use UIImagePNGRepresentation (or UIImageJPGRepresentation with quality setting of 1.0) if the original asset was not in a format accepted by the server. But the choice of using these UIImageXXXRepresentation functions is a question of your business requirements and what the server accepts.
Rob points out a lot of very good things to consider when working with images (+1), however here is an example of how to create tiff's and gif's as you asked:
First, you need to link to the ImageIO library (under the Build Phases of your app).
Next you need to #import <ImageIO/ImageIO.h> at the top of your file.
Then, the following code will convert the image for you:
// Get a reference to the image that you already have stored on disk somehow.
// If it isn't stored on disk, then you can use CGImageSourceCreateWithData() to create it from an NSData representation of your image.
NSURL *url = [[NSBundle mainBundle] URLForResource:#"01" withExtension:#"jpg"];
CGImageSourceRef src = CGImageSourceCreateWithURL((__bridge CFURLRef)(url), NULL);
// Create a URL referencing the Application Support Directory. We will save the new image there.
NSFileManager *fm = [NSFileManager defaultManager];
NSURL *suppurl = [fm URLForDirectory:NSApplicationSupportDirectory
inDomain:NSUserDomainMask
appropriateForURL:nil
create:YES
error:NULL];
// Append the name of the output file to the app support directory
// For tiff change the extension in the next line to .tiff
NSURL *gifURL = [suppurl URLByAppendingPathComponent:#"mytiff.gif"];
// Create the destination for the new image
// For tiff, use #"public.tiff" for the second argument of the next line (instead of #com.compuserve.gif".
CGImageDestinationRef dest = CGImageDestinationCreateWithURL((__bridge CFURLRef)gifURL,
(CFStringRef)#"com.compuserve.gif",
1,
NULL);
CGImageDestinationAddImageFromSource(dest, src, 0, NULL);
// Write the image data to the URL.
bool ok = CGImageDestinationFinalize(dest);
if (!ok)
NSLog(#"Unable to create gif file.");
// Cleanup
CFRelease(src);
CFRelease(dest);
This was adapted from the code in this book.
The situation is this:
I've written a simple MovieClip replacement that converts an existing imported MovieClip to a sequence of BitmapData. This removes the requirement for Flash to render vector data in the MovieClip on each frame.
But BitmapData has a huge memory footprint. I've tried converting the BitmapData to a ByteArray and using the compress() method. This results in a significantly smaller memory footprint. But it has proven impractical. For each redraw, I tried uncompressing()'ing the ByteArray, then using SetPixels to blit the data to the screen, then re-compressing() the frame. This works but is terribly slow.
So I was wondering if anybody else has an approach I could try. In Flash, is it possible to compress bitmap data in memory and quickly blit it to the screen?
I wonder how native animated GIFs work in Flash. Does it uncompress them to BitmapData behind the scenes, or is frame decompression done on the fly?
Perhaps there is an Alchemy project that attempts to blit compressed images?
Thanks for any advice you can offer :)
#thienhaflash's response is good but has aged a year and since then Flash Player and AIR Runtime have expanded their capabilities. Today I stumbeled on this little tidbit from Adobe's AS3 Guide. As of player 11.3 there are native image compression techniques available. Here's a snippet:
// Compress a BitmapData object as a JPEG file.
var bitmapData:BitmapData = new BitmapData(640,480,false,0x00FF00);
var byteArray:ByteArray = new ByteArray();
bitmapData.encode(new Rectangle(0,0,640,480), new flash.display.JPEGEncoderOptions(), byteArray);
Not sure about the practicality for blitting but it's nice that it can be done natively.
For memory reservation you need to think twice before convert a MovieClip to a Bitmap sequence. Is it really that need ? Can you break things down as there are several things (like the background) is static (or just moving around) why don't cache bitmap for each elements instead of one big Bitmap sequence ?
I usually used AnimatedBitmap (the name for bitmap sequence alternative for a MovieClip) only for small size animated icons, and other heavy calculation stuffs (like fire / smoke effects ...). Just break things down as much as you can !
As far as i know, there are no way to compress the memory used by a BitmapData located in the memory and there are nothing related to Alchemy could help improve memory used in this case.
Animated GIF won't works in Flash natively, you will need some library to do that. Search for AnimatedGIF as3 library from bytearray.com, actually the library just read the gif file in raw byteArray and convert to an animatedBitmap just like how you've done.
this is an old question, but there is recent info on this : jackson Dunstan has had a run with bitmapdatas and it turns out that Bitmap data obtained from compressed sources will "deflate" after some time unused.
here are the articles : http://jacksondunstan.com/articles/2112, and the two referred at the beginning of it.
So you could absolutely do something like :
var byteArray:ByteArray = new ByteArray();
myBitmapData.encode(new Rectangle(0,0,640,480), new flash.display.JPEGEncoderOptions(), byteArray);
var loader = new Loader();
loader.addEventListener(Event.COMPLETE, function(_e:Event):void{
if(loader.content is Bitmap){
myBitmapData.dispose()
myBitmapData= Bitmap(loader.content).bitmapData;
}
});
loader.loadBytes(byteArray);
I'm not sure if it would work as is, and you definitely want to handle your memory better. but now, myBitmapData will be uncompressed when you try to read from it, and then re-compressed when you don't use it for about ten seconds.