It seems there are lots of related topics out there, but I was not able to find the answer I'm searching for.
I'm using JavaCV and FFmpegFrameGrabber to get an image from the middle of the video. If an mp4 file has a "Rotation" metadata field (like 90 or 270), I'm getting an image that is positioned not correctly. I wanted to get Orientation from FFmpegFrameGrabber, but could not find a way to do so.
Is there a way to tell FFmpegFrameGrabber to respect orientation, or is there a way to get this value somehow using JavaCV?
Just in case the code that I have so far
FFmpegFrameGrabber g = new FFmpegFrameGrabber(input);
g.start();
g.getVideoMetadata(); // <-- this thing is empty
try {
g.setFrameNumber(g.getLengthInFrames() / 2);
Java2DFrameConverter converter = new Java2DFrameConverter();
Frame frame = g.grabImage();
BufferedImage bufferedImage = converter.convert(frame);
ImageIO.write(bufferedImage, "jpeg", output);
} finally {
g.stop();
}
Related
I have video tag and he plays simple video. [works]
I have canvas2d with playing same video [works]
opencvjs video processing (canvas is output , video is input)- also works
I have three.js with plane mesh
texture = new THREE.CanvasTexture(this.$refs.testcanvas)
texture.needsUpdate = true;
materialLocal = new THREE.MeshBasicMaterial({ map: texture })
materialLocal.needsUpdate = true;
materialLocal.map.needsUpdate = true;
this.mainVideoMesh.material = materialLocal
this.mainVideoMesh.material.needsUpdate = true;
No hepls . I got just first image screen texture and than stops updating.
In runtime i found ->
this.scene.children[2].material.map.needsUpdate: undefined
Strange situation any suggestion.
When using a video as a data source for a texture, the idea is to use the THREE.VideoTexture class. This type of texture will automatically manage its needsUpdate flag in order to ensure new frames are correctly displayed on your plane mesh.
Using THREE.VideoTexture requires that you use the video element as an argument, not the canvas.
I'm using the Xam.Plugin.Media in my Forms app to take pictures.
I take the Image stream (GetStream) and convert to a byte[] and store in my DB.
I then use that as the image source.
On Android and UWP, its working fine.
On iOS, if the picture is taken in portrait mode, the image once, selected is always rotated 90 deg.
I will later, upload this to a server and that image could be used on a different device.
For this, I also tried the GetStreamWithImageRotatedForExternalStorage but in this case, I cant see the image at all.
There are bytes in the stream (I DisplayAlert the length) but the image does not display.
Any idea what I might be doing wrong?
My code:-
private async Task TakePicture(WineDetails details)
{
await CrossMedia.Current.Initialize();
if (CrossMedia.Current.IsCameraAvailable && CrossMedia.Current.IsTakePhotoSupported)
{
var file = await CrossMedia.Current.TakePhotoAsync(new Plugin.Media.Abstractions.StoreCameraMediaOptions
{
AllowCropping = true,
PhotoSize = Plugin.Media.Abstractions.PhotoSize.Medium,
SaveToAlbum = false,
RotateImage = true
});
if (file == null)
return;
using (var ms = new MemoryStream())
{
var stream = file.GetStreamWithImageRotatedForExternalStorage();
stream.CopyTo(ms);
details.LabelImage = ms.ToArray();
details.NotifyChange("ImageSource");
}
}
}
The image is updated in the page via the NotifyChange and looks like this:-
ImageSource.FromStream(() => new MemoryStream(this.LabelImage))
This works fine in all cases on Android and UWP, works on iOS using GetStream (except the image is incorrectly rotated) but does not work using GetStreamWithImageRotatedForExternalStorage on iOS.
Anyone else using this plugin?
Any idea why GetStream returns a rotated image?
Any idea why GetStreamWithImageRotatedForExternalStorage is not working?
Thanks
Update:-
Changed SaveToAlbum = true and when I open the gallery, the image is rotated 90 deg.
Have RotateImage = true which could cause the issue? I'll try setting it to false.
I still can't set the image source to the byte array of the image using GetStreamWithImageRotatedForExternalStorage.
using (var ms = new MemoryStream())
{
file.GetStreamWithImageRotatedForExternalStorage().CopyTo(ms);
details.LabelImage = ms.ToArray();
}
using the byte array for an image
return ImageSource.FromStream(() => new MemoryStream(this.LabelImage));
This does not work for me, GetStream works ok.
Update:-
Ok so, RotateImage = false + GetStreamWithImageRotatedForExternalStorage allows me to display the image but its still incorrectly rotated in my app and the gallery.
I'm using this plugin, which is similar (if not the same thing - I know James Montemagno has recently packaged/bundled his work with Xamarin).
If you check the issues board there, you'll see there are quite a few people that have similar troubles (image rotation on iOS). Almost every 'solution' mentions using GetStreamWithImageRotatedForExternalStorage.
My issue was similar - I was unable to take a photo on iOS in portrait mode, without other (non-ios Devices) rotating the image. I have tried for weeks to solve this issue, but support on the plugin seems to be quite limited.
Ultimately I had to solve this with a huge workaround - using a custom renderer extending from FFImageLoading to display our images and MetadataExtractor. We were then able to extract the EXIF data from the stream and apply a rotation transformation to the FFImageLoding image control.
The rotation information was stored in a sort of weird way, as a string. This is the method I'm using to extract the rotation information, and return the amount it needs to be rotated as an int. Note that for me, iOS was able to display the image correctly still, so it's only returned a rotation change for Android devices.
public static int GetImageRotationCorrection(byte[] image)
{
try
{
var directories = ImageMetadataReader.ReadMetadata(new MemoryStream(image));
if (Device.Android == "Android")
{
foreach (var directory in directories)
{
foreach (var tag in directory.Tags)
{
if (tag.Name == "Orientation")
{
if (tag.Description == "Top, left side(Horizontal / normal)")
return 0;
else if (tag.Description == "Left side, bottom (Rotate 270 CW)")
return 270;
else if (tag.Description == "Right side, top (Rotate 90 CW")
return 90;
}
}
}
}
return 0;
}
catch (Exception ex)
{
return 0;
}
}
Note that there is also a custom renderer for the image for FFImage Loading.
public class RotatedImage : CachedImage
{
public static BindableProperty MyRotationProperty = BindableProperty.Create(nameof(MyRotation), typeof(int), typeof(RotatedImage), 0, propertyChanged: UpdateRotation);
public int MyRotation
{
get { return (int)GetValue(MyRotationProperty); }
set { SetValue(MyRotationProperty, value); }
}
private static void UpdateRotation(BindableObject bindable, object oldRotation, object newRotation)
{
var _oldRotation = (int)oldRotation;
var _newRotation = (int)newRotation;
if (!_oldRotation.Equals(_newRotation))
{
var view = (RotatedImage)bindable;
var transformations = new System.Collections.Generic.List<ITransformation>() {
new RotateTransformation(_newRotation)
};
view.Transformations = transformations;
}
}
}
So, in my XAML - I had declared a RotatedImage instead of the standard Image. With the custom renderer, I'm able to do this and have the image display rotated the correct amount.
image.MyRotation = GetImageRotationCorrection(imageAsBytes)
It's a totally unnecessary workaround - but these are the lengths that I had to go to to get around this issue.
I'll definitely be following along with this question, there might be someone in the community who could help us both!
The SaveMetaData flag is causing the rotation issue.
Setting it to false (default is true) now displays the photo correctly.
One side effect of that, the image no longer appears in the gallery if SaveToAlbum=true.
Still can't use an image take when using GetStreamWithImageRotatedForExternalStorage, even using FFImageLoading.
I found that while using Xam.Plugin.Media v5.0.1 (https://github.com/jamesmontemagno/MediaPlugin), the combination of three different inputs produced different results on Android vs. iOS:
StoreCameraMediaOptions.SaveMetaData
StoreCameraMediaOptions.RotateImage
Using MediaFile.GetStream() vs. MediaFile.GetStreamWithImageRotatedForExternalStorage()
On Android, SaveMetaData = false, RotateImage = true, and using MediaFile.GetStreamWithImageRotatedForExternalStorage() worked for me whether I was saving the result stream externally or processing the result stream locally for display.
On iOS, the combination of RotateImage = true and StreamRotated = true would result in a NullReferenceException coming out of the plugin library. Using MediaFile.GetStreamWithImageRotatedForExternalStorage() appeared to have no impact on behaivor.
--
Before going further, it's important to understand that image orientation in the JPEG format (which Xam.Plugin.Media seems to use) isn't as straightforward as you might think. Rather than rotating the raw image bytes 90 or 180 or 270 degrees, JPEG orientation can be set through embedded EXIF metadata. Orientation issues will happen with JPEGs either if EXIF data is stripped or if downstream consumers don't handle the EXIF data properly.
The approach I landed on was to normalize JPEG image orientation at the point the image is captured without relying on EXIF metadata. This way, downstream consumers shouldn't need to be relied on to properly inspect and handle EXIF orientation metadata.
The basic solution is:
Scan a JPEG for EXIF orientation metadata
Transform the JPEG to rotate/flip as needed
Set the result JPEG's orientation metadata to default
--
Code example compatible with Xamarin, using ExifLib.Standard (1.7.0) and SixLabors.ImageSharp (1.0.4) NuGet packages. Based on (Problem reading JPEG Metadata (Orientation))
using System;
using System.IO;
using ExifLib;
using SixLabors.ImageSharp;
using SixLabors.ImageSharp.Formats.Jpeg;
using SixLabors.ImageSharp.Metadata.Profiles.Exif;
using SixLabors.ImageSharp.Processing;
namespace Your.Namespace
{
public static class ImageOrientationUtility
{
public static Stream NormalizeOrientation(Func<Stream> inputStreamFunc)
{
using Stream exifStream = inputStreamFunc();
using var exifReader = new ExifReader(exifStream);
bool orientationTagExists = exifReader.GetTagValue(ExifTags.Orientation, out ushort orientationTagValue);
if (!orientationTagExists)
// You may wish to do something less aggressive than throw an exception in this case.
throw new InvalidOperationException("Input stream does not contain an orientation EXIF tag.");
using Stream processStream = inputStreamFunc();
using Image image = Image.Load(processStream);
switch (orientationTagValue)
{
case 1:
// No rotation required.
break;
case 2:
image.Mutate(x => x.RotateFlip(RotateMode.None, FlipMode.Horizontal));
break;
case 3:
image.Mutate(x => x.RotateFlip(RotateMode.Rotate180, FlipMode.None));
break;
case 4:
image.Mutate(x => x.RotateFlip(RotateMode.Rotate180, FlipMode.Horizontal));
break;
case 5:
image.Mutate(x => x.RotateFlip(RotateMode.Rotate90, FlipMode.Horizontal));
break;
case 6:
image.Mutate(x => x.RotateFlip(RotateMode.Rotate90, FlipMode.None));
break;
case 7:
image.Mutate(x => x.RotateFlip(RotateMode.Rotate270, FlipMode.Horizontal));
break;
case 8:
image.Mutate(x => x.RotateFlip(RotateMode.Rotate270, FlipMode.None));
break;
}
image.Metadata.ExifProfile.SetValue(ExifTag.Orientation, (ushort)1);
var outStream = new MemoryStream();
image.Save(outStream, new JpegEncoder{Quality = 100});
outStream.Position = 0;
return outStream;
}
}
}
And to use in conjunction with Xam.Plugin.Media:
MediaFile photo = await CrossMedia.Current.TakePhotoAsync(options);
await using Stream stream = ImageOrientationUtility.NormalizeOrientation(photo.GetStream);
In Android4.1, a key frame is often requested in a real-time encoding application. But how to do it using MediaCodec object? The current Android4.2 SDK seems not support it.
You can produce random keyframe by specifying MediaCodec.BUFFER_FLAG_SYNC_FRAME when queuing input buffers:
MediaCodec codec = MediaCodec.createDecoderByType(type);
codec.configure(format, ...);
codec.start();
ByteBuffer[] inputBuffers = codec.getInputBuffers();
for (;;) {
int inputBufferIndex = codec.dequeueInputBuffer(timeoutUs);
if (inputBufferIndex >= 0) {
// fill inputBuffers[inputBufferIndex] with valid data
...
codec.queueInputBuffer(inputBufferIndex, 0, inputBuffers[inputBufferIndex].limit(), presentationTime,
isKeyFrame ? MediaCodec.BUFFER_FLAG_SYNC_FRAME : 0);
}
}
Stumbled upon the need to insert random keyframe when encoding video on Galaxy Nexus.
On it, MediaCodec didn't automatically produce keyframe at the start of the video.
MediaCodec has a method called setParameters which comes to the rescue.
In Kotlin you can do it like:
fun yieldKeyFrame(): Boolean {
val param = Bundle()
param.putInt(MediaCodec.PARAMETER_KEY_REQUEST_SYNC_FRAME, 0)
try {
videoEncoder.setParameters(param)
return true
} catch (e: IllegalStateException) {
return false
}
}
in above snippet, the videoEncoder is an instance of MediaCodec configured to encode.
You can request a periodic key frame by setting the KEY_I_FRAME_INTERVAL key when configuring the encoder. In the example below I am requesting one every two seconds. I've omitted the other keys like frame rate or color format for the sake of clarity, but you will still want to include them.
encoder = MediaCodec.createByCodecName(codecInfo.getName());
MediaFormat inputFormat = MediaFormat.createVideoFormat(mimeType, width, height);
/* ..... set various format options here ..... */
inputFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 2);
encoder.configure(inputFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
encoder.start();
I suspect, however, that what you are really asking is how to request a random key frame while encoding, like at the start of a cut scene. Unfortunately I haven't seen an interface for that. It is possible that stopping and restarting the encoder would have the effect of creating a new key frame at the restart. When I have the opportunity to try that, I'll post the result here.
I hope this was helpful.
Thad Phetteplace - GLACI, Inc.
I'm using openCV to split a video into frames. For that I need the fps and duration. Both of these value return 1 when asking them via cvGetCaptureProperty.
I've made a hack where I use AVURLAsset to get the fps and duration, but when I combine that with openCV I get only a partial video. It seems like it's missing frames.
This is my code right now:
while (cvGrabFrame(capture)) {
frameCounter++;
if (frameCounter % (int)(videoFPS / MyDesiredFramesPerSecond) == 0) {
IplImage *frame = cvCloneImage(cvRetrieveFrame(capture));
// Do Stuff
}
if (frameCounter > duration*fps)
break; // this is here because the loop never stops on its own
}
How can I get all the frames of a video using openCV on iOS? (opencv 2.3.2)
According to the documentation you should check the value returned by cvRetrieveFrame(), if a null pointer is returned you're at the end of the video sequence. Then you break the loop when that happens, instead of relying on the accuracy of FPS*frame_number.
I am trying to render a string over an image chosen by user via Photochooser task. I have seen various replies to similar question but none of the replies have nailed it.
This is what I have come up with -
void photochoosertask_Completed(object sender, PhotoResult e)
{
if (e.TaskResult == TaskResult.OK)
{
System.Windows.Media.Imaging.BitmapImage bmp = new System.Windows.Media.Imaging.BitmapImage();
bmp.SetSource(e.ChosenPhoto);
image1.Source = bmp;
string steamer = "SO!";
System.Windows.Media.Imaging.WriteableBitmap bmps = new System.Windows.Media.Imaging.WriteableBitmap(bmp);
RenderString(bmps, steamer);
}
}
private void RenderString(System.Windows.Media.Imaging.WriteableBitmap bitmap, string steamer)
{
textBlock1.Text = steamer;
bitmap.Render(textBlock1 , null);
bitmap.Invalidate();
}
}
The code however doesn't work. I am most likely doing a major mistake. Any help appreciated, thanks!
According to the documentation:
If an empty transform is supplied [i.e. the null you're passing as the second parameter], the bits representing the element show up at the same offset as if they were placed within their parent.
So if I understand what's happening correctly (and I probably don't), your textBlock1 element is being rendered with the same offset as it has on your parent form. So it may be that textBlock1 is so far down from the top and left that it doesn't show up in your writeable bitmap.
BTW, I'm not familiar with WriteableBitmap, but what you're doing (putting text into a UI element and then rendering that element onto your bitmap) seems like a strange way to add text to a bitmap.
I just figured it out. Thought I should post the solution code here, might help somebody - someday :)
//setup a writeable bitmap with required dimensions
System.Windows.Media.Imaging.WriteableBitmap wbmps = new System.Windows.Media.Imaging.WriteableBitmap(x,y);
//set up a transform, we'll use ScaleTransform and we'll keep things simple here, 1x on both the axis
ScaleTransform transform = new System.Windows.Media.ScaleTransform();
transform.ScaleX=1;
transform.ScaleY=1;
//now we need to render the image on the writeablebitmap and follow it up by rendering a //string
wbmps.Render(imageelement,transform);
//Now render the string which is equivalent to TextBlock.Text
wbmps.Render(texblock,transform);
//Finally - redraw the writeablebitmap to complete the rendering
wbmps.Invalidate();