Unity 3d 2017.2.1 camera feed looks zoomed in iPad - ipad

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
public class CameraFeed : MonoBehaviour {
// Use this for initialization
public Renderer rend;
void Start () {
WebCamDevice[] devices = WebCamTexture.devices;
// for debugging purposes, prints available devices to the console
for(int i = 0; i < devices.Length; i++)
{
print("Webcam available: " + devices[i].name);
}
Renderer rend = this.GetComponentInChildren<Renderer>();
// assuming the first available WebCam is desired
WebCamTexture tex = new WebCamTexture(devices[0].name);
rend.material.mainTexture = tex;
//rend.material.SetTextureScale("tex", new Vector2(1.2f, 1.2f));
tex.Play();
}
// Update is called once per frame
void Update () {
}
}
I use the above code to render camera feed on a plane. The code works fine but I see zoomed feed(around 1.2x) when deployed to iPad. How can I get the normal feed. Or How do I zoom out the camera feed

According to https://answers.unity.com/questions/1421387/webcamtexture-zoomed-in-on-ipad-pro.html if you insert this code it should scale properly:
rend.material.SetTextureScale("_Texture", new Vector2(1f, 1f))
Furthermore, it seems that you were trying something like this but your vector was off (by 0.2f) though you have commented out that line as well. You could reduce the scale even more though that could make the image just smaller if the camera couldn't support that zoomout :).

Related

Pass iOS CMDeviceMotion to THREE.js to Rotate Spherical Panorama

I need to pass iOS app motion data to my WKWebView that loads a spherical panorama using THREE.js. The THREE.js relies on Quaternion calculated from alpha, beta, gamma, but since CMDeviceMotion on iOS does not provide those, but does provide Quaternion, I decided to pass Quaternion to the THREE.js web code from Swift iOS like this:
func didUpdateMotion(data: CMDeviceMotion) {
let quaternion = data.attitude.quaternion
let js = "iosQuaternion={x:\(quaternion.x),y:\(quaternion.y),z:\(quaternion.z),w:\(quaternion.w)}"
self.webView.evaluateJavaScript(js, completionHandler: {(result,error) in })}}
}
Now, here is a part of my web view code for THREE.js can uses Quaternion set by iOS:
function animate() {
if(typeof iosQuaternion === "undefined") return;
window.requestAnimationFrame( animate );
updateWithQuaternion();
renderer.render( scene, camera );
}
updateWithQuaternion = ( function () {
var lastQuaternion = new THREE.Quaternion();
return function () {
if (typeof iosQuaternion === "undefined") return;
scope.object.quaternion = new THREE.Quaternion();
scope.object.quaternion.x = iosQuaternion.x;
scope.object.quaternion.y = iosQuaternion.y;
scope.object.quaternion.z = iosQuaternion.z;
if ( 8 * ( 1 - lastQuaternion.dot( scope.object.quaternion ) ) > EPS ) {
lastQuaternion.copy( scope.object.quaternion );
scope.dispatchEvent( changeEvent );
}
};
})();
The above implementation does give me a functionality where spherical panorama is loaded and rotates based on passed device motion data, but the direction and initial orientation are completely off. Panorama is loaded in incorrect position (may be facing down when it should face a specific direction) and rotation direction is also affected as a result.
What may be a recommendation for me to ensure iOS motion data is correctly passed to the spherical panorama driven by THREE.js?

Xamarin.Forms Xam.Plugin.Media Take Picture on iOS

I'm using the Xam.Plugin.Media in my Forms app to take pictures.
I take the Image stream (GetStream) and convert to a byte[] and store in my DB.
I then use that as the image source.
On Android and UWP, its working fine.
On iOS, if the picture is taken in portrait mode, the image once, selected is always rotated 90 deg.
I will later, upload this to a server and that image could be used on a different device.
For this, I also tried the GetStreamWithImageRotatedForExternalStorage but in this case, I cant see the image at all.
There are bytes in the stream (I DisplayAlert the length) but the image does not display.
Any idea what I might be doing wrong?
My code:-
private async Task TakePicture(WineDetails details)
{
await CrossMedia.Current.Initialize();
if (CrossMedia.Current.IsCameraAvailable && CrossMedia.Current.IsTakePhotoSupported)
{
var file = await CrossMedia.Current.TakePhotoAsync(new Plugin.Media.Abstractions.StoreCameraMediaOptions
{
AllowCropping = true,
PhotoSize = Plugin.Media.Abstractions.PhotoSize.Medium,
SaveToAlbum = false,
RotateImage = true
});
if (file == null)
return;
using (var ms = new MemoryStream())
{
var stream = file.GetStreamWithImageRotatedForExternalStorage();
stream.CopyTo(ms);
details.LabelImage = ms.ToArray();
details.NotifyChange("ImageSource");
}
}
}
The image is updated in the page via the NotifyChange and looks like this:-
ImageSource.FromStream(() => new MemoryStream(this.LabelImage))
This works fine in all cases on Android and UWP, works on iOS using GetStream (except the image is incorrectly rotated) but does not work using GetStreamWithImageRotatedForExternalStorage on iOS.
Anyone else using this plugin?
Any idea why GetStream returns a rotated image?
Any idea why GetStreamWithImageRotatedForExternalStorage is not working?
Thanks
Update:-
Changed SaveToAlbum = true and when I open the gallery, the image is rotated 90 deg.
Have RotateImage = true which could cause the issue? I'll try setting it to false.
I still can't set the image source to the byte array of the image using GetStreamWithImageRotatedForExternalStorage.
using (var ms = new MemoryStream())
{
file.GetStreamWithImageRotatedForExternalStorage().CopyTo(ms);
details.LabelImage = ms.ToArray();
}
using the byte array for an image
return ImageSource.FromStream(() => new MemoryStream(this.LabelImage));
This does not work for me, GetStream works ok.
Update:-
Ok so, RotateImage = false + GetStreamWithImageRotatedForExternalStorage allows me to display the image but its still incorrectly rotated in my app and the gallery.
I'm using this plugin, which is similar (if not the same thing - I know James Montemagno has recently packaged/bundled his work with Xamarin).
If you check the issues board there, you'll see there are quite a few people that have similar troubles (image rotation on iOS). Almost every 'solution' mentions using GetStreamWithImageRotatedForExternalStorage.
My issue was similar - I was unable to take a photo on iOS in portrait mode, without other (non-ios Devices) rotating the image. I have tried for weeks to solve this issue, but support on the plugin seems to be quite limited.
Ultimately I had to solve this with a huge workaround - using a custom renderer extending from FFImageLoading to display our images and MetadataExtractor. We were then able to extract the EXIF data from the stream and apply a rotation transformation to the FFImageLoding image control.
The rotation information was stored in a sort of weird way, as a string. This is the method I'm using to extract the rotation information, and return the amount it needs to be rotated as an int. Note that for me, iOS was able to display the image correctly still, so it's only returned a rotation change for Android devices.
public static int GetImageRotationCorrection(byte[] image)
{
try
{
var directories = ImageMetadataReader.ReadMetadata(new MemoryStream(image));
if (Device.Android == "Android")
{
foreach (var directory in directories)
{
foreach (var tag in directory.Tags)
{
if (tag.Name == "Orientation")
{
if (tag.Description == "Top, left side(Horizontal / normal)")
return 0;
else if (tag.Description == "Left side, bottom (Rotate 270 CW)")
return 270;
else if (tag.Description == "Right side, top (Rotate 90 CW")
return 90;
}
}
}
}
return 0;
}
catch (Exception ex)
{
return 0;
}
}
Note that there is also a custom renderer for the image for FFImage Loading.
public class RotatedImage : CachedImage
{
public static BindableProperty MyRotationProperty = BindableProperty.Create(nameof(MyRotation), typeof(int), typeof(RotatedImage), 0, propertyChanged: UpdateRotation);
public int MyRotation
{
get { return (int)GetValue(MyRotationProperty); }
set { SetValue(MyRotationProperty, value); }
}
private static void UpdateRotation(BindableObject bindable, object oldRotation, object newRotation)
{
var _oldRotation = (int)oldRotation;
var _newRotation = (int)newRotation;
if (!_oldRotation.Equals(_newRotation))
{
var view = (RotatedImage)bindable;
var transformations = new System.Collections.Generic.List<ITransformation>() {
new RotateTransformation(_newRotation)
};
view.Transformations = transformations;
}
}
}
So, in my XAML - I had declared a RotatedImage instead of the standard Image. With the custom renderer, I'm able to do this and have the image display rotated the correct amount.
image.MyRotation = GetImageRotationCorrection(imageAsBytes)
It's a totally unnecessary workaround - but these are the lengths that I had to go to to get around this issue.
I'll definitely be following along with this question, there might be someone in the community who could help us both!
The SaveMetaData flag is causing the rotation issue.
Setting it to false (default is true) now displays the photo correctly.
One side effect of that, the image no longer appears in the gallery if SaveToAlbum=true.
Still can't use an image take when using GetStreamWithImageRotatedForExternalStorage, even using FFImageLoading.
I found that while using Xam.Plugin.Media v5.0.1 (https://github.com/jamesmontemagno/MediaPlugin), the combination of three different inputs produced different results on Android vs. iOS:
StoreCameraMediaOptions.SaveMetaData
StoreCameraMediaOptions.RotateImage
Using MediaFile.GetStream() vs. MediaFile.GetStreamWithImageRotatedForExternalStorage()
On Android, SaveMetaData = false, RotateImage = true, and using MediaFile.GetStreamWithImageRotatedForExternalStorage() worked for me whether I was saving the result stream externally or processing the result stream locally for display.
On iOS, the combination of RotateImage = true and StreamRotated = true would result in a NullReferenceException coming out of the plugin library. Using MediaFile.GetStreamWithImageRotatedForExternalStorage() appeared to have no impact on behaivor.
--
Before going further, it's important to understand that image orientation in the JPEG format (which Xam.Plugin.Media seems to use) isn't as straightforward as you might think. Rather than rotating the raw image bytes 90 or 180 or 270 degrees, JPEG orientation can be set through embedded EXIF metadata. Orientation issues will happen with JPEGs either if EXIF data is stripped or if downstream consumers don't handle the EXIF data properly.
The approach I landed on was to normalize JPEG image orientation at the point the image is captured without relying on EXIF metadata. This way, downstream consumers shouldn't need to be relied on to properly inspect and handle EXIF orientation metadata.
The basic solution is:
Scan a JPEG for EXIF orientation metadata
Transform the JPEG to rotate/flip as needed
Set the result JPEG's orientation metadata to default
--
Code example compatible with Xamarin, using ExifLib.Standard (1.7.0) and SixLabors.ImageSharp (1.0.4) NuGet packages. Based on (Problem reading JPEG Metadata (Orientation))
using System;
using System.IO;
using ExifLib;
using SixLabors.ImageSharp;
using SixLabors.ImageSharp.Formats.Jpeg;
using SixLabors.ImageSharp.Metadata.Profiles.Exif;
using SixLabors.ImageSharp.Processing;
namespace Your.Namespace
{
public static class ImageOrientationUtility
{
public static Stream NormalizeOrientation(Func<Stream> inputStreamFunc)
{
using Stream exifStream = inputStreamFunc();
using var exifReader = new ExifReader(exifStream);
bool orientationTagExists = exifReader.GetTagValue(ExifTags.Orientation, out ushort orientationTagValue);
if (!orientationTagExists)
// You may wish to do something less aggressive than throw an exception in this case.
throw new InvalidOperationException("Input stream does not contain an orientation EXIF tag.");
using Stream processStream = inputStreamFunc();
using Image image = Image.Load(processStream);
switch (orientationTagValue)
{
case 1:
// No rotation required.
break;
case 2:
image.Mutate(x => x.RotateFlip(RotateMode.None, FlipMode.Horizontal));
break;
case 3:
image.Mutate(x => x.RotateFlip(RotateMode.Rotate180, FlipMode.None));
break;
case 4:
image.Mutate(x => x.RotateFlip(RotateMode.Rotate180, FlipMode.Horizontal));
break;
case 5:
image.Mutate(x => x.RotateFlip(RotateMode.Rotate90, FlipMode.Horizontal));
break;
case 6:
image.Mutate(x => x.RotateFlip(RotateMode.Rotate90, FlipMode.None));
break;
case 7:
image.Mutate(x => x.RotateFlip(RotateMode.Rotate270, FlipMode.Horizontal));
break;
case 8:
image.Mutate(x => x.RotateFlip(RotateMode.Rotate270, FlipMode.None));
break;
}
image.Metadata.ExifProfile.SetValue(ExifTag.Orientation, (ushort)1);
var outStream = new MemoryStream();
image.Save(outStream, new JpegEncoder{Quality = 100});
outStream.Position = 0;
return outStream;
}
}
}
And to use in conjunction with Xam.Plugin.Media:
MediaFile photo = await CrossMedia.Current.TakePhotoAsync(options);
await using Stream stream = ImageOrientationUtility.NormalizeOrientation(photo.GetStream);

ARCore Unity: How do I Start and Stop Plane Detection on Command?

I am creating an app with ARCore, but I don't want ARCore to look for planes as soon as the app starts. Instead, I want the plane detection to begin when I hit a button in my app. It would also be great if I could stop the plane detection on command as well.
Does anyone know how I could do start and stop the ARCore plane detection on command?
I am building the app in Unity.
Thanks so much in advance!
on ARPlaneVisualizer.cs, there is this code
void OnEnable()
{
m_PlaneLayer = LayerMask.NameToLayer ("ARGameObject");
ARInterface.planeAdded += PlaneAddedHandler;
ARInterface.planeUpdated += PlaneUpdatedHandler;
ARInterface.planeRemoved += PlaneRemovedHandler;
HidePlane(true);
}
void OnDisable()
{
ARInterface.planeAdded -= PlaneAddedHandler;
ARInterface.planeUpdated -= PlaneUpdatedHandler;
ARInterface.planeRemoved -= PlaneRemovedHandler;
HidePlane(false);
}
you can use the OnEnable() code as start tracking and OnDisable() code to stop tracking.
Initially create bool to restrict surface detection code and inatially make bool to true.
bool isSurfaceDetected = true;
if (isSurfaceDetected) {
Session.GetTrackables<TrackedPlane> (_newPlanes, TrackableQueryFilter.New);
// Iterate over planes found in this frame and instantiate corresponding GameObjects to visualize them.
foreach (var curPlane in _newPlanes) {
// Instantiate a plane visualization prefab and set it to track the new plane. The transform is set to
// the origin with an identity rotation since the mesh for our prefab is updated in Unity World
// coordinates.
var planeObject = Instantiate (plane, Vector3.zero, Quaternion.identity,
transform);
planeObject.GetComponent<DetectedPlaneVisualizer> ().Initialize (curPlane);
// Debug.Log ("test....");
// Apply a random color and grid rotation.
// planeObject.GetComponent<Renderer>().material.SetColor("_GridColor", new Color(Random.Range(0.0f, 1.0f), Random.Range(0.0f, 1.0f), Random.Range(0.0f, 1.0f)));
// planeObject.GetComponent<Renderer>().material.SetFloat("_UvRotation", Random.Range(0.0f, 360.0f));
//
}
Create a stop button in canvas and attatch below method
public void StopTrack()
{
// Make isSurfaceDetected to false to disable plane detection code
isSurfaceDetected = false;
// Tag DetectedPlaneVisualizer prefab to Plane(or anything else)
GameObject[] anyName = GameObject.FindGameObjectsWithTag ("Plane");
// In DetectedPlaneVisualizer we have multiple polygons so we need to loop and diable DetectedPlaneVisualizer script attatched to that prefab.
for (int i = 0; i < anyName.Length; i++)
{
anyName[i].GetComponent<DetectedPlaneVisualizer> ().enabled = false;
}
}
Make sure that stop button method is in ARController

How we can implement object recognition in real time through Augmented Reality way?

I need to implement image recognition in real time. So I have tried metaio SDK for Augmented reality. In Metaio SDK I have used extended image tracking. But through that, only one image and text to that image can be added in static. So that if I scan that image, I can get the text. How can we achieve this dynamically for object recognition through Augmented Reality way. Are there any other SDK's(trial versions) that supports object recognition and 3D tracking.
Help is highly appreciable.
Thanks in advance,
laxmi.
Well, I don't know about Metaio, but Vuforia has this feature you seek.
Vuforia has this concept of User-defined targets. You can download the Vuforia SDK or Unity extension of Vuforia ot do so.
I personally use Vuforia Unity and the ARCamera from there just needs to be changed so that it fits the User Defined settings.
After that you simply need to add this script that allows access to the camera Images as follows:
using UnityEngine;
using System.Collections;
public class CameraImageAccess : MonoBehaviour
{
private Image.PIXEL_FORMAT m_PixelFormat = Image.PIXEL_FORMAT.RGB888;
private bool m_RegisteredFormat = false;
private bool m_LogInfo = true;
void Start()
{
QCARBehaviour qcarBehaviour = (QCARBehaviour) FindObjectOfType(typeof(QCARBehaviour));
if (qcarBehaviour)
{
qcarBehaviour.RegisterTrackablesUpdatedCallback(OnTrackablesUpdated);
}
}
public void OnTrackablesUpdated()
{
if (!m_RegisteredFormat)
{
CameraDevice.Instance.SetFrameFormat(m_PixelFormat, true);
m_RegisteredFormat = true;
}
if (m_LogInfo)
{
CameraDevice cam = CameraDevice.Instance;
Image image = cam.GetCameraImage(m_PixelFormat);
if (image == null)
{
Debug.Log(m_PixelFormat + " image is not available yet");
}
else
{
string s = m_PixelFormat + " image: \n";
s += " size: " + image.Width + "x" + image.Height + "\n";
s += " bufferSize: " + image.BufferWidth + "x" + image.BufferHeight + "\n";
s += " stride: " + image.Stride;
Debug.Log(s);
m_LogInfo = false;
}
}
}
If you're planning to use the Vuforia SDK instead then you should Use the image class so works much like the native version.
Register for the desired image format using the CameraDevice.SetFrameFormat method: CameraDevice.Instance.SetFrameFormat(Image.PIXEL_FORMAT.RGB888, true);
Call this method after QCARBehaviour has a chance to run its Start method.
Use the Unity script-ordering feature, or do this once in an Update callback.
Retrieve the image using the CameraDevice.GetCameraImage method.
Take this action from the ITrackerEventHandler.OnTrackablesUpdated callback. That way you can ensure that you retrieve the latest camera image that matches the current frame.
Always make sure that the camera image is not null, since it can take a few frames for the image to become available after registering for an image format.
Thumbs up, if this was helpful.
Take a look at ARtoolkit (open source), which can recognize and track your own images (So only 2d surfaces, not 3d objects) using Natural Feature Tracking.
Another alternative is OpenCV, which is a huge open source computer vision library.

SetMatrix has no effect (WebCamTexture Unity)

I'm using WebCamTexture to get input from the camera (iOS&Android). However, since this is raw input, the rotation is wrong when rendered to a texture. I read around a lot, and found this (look at the bottom): WebCamTexture rotated and flipped on iPhone
His code (but with test-values):
Quaternion rotation = Quaternion.Euler(45f, 30f, 90f);
Matrix4x4 rotationMatrix = Matrix4x4.TRS(Vector3.zero, rotation, new Vector3(1, 1, 1));
material.SetMatrix("_Rotation", rotationMatrix);
But whatever value I use, nothing happens (neither in the editor or on devices)...
Thanks!
Edit
After some intense testing, I found that material.SetMatrix, SetFloat, SetWhatever has NO effect (not setting the value) unless it's declared inside the "Properties"-block. Looking at unity:s own example, this should't have to (and can't) be done for a matrix (can't be declared inside Properties, only inside the CGProgram). So... How do you set a matrix then? Or what else am I doing wrong?
You should be using: WebCamTexture.videoRotationAngle
its designed to solve exactly this problem, read more about this here.
Example code:
using UnityEngine;
using System.Collections;
public class ExampleClass : MonoBehaviour {
public WebCamTexture webcamTexture;
public Quaternion baseRotation;
void Start() {
webcamTexture = new WebCamTexture();
renderer.material.mainTexture = webcamTexture;
baseRotation = transform.rotation;
webcamTexture.Play();
}
void Update() {
transform.rotation = baseRotation * Quaternion.AngleAxis(webcamTexture.videoRotationAngle, Vector3.up);
}
}
Just rotate the camera to 90 degrees along the z axis (the camera is which is rendering the webcamtexture gameobject).

Resources