Set an initial focal distance on iOS - ios

I'm working on an iOS-app where one of the features is scanning QR-codes. For this I'm using the excellent library, ZBar. The scanning works fine and is generally really quick. However when you use smaller QR-codes it takes a bit longer to scan, mostly due to the fact that the autofocus needs some time to adjust. I was experimenting and noticed that the focus could be locked using the following code:
AVCaptureDevice *cameraDevice = readerView.device;
if ([cameraDevice lockForConfiguration:nil]) {
[cameraDevice setFocusMode:AVCaptureFocusModeLocked];
[cameraDevice unlockForConfiguration];
}
When this code is used after a successful scan, the coming scans are really quick. That made me wonder, could I somehow lock the focus before even scanning one code? The app will only scan rather small QR-codes so there will never be a need for focusing on something far away. Sure, I could implement something like tap to focus, but preferably I would like to avoid that extra step.
Is there a way to achieve this? Or are there maybe another way of speeding things up when dealing with smaller QR-codes?
// Alexander

In iOS7 this is now possible!
Apple has added the property autoFocusRangeRestriction to the AVCaptureDevice class. This property is of the enum AVCaptureAutoFocusRangeRestriction which has three different values:
AVCaptureAutoFocusRangeRestrictionNone - Default, no restrictions
AVCaptureAutoFocusRangeRestrictionNear - The subject that matters is close to the camera
AVCaptureAutoFocusRangeRestrictionFar - The subject that matters is far from the camera
To check if the method is available we should first check if the property autoFocusRangeRestrictionSupported is true. And since it's only supported in iOS7 an onwards we should also use respondsToSelector so we don't get an exception on earlier iOS-versions.
So the resulting code should look something like this:
AVCaptureDevice *cameraDevice = zbarReaderView.device;
if ([cameraDevice respondsToSelector:#selector(isAutoFocusRangeRestrictionSupported)] && cameraDevice.autoFocusRangeRestrictionSupported) {
// If we are on an iOS version that supports AutoFocusRangeRestriction and the device supports it
// Set the focus range to "near"
if ([cameraDevice lockForConfiguration:nil]) {
cameraDevice.autoFocusRangeRestriction = AVCaptureAutoFocusRangeRestrictionNear;
[cameraDevice unlockForConfiguration];
}
}
This seems to somewhat speed up the scanning of small QR-codes according to my initial tests :)
Update - iOS8
With iOS8, Apple has given us lots of new camera API's to play with. One of this new methods is this one:
- (void)setFocusModeLockedWithLensPosition:(float)lensPosition completionHandler:(void (^)(CMTime syncTime))handler
This method locks focus by moving the lens to a position between 0.0 and 1.0. I played around with the method, locking the lens at close values. However, in general it caused more problems then it solved. You had to keep the QR-codes/barcodes at a very specific distance, which could cause issues when you had codes of different sizes.
But. I think I have found a pretty good alternative to locking focus altogether. When the user press the scan button, I lock the lens to a close distance, and when it's finished I switch the camera back to auto focus. This gives us the benefits of keeping auto focus on, but forces the camera to begin at a close distance where a QR-code/barcode is likely to be found. This in combination with:
cameraDevice.autoFocusRangeRestriction = AVCaptureAutoFocusRangeRestrictionNear;
And:
cameraDevice.focusPointOfInterest = CGPointMake(0.5,0.5);
Results in a pretty snappy scanner.
I also built a custom scanner with the API's introduced in iOS7, instead of using ZBar. Mostly because the ZBar-libs are quite outdated and as when iPhone 5 introduced ARMv7s I now had to recompile it again for ARM64.
// Alexander

iOS 8 recently added this configuration! It is almost like they read stack overflow
/*!
#method setFocusModeLockedWithLensPosition:completionHandler:
#abstract
Sets focusMode to AVCaptureFocusModeLocked and locks lensPosition at an explicit value.
#param lensPosition
The lens position, as described in the documentation for the lensPosition property. A value of AVCaptureLensPositionCurrent can be used
to indicate that the caller does not wish to specify a value for lensPosition.
#param handler
A block to be called when lensPosition has been set to the value specified and focusMode is set to AVCaptureFocusModeLocked. If
setFocusModeLockedWithLensPosition:completionHandler: is called multiple times, the completion handlers will be called in FIFO order.
The block receives a timestamp which matches that of the first buffer to which all settings have been applied. Note that the timestamp
is synchronized to the device clock, and thus must be converted to the master clock prior to comparison with the timestamps of buffers
delivered via an AVCaptureVideoDataOutput. The client may pass nil for the handler parameter if knowledge of the operation's completion
is not required.
#discussion
This is the only way of setting lensPosition.
This method throws an NSRangeException if lensPosition is set to an unsupported level.
This method throws an NSGenericException if called without first obtaining exclusive access to the receiver using lockForConfiguration:.
*/
- (void)setFocusModeLockedWithLensPosition:(float)lensPosition completionHandler:(void (^)(CMTime syncTime))handler NS_AVAILABLE_IOS(8_0);
EDIT: this is a method of AVCaptureDevice

Related

iOS - AVAudioPlayerNode.play() execution is very slow

I'm using AVAudioEngine for audio in an iOS game application. A problem I've encountered is that AVAudioPlayerNode.play() takes a long time to execute, which can be a problem in real-time applications such as games.
play() just activates the player node - you don't have to call it every time you play a sound. As such, it doesn't have to be called that often, but it does have to be called occasionally, such as to activate the player initially, or after it's been deactivated (which happens in some situations). Even if only called occasionally, the long execution times can be a problem, especially if you need to call play() on multiple players at once.
The execution time for play() seems to be proportional to the value of AVAudioSession.ioBufferDuration, which you can request to be changed using AVAudioSession.setPreferredIOBufferDuration(). Here's some code I'm using to test this:
import AVFoundation
import UIKit
class ViewController: UIViewController {
private let engine = AVAudioEngine()
private let player = AVAudioPlayerNode()
private let ioBufferSize = 1024.0 // Or 256.0
override func viewDidLoad() {
super.viewDidLoad()
let audioSession = AVAudioSession.sharedInstance()
try! audioSession.setPreferredIOBufferDuration(ioBufferSize / 44100.0)
try! audioSession.setActive(true)
engine.attach(player)
engine.connect(player, to: engine.mainMixerNode, format: nil)
try! engine.start()
print("IO buffer duration: \(audioSession.ioBufferDuration)")
}
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
if player.isPlaying {
player.stop()
} else {
let startTime = CACurrentMediaTime()
player.play()
let endTime = CACurrentMediaTime()
print("\(endTime - startTime)")
}
}
}
Here are some sample timings for play() that I got using a buffer size of 1024 (which I believe is the default):
0.0218
0.0147
0.0211
0.0160
0.0184
0.0194
0.0129
0.0160
Here are some sample timings using a buffer size of 256:
0.0014
0.0029
0.0033
0.0023
0.0030
0.0039
0.0031
0.0032
As you can see above, for a buffer size of 1024, execution times tend to be in the 15-20 ms range (around a full frame at 60 FPS). With a buffer size of 256, it's around 3 ms - not as bad, but still costly when you only have ~17 ms per frame to work with.
This is on an iPad Mini 2 running iOS 12.4.2. This is obviously an old device, but the results I see on the simulator seem similarly proportional, so it seems to have more to do with the buffer size and the behavior of the function itself than with the hardware being used. I don't know what's going on under the hood, but it seems possible that play() blocks until the beginning of the next audio cycle, or something like that.
Requesting a lower buffer size seems like a partial solution, but there are some potential drawbacks. According to the documentation here, lower buffer sizes can mean more disk access when streaming from a file, and irrespective of that, the request may not be honored at all. Also, here, someone reports playback problems related to low buffer sizes. Taking all this into account, I'm disinclined to pursue this as a solution.
That leaves me with execution times for play() in the 15-20 ms range, which typically means a missed frame at 60 FPS. If I arrange things so that only one call to play() is made at a time, and only infrequently, maybe it won't be noticeable, but it's not ideal.
I've searched for information and asked about this in other places, but it seems either not many people are encountering this behavior in practice, or it isn't an issue for them.
AVAudioEngine is intended for use in real-time applications, so if I'm right that AVAudioPlayerNode.play() blocks for a significant amount of time proportional to the buffer size, that seems like a design issue. I realize this probably isn't an issue many are dealing with, but I'm posting here to ask if anyone has encountered this specific issue with AVAudioEngine, and if so, if there's any insight, suggestions, or workarounds anyone can offer.
I've investigated this fairly thoroughly. Here are my findings.
Having now tested the behavior on a variety of devices and iOS versions (including the latest version at the time of this writing, 13.2), and having had others test it as well, my current conclusion is that the long execution times for AVAudioPlayerNode.play() are inherent and that there's no obvious workaround. As noted in my original post, the execution times can be reduced by requesting a lower buffer duration, but as discussed earlier, this doesn't seem like a viable solution.
I heard from a credible source that calling play() on a background thread (e.g. using Grand Central Dispatch) should be safe, and indeed this would be one way to solve the problem. However, although it may technically be safe to call play() (or other AVAudioEngine-related functions) on different threads, I'm skeptical as to whether this is a good idea (further explanation below).
The documentation doesn't state this as far as I can tell, but AVAudioEngine will throw NSException's under various circumstances, which, without special handling, will result in application termination in Swift.
One of the things that will cause an NSException to be thrown is if you call AVAudioPlayerNode.play() while the engine is not running. Obviously if you only have your own code to worry about, you can take steps to ensure this doesn't occur.
However, iOS itself will sometimes stop the engine of its own accord, for example when an audio interruption occurs. If you call play() subsequent to that and before restarting the engine, an NSException will be thrown. It's fairly easy to avoid this mistake if all your calls to play() are on the main thread, but multithreading complicates the issue and seems like it could introduce the risk of accidentally calling play() after the engine has been stopped. Although there may be ways to work around this, multithreading seems to introduce undesirable complexity and fragility, so I've opted not to pursue it.
My current strategy is as follows. For the reasons discussed earlier, I'm not using multithreading. Instead, I'm doing everything I can to reduce the number of calls to play(), both overall and per-frame. This includes, among other things, only supporting stereo audio (for various reasons, supporting both mono and stereo can lead to more calls to play(), which is undesirable).
Lastly, I also investigated alternatives to AVAudioEngine. OpenAL is still supported on iOS, but is deprecated. A custom implementation using low-level APIs such as Audio Queue Services or Audio Units would be a possibility, but would be non-trivial. I've also looked at some open-source solutions, but the options I looked at use AVAudioEngine under the hood themselves and therefore suffer from the same problems, and/or have other shortcomings or limitations of their own. Of course there are also commercial options available, which may provide a solution for some developers.

What is ID3D12GraphicsCommandList::DiscardResource?

What exactly should I expect to happen when using DiscardResource?
What's the difference between discard and destroying/deleting a resource.
When is a good time/use-case to discard a resource?
Unfortunately Microsoft doesn't seem to say much about it other than it "discards a resource".
TL;DR: Is a rarely used function that provides a driver hint related to handling clear compression structures. You are unlikely to use it except based on specific performance advice.
DiscardResource is the DirectX 12 version of the Direct3D 11.1 method. See Microsoft Docs
The primary use of these methods it to optimize the performance of tiled-based deferred rasterizer graphics parts by discarding the render target after present. This is a hint to the driver that the contents of the render target are no longer relevant to the operation of the program, so it can avoid some internal clearing operations on the next use.
For DirectX 11, this is in the DirectX 11 App template to use DiscardView because it makes use of DXGI_SWAP_EFFECT_FLIP_SEQUENTIAL:
void DX::DeviceResources::Present()
{
// The first argument instructs DXGI to block until VSync, putting the application
// to sleep until the next VSync. This ensures we don't waste any cycles rendering
// frames that will never be displayed to the screen.
DXGI_PRESENT_PARAMETERS parameters = { 0 };
HRESULT hr = m_swapChain->Present1(1, 0, &parameters);
// Discard the contents of the render target.
// This is a valid operation only when the existing contents will be entirely
// overwritten. If dirty or scroll rects are used, this call should be removed.
m_d3dContext->DiscardView1(m_d3dRenderTargetView.Get(), nullptr, 0);
// Discard the contents of the depth stencil.
m_d3dContext->DiscardView1(m_d3dDepthStencilView.Get(), nullptr, 0);
// If the device was removed either by a disconnection or a driver upgrade, we
// must recreate all device resources.
if (hr == DXGI_ERROR_DEVICE_REMOVED || hr == DXGI_ERROR_DEVICE_RESET)
{
HandleDeviceLost();
}
else
{
DX::ThrowIfFailed(hr);
}
}
The DirectX 12 App template doesn't need those explicit calls because it uses DXGI_SWAP_EFFECT_FLIP_DISCARD.
If you are wondering why the DirectX 11 app doesn't just use DXGI_SWAP_EFFECT_FLIP_DISCARD, it probably should. The DXGI_SWAP_EFFECT_FLIP_SEQUENTIAL swap effect was the only one supported by Windows 8.x for Windows Store apps, which is when DiscardView was introduced. For Windows 10 / DirectX 12 / UWP, it's probably better to always use DXGI_SWAP_EFFECT_FLIP_DISCARD unless you specifically don't want the backbuffer discarded.
It is also useful for multi-GPU SLI / Crossfire configurations since the clearing operation can require synchronization between the GPUs. See this GDC 2015 talk
There are also other scenario-specific usages. For example, if doing deferred rendering for the G-buffer where you know every single pixel will be overwritten, you can use DiscardResource instead of doing ClearRenderTargetView / ClearDepthStencilView.

ios - presentRenderbuffer triggers EXC_BAD_ACCESS

I've found that when I use textures above GL_TEXTURE18 on ios (tested on iOS 10), presentRenderbuffer triggers an EXC_BAD_ACCESS. Is there any reason for that? Can I not use textures up to GL_TEXTURE31
The GL_TEXTUREX are just some defined values, defined enumerations. In your case a GPU is the one that defines the actual number of supported textures and it is your responsibility to check what are these limitations.
You can get that by using glGet something like:
GLint max_combined_texture_image_units;
glGetIntegerv(GL_MAX_COMBINED_TEXTURE_IMAGE_UNITS, &max_combined_texture_image_units);
Try this thread.
Do note that these defines/enumerations are here just to help you, it does not mean they are actually valid or supported. The openGL API is mostly designed by passing integer values typedef uint32_t GLenum; so as far as the API goes you may replace GL_TEXTURE0 with 1200 or any other value but you do need to ensure that the value is actually valid.

How to get GCKViewVideoFrameInput working in swift?

Trying to mirror views to chromecast with the remote display API. On Android it is well documented and easy to implement. The iOS samples/docs are less complete. I understand it only supports 15 fps but that is fine for my needs.
If anyone has gotten it to work, I'd love to see a small swift sample that shows how to mirror a simple view. I'm trying to test it with this, which shows nothing on the TV and gives the device has disconnected after a few seconds. From reading the docs, that happens when you don't send anything within the first 15 secs of getting the session.
var testSession: GCKRemoteDisplaySession!
func remoteDisplayChannel(channel: GCKRemoteDisplayChannel,
didBeginSession session: GCKRemoteDisplaySession) {
// Use the session.
testSession = session
frameInput = GCKViewVideoFrameInput(session: testSession)
// any view
frameInput.view = testView
}
Make sure you are strongly referencing the session as well as the frame input. Inputs have weak references to sessions (to avoid cycles between sessions and inputs). If the session is not strongly referenced and gets destroyed, you'll see black on your remote screen followed by a timeout disconnect.

Using ReactiveCocoa to track UI updates with a remote object

I'm making an iOS app which lets you remotely control music in an app playing on your desktop.
One of the hardest problems is being able to update the position of the "tracker" (which shows the time position and duration of the currently playing song) correctly. There are several sources of input here:
At launch, the remote sends a network request to get the initial position and duration of the currently playing song.
When the user adjusts the position of the tracker using the remote, it sends a network request to the music app to change the position of the song.
If the user uses the app on the desktop to change the position of the tracker, the app sends a network request to the remote with the new position of the tracker.
If the song is currently playing, the position of the tracker is updated every 0.5 seconds or so.
At the moment, the tracker is a UISlider which is backed by a "Player" model. Whenever the user changes the position on the slider, it updates the model and sends a network request, like so:
In NowPlayingViewController.m
[[slider rac_signalForControlEvents:UIControlEventTouchUpInside] subscribeNext:^(UISlider *x) {
[playerModel seekToPosition:x.value];
}];
[RACObserve(playerModel, position) subscribeNext:^(id x) {
slider.value = player.position;
}];
In PlayerModel.m:
#property (nonatomic) NSTimeInterval position;
- (void)seekToPosition:(NSTimeInterval)position
{
self.position = position;
[self.client newRequestWithMethod:#"seekTo" params:#[positionArg] callback:NULL];
}
- (void)receivedPlayerUpdate:(NSDictionary *)json
{
self.position = [json objectForKey:#"position"]
}
The problem is when a user "fiddles" with the slider, and queues up a number of network requests which all come back at different times. The user could be have moved the slider again when a response is received, moving the slider back to a previous value.
My question: How do I use ReactiveCocoa correctly in this example, ensuring that updates from the network are dealt with, but only if the user hasn't moved the slider since?
In your GitHub thread about this you say that you want to consider the remote's updates as canonical. That's good, because (as Josh Abernathy suggested there), RAC or not, you need to pick one of the two sources to take priority (or you need timestamps, but then you need a reference clock...).
Given your code and disregarding RAC, the solution is just setting a flag in seekToPosition: and unsetting it using a timer. Check the flag in recievedPlayerUpdate:, ignoring the update if it's set.
By the way, you should use the RAC() macro to bind your slider's value, rather than the subscribeNext: that you've got:
RAC(slider, value) = RACObserve(playerModel, position);
You can definitely construct a signal chain to do what you want, though. You've got four signals you need to combine.
For the last item, the periodic update, you can use interval:onScheduler::
[[RACSignal interval:kPositionFetchSeconds
onScheduler:[RACScheduler scheduler]] map:^(id _){
return /* Request position over network */;
}];
The map: just ignores the date that the interval:... signal produces, and fetches the position. Since your requests and messages from the desktop have equal priority, merge: those together:
[RACSignal merge:#[desktopPositionSignal, timedRequestSignal]];
You decided that you don't want either of those signals going through if the user has touched the slider, though. This can be accomplished in one of two ways. Using the flag I suggested, you could filter: that merged signal:
[mergedSignal filter:^BOOL (id _){ return userFiddlingWithSlider; }];
Better than that -- avoiding extra state -- would be to build an operation out of a combination of throttle: and sample: that passes a value from a signal at a certain interval after another signal has not sent anything:
[mergedSignal sample:
[sliderSignal throttle:kUserFiddlingWithSliderInterval]];
(And you might, of course, want to throttle/sample the interval:onScheduler: signal in the same way -- before the merge -- in order to avoid unncessary network requests.)
You can put this all together in PlayerModel, binding it to position. You'll just need to give the PlayerModel the slider's rac_signalForControlEvents:, and then merge in the slider value. Since you're using the same signal multiple places in one chain, I believe that you want to "multicast" it.
Finally, use startWith: to get your first item above, the inital position from the desktop app, into the stream.
RAC(self, position) =
[[RACSignal merge:#[sampledSignal,
[sliderSignal map:^id(UISlider * slider){
return [slider value];
}]]
] startWith:/* Request position over network */];
The decision to break each signal out into its own variable or string them all together Lisp-style I'll leave to you.
Incidentally, I've found it helpful to actually draw out the signal chains when working on problems like this. I made a quick diagram for your scenario. It helps with thinking of the signals as entities in their own right, as opposed to worrying about the values that they carry.

Resources