I'm working on a game that heavily depends on audio API of Corona SDK. The game neads to load a couple of sounds (wavs) and a relatively big background music (+15mb) I don't want to steam the files couse of a statement that I read in corona guides.
Note that streamed files may have a slightly higher latency cost and CPU cost than files loaded with audio.loadSound().
I'm using composer API and I'm planning to develop a loading screen based on event cycle.
My question is can I depend on audio API that if I put all of my loadSounds to the create event handler and they all going to load synchronously and then the show event will be dispatched after all of audio files loaded?..
...Or should I use a diffrent approach for my loading screen?
Yes I believe they are loaded synchronously.
On my apps with a larger background music file the load screen will take longer on slower devices. So it blocks until it is loaded.
I would imagine this is the case since there is no mechanism to 'query' or a callback for when audio is loaded if it was asynchronous.
But don't take my word for it, test it.
PS: Your solution does seem to be solid though, that's about what I do.
Related
I am working on creating an audio unit v3. I have made decent progress up to this point, I made a host app that loads all my 3rd party plugins. I created an auv3 plugin that processes audio and can be loaded by other hosts.
I now want to make an au that loads audio data from disk and scans the data at random positions with sample precision (timestretch, granular stuff, etc). I thought it would be a cool sample playback addittion for contributing to AudioKit.
So this would be making something at the level of AKSampler in the AudioKit framework. While looking through AK source, I feel like I am missing something.
While browsing github, I ended up at places like here:
https://github.com/AudioKit/AudioKit/tree/118affee65f4c9b8d4780ef1a03a6d03004bbcee/AudioKit/Common/Nodes/Playback/Samplers
then I looked in here:
https://github.com/AudioKit/AudioKit/blob/118affee65f4c9b8d4780ef1a03a6d03004bbcee/AudioKit/Common/Nodes/Playback/Samplers/Disk%20Streamer/AKDiskStreamerAudioUnit.mm
which brought me here:
https://github.com/AudioKit/AudioKit/blob/d69dabf090a5e78d4495d938bf6c0aea9f672630/AudioKit/Common/Nodes/Playback/Samplers/Disk%20Streamer/AKDiskStreamerDSPKernel.hpp
and then eventually here:
https://github.com/AudioKit/AudioKit/blob/d69dabf090a5e78d4495d938bf6c0aea9f672630/AudioKit/Core/Soundpipe/modules/wavin.c
I am not looking for info about AKSampler, specifically, just how in general audio files are being loaded and how it makes sense with the realtime nature of the au extension process..
I couldn't find any IPC/XPC code anywhere, so I am guessing that its not about circular buffers connecting to other processes or something.
Does AudioKit allocate memory in the realtime process for audiofile playback? This would seem to go against all the warnings from experienced audio programmers(articles like http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing), but I can't figure out what is being done in AudioKit and generally in iOS..
what am I just not understanding or finding? :D
Opening files and allocating memory for file reads should be done outside the real-time audio context, perhaps during UI file selection, never inside an Audio Unit callback.
One way to get random access to samples inside an AU callback is to memory map the file (mmap C API), and then touch every sample in the memory map before passing the memory pointer (unsafe raw etc.) and file length (mapped bounds) to the audio unit. Then you can do virtual random access file reads inside the callback with a fixed latency.
One way to touch every sample in an array by doing a checksum (and perhaps discarding the result later). This memory read is required to get the iOS virtual memory system to swap blocks from the file VM into RAM, so that storage system reads won't happen in the real-time context.
I need a multi-window app to share media streams. Is there anyway to do that? In nw.js I can create a proof of concept, where a MediaStream created in one window can be played in the other, but it appears I cannot do this in Electron. Am I correct?
I know for certain that it's possible with WebRTC to stream audio/video from a MediaStream to another window process. Been there, done that, based on the electron-peer-connection library (it makes the process quite easy, actually).
Unfortunately, there are a lot of limitations to consider if you take this approach (WebRTC will compress your audio with lossy compression, you'll have a big latency, an Electron bug currently causes the audio to become mono, things like that).
So this is fine for things like voice, but not for e.g. high-end native-quality audio processing.
Additionally, if your app is not a monster beast with insane performance requirements, you can also use Web Audio API and a ScriptProcessorNode (AudioWorklet is still not available in Electron) to access audio sample data from the MediaStream directly, and send that over with standard electron-IPC.
You can then rebuild the MediaStream in the other window process using Web Audio API and MediaStreamDestinationNode.
You should be able to communicate between windows using the ipc module by emitting events through main process and add listeners for them in the windows.
I've got an app that currently ships with all the videos it can play embedded in it. This doesn't scale well, and unless you want to play all the movies, wastes disk space. It also makes it less desirable to upgrade the app because you have to re-download all movies.
What I would like to do is download the movie on the fly, play it back while downloading, and then if it's successfully downloaded, save it to the file system so that next time they want to watch it, it streams from the local file.
I can do whatever is needed to the video, but currently I'm serving it up as an .mp4 file from Amazon S3, with a mimetype of video/mp4, and so the first half of my issue works fine: the movie downloads, and MPMovieViewController will start playing it as soon as it thinks it has downloaded "enough."
Is there any way to tap into the cache of that video file so that I can save it and control how long it resides on the filesystem? This seems like it would be the easiest approach.
I am targeting iOS 5+6, but if the only solution available required iOS 6, I would consider it also. Thanks!
UPDATE: Using AFNetworking, I am now half-way there, I think. I am downloading the video file from the server, and listening for the download progress. Once I see 25% of the video has been downloaded, I start playback on the local file using an MPMoviePlayerController.
The main issue I'm running into now is playback seems to get screwed up. It's going along fine, 25% downloaded, playback starts... download continues normally... then the file finishes downloading completely, and shortly thereafter video freezes. The onscreen playback timer still indicates playback is ongoing and I don't see any "playback finished" type notifications, but the video is frozen. My guess based on the behavior is that perhaps the initial buffer for the video playback was used up, and it isn't detecting that more video is available on disk now?
Is there any way to interact with MPMoviePlayerController to let it know periodically to refresh the buffer it's playing out of? Or some other way to handle this situation?
UPDATE: Make sure to see the newer answer from #TomHamming.
I have yet to find a conclusive answer, but at this time I believe the answer is: you can't reliably do this. At least not without a lot of work which seems too much like a hack. I filed a feature request with Apple as it really seems like this should be possible with some adjustments to MPMoviePlayerController.
I will go over the variety of things I tried or considered, and the results I encountered.
Pass MPMoviePlayerController a URL to your movie file, which allows it to stream, and then pull the file out of the cache it was saved into, into your local Documents folder. Won't work, as of iOS 6. I filed a feature request with Apple, but as it stands now there's no way to get your hands on the file they are downloading, AFAIK.
Start downloading the movie file with NSURLConnection (or something like AFNetwork), and then when a "decent amount" has been downloaded to the device, pass the file URL to the MPMoviePlayerController and let it stream from disk. Sort of works, but not well. Three problems:
It's really hard to know when to start playing the file. I haven't figured out the algorithm Apple uses, and so I always erred on the side of caution, waiting for 25% to be downloaded before playing.
The MPMoviePlayerController interface provides no sense of the movie being streamed, as it does when Apple is doing the calculations via the network. It appears to the user that the file is totally downloaded when it really is not.
And most importantly, MPMoviePlayerController seems to not work well with playing a file that is not completely downloaded. I experienced playback problems once the file finished downloading, or if the player caught up with the amount downloaded, and never found a graceful way to handle these situations.
Same procedure as above, but use AVFoundation classes to more finely control the playback process, and avoid the issues described above regarding playback stopping, etc. Might work, but I want all the features of MPMoviePlayerController. Re-implementing MPMoviePlayerController myself just to get this one feature seems like a waste of time.
Same procedure as #1 above, but run a small web server in your app to handle streaming the video from the disk to MPMoviePlayerController, with the hope being that the streaming would work more like it normally does when streaming the file directly from an external web server. Works, but results were still sporadic and performance seemed to suffer. I did my test with CocoaHTTP. I decided against this approach because it just felt like a terrible hack.
Run a lightweight HTTP proxy, thus intercepting the downloaded movie file data as it gets streamed from the internet into your MPMoviePlayerController. Not sure if this works or not. I was not able to test this yet, as I have not found a lightweight HTTP proxy written in Objective-C, and at this point don't feel like implementing one just to try this experiment. It seems like the next easiest of all these hacks to implement -- if you don't have to write the proxy!
At this point I've decided to go the less-hacky, but also less user-friendly route of simply downloading the file completely, and then passing it to MPMoviePlayerController, until a better solution comes along.
You can do this as of iOS 10 with AVAssetDownloadTask. See this WWDC 2016 session and this documentation.
Alternatively, if your movie isn't DRM'd, you can do it with AVAssetResourceLoaderDelegate, which effectively lets you give an AVPlayer an arbitrary stream of bytes. See this walkthrough.
I am building a game that lets users remix songs. I have built a mixer (based upon the apple sample code MixerHost (creating an audioGraph with a mixer audioUnit), but expanded to load 12 tracks. everything is working fine, however it takes a really long time for the songs to load when the gamer selects the songs they want to remix. This is because the program has to load 12 separate mp4 files into memory before I can start playing the music.
I think what I need is to create a AUFilePlayer audioUnit that is in charge of loading the file into the mixer. If the AUFilePlayer can handle loading the file on the fly then the user will not have to wait for the files to load 100% into memory. My two questions are, 1. can an AUFilePlayer be used this way? 2. The documentation on AUFilePlayer is very very very thin. Where can I find some example code demonstrated how to implement a AUFilePlayer properly in IOS (not in MacOS)?
Thanks
I think you're right - in this case a 'direct-from-disk' buffering approach is probably what you need. I believe the correct AudioUnit subtype is AudioFilePlayer. From the documentation:
The unit reads and converts
audio file data into its own internal
buffers. It performs disk I/O on a
high-priority thread shared among all
instances of this unit within a process.
Upon completion of a disk read, the unit
internally schedules buffers for playback.
A working example of using this unit on Mac OS X is given in Chris Adamson's book Learning Core Audio. The code for iOS isn't much different, and is discussed in this thread on the CoreAudio-API mailing list. Adamson's working code example can be found here. You should be able to adapt this to your requirements.
I’m working on a small iPhone app which is streaming movie content over a network connection using regular sockets. The video is in H.264 format. I’m however having difficulties with playing/decoding the data. I’ve been considering using FFMPEG, but the license makes it unsuitable for the project. I’ve been looking into Apple’s AVFoundation framework (AVPlayer in particular), which seems to be able to handle h264 content, however I’m only able to find methods to initiate the movie using an url – not by proving a memory buffer streamed from the network.
I’ve been doing some tests to make this happen anyway, using the following approaches:
Play the movie using a regular AVPlayer. Every time data is received on the network, it’s written to a file using fopen with append-mode. The AVPlayer’s asset is then reloaded/recreated with the updated data. There seems to be two issues with this approach: firstly, the screen goes black for a short moment while the first asset is unloaded and the new loaded. Secondly, I do not know exactly where the playing stopped, so I’m unsure how I would find out the right place to start playing the new asset from.
The second approach is to write the data to the file as in the first approach, but with the difference that the data is loaded into a second asset. A AVQueuedPlayer is then used where the second asset is inserted/queued in the player and then called when the buffering has been done. The first asset can then be unloaded without a black screen. However, using this approach it’s even more troublesome (than the first approach) to find out where to start playing the new asset.
Has anyone done something like this and made it work? Is there a proper way of doing this using AVFoundation?
The official method to do this is the HTTP Live Streaming format which supports multiple quality levels (among other things) and automatically switches between them (eg: if the user moves from WiFi to cellular).
You can find the docs here: Apple Http Streaming Docs