I use AVPlayer to play audio(streaming or local file). For this audio I want to apply some effects - boost volume, skip silence, reduce noise, change speed(in 0.1 intervals).
I did same thing in android by creating own player, decoding different audio formats into pcm data and then using some c libraries to modify it. It was quite complicated.
Is it possible to do with AVPlayer or how can I do that? Something like modifying audio already decoded by AVPlayer. Is there some ios api (AVAudioEngine?) or frameworks (audioKit?) which can do this?
Thanks!
IMHO the best solution is to use https://github.com/audiokit/AudioKit as it is well maintained and supports most of the requirements you listed.
Another approach is to import the C library you used in the Android project and have a wrapper around it so it can easily used in ObjectiveC/Swift. With this approach you will have less code to maintain and you guarantee similar results between the two platforms. Do you care to share more about this code ?
Related
Working in audio kit and I am looking to understand how people have incorporated drums. Obviously, the sampler is an option, but I am wondering if there is a built in option similar to some of the basic synthesis options.
There are a few options. I personally like the AppleSampler/MidiSampler like in the example but instead of using audio files you can create a EXS Sampler instrument in Logic where you can assign notes for different velocities. AppleSampler can also load AUPresets made in GarageBand and SoundFonts (SF2). The DunneAudioKit Sampler is an option if you are working with SFZ files, but I think that might be a work-in-progress in AudioKit 5. Loading WAV files directly into AppleSampler is also a good option if you just want one shot sounds.
I'm assuming you're mostly talking about playback of samples, not recording.
The best built-in option I've seen (other than AppleSampler/MidiSampler) is AudioPlayer, which lets you load in a sample and play it back on demand (from an on-screen pad, etc). MIDIListener can then help you respond to external MIDI events, etc. It works (I have a pretty big branch in my app where I tried it), but not sure it works well.
I wouldn't recommend DunneAudioKit Sampler for drums. There is no one-shot playback (so playing the same note in quick succession will cut off the previous note, even if you mess with the release). If you're trying to build a complex/realistic acoustic drum instrument, you'll also want round-robins so that variations of the same hit can be played, which Dunne also doesn't have. It can load SFZ files, but only a very limited subset of SFZ's opcodes (so again, it's missing things like round robins, mute groups, one-shot, etc).
Having gone down all those roads, I would suggest starting with AppleSampler, and I would build the EXS or aupreset file in Logic or Mainstage rather than trying to build something programmatically.
If your needs are really simple, the examples in AudioKit's recently released drum pad playground is a great place to start, loading single samples into a specific note on AppleSampler.
I'm trying to put together an open source library that allows iOS devices to play files with unsupported containers, as long as the track formats/codecs are supported. e.g.: a Matroska video (MKV) file with an H264 video track and an AAC audio track. I'm making an app that surely could use that functionality and I bet there are many more out there that would benefit from it. Any help you can give (by commenting here or—even better— collaborating with me) is much appreciated. This is where I'm at so far:
I did a bit of research trying to find out how players like AVPlayerHD or Infuse can play non-standard containers and still have hardware acceleration. It seems like they transcode small chunks of the whole video file and play those in sequence instead.
It's a good solution. But if you want to throw that video to an Apple TV, things don't work as planned since the video is actually a bunch of smaller chunks being played as a playlist. This site has way more info, but at its core streaming to Apple TV is essentially a progressive download of the MP4/MPV file being played.
I'm thinking a sort of streaming proxy is the way to go. For the playing side of things, I've been investigating AVSampleBufferDisplayLayer (more info here) as a way of playing the video track. I haven't gotten to audio yet. Things get interesting when you think about the AirPlay side of things: by having a "container proxy", we can make any file look like it has the right container without the file size implications of transcoding.
It seems like GStreamer might be a good starting point for the proxy. I need to read up on it; I've never used it before. Does this approach sound like a good one for a library that could be used for App Store apps?
Thanks!
Finally got some extra time to go over GStreamer. Especially this article about how it is already updated to use the hardware decoding provided by iOS 8. So no need to develop this; GStreamer seems to be the answer.
Thanks!
The 'chucked' solution is no longer necessary in iOS 8. You should simply set up a video decode session and pass in NALUs.
https://developer.apple.com/videos/wwdc/2014/#513
I'm trying to build an iOS app that plays video files and does some interesting things using MTAudioProcessingTap. I need it to be able to play all sorts of formats, including some that are not supported by Apple. I'm thinking of branching out from VLC, but I can't figure out if it uses Core Audio/Video at any point or if it's running something else completely.
If it's not, is there a library I can use to take care of the 203572964 codecs being used out there?
Thanks.
Preliminary note: I'm the developer of VLC for iOS so the following may be biased.
MobileVLCKit for iOS includes 2 different audio output modules. One of them is a high level module based on AudioQueue which is fairly incomplex but a bit slow. The other is based on AudioUnit, the low level framework of CoreAudio, quite a bit more complex, but way faster. Depending on your current experience, either module would be a good way to start.
Regarding the one library supporting all codecs thing: basically there are two forks of the same library: libav and FFmpeg. VLC supports either flavor and abstracts the complexity and the ever-changing APIs (which are a real pain if you intend to keep maintaining your app across multiple releases of those libraries). Additionally, we include a quite well performing OpenGL ES 2 video output module which is using shaders to do chroma conversation. All you need to do is embedding a UIView. MobileVLCKit handles the rest.
Speaking of MobileVLCKit: this is a thin ObjC layer on top of libvlc simplifying the use of this library in third party applications by abstracting most commonly used features.
As implicitly mentioned by HalR, libvlc does not use hardware accelerated decoding on iOS yet. We are working with the libav developers on a generic approach, but we are not quite there yet. Thus, we have to do all the decoding on the CPU, which leads to the heating but allows us to play virtually anything instead of H264/MP4 using the default, accelerated API.
If you can't figure out how its playing the video, at its lower level, that perhaps is a sign that you should keep working with it instead of trying to outdo it. Video processing is pretty difficult and often unsupported formats are unsupported due to patent issues. I really haven't seen anything better than VLC that is publicly available.
VLC 2.1.x appears to use AudioToolbox and AVFoundation.
One other issue, though, is that when I was doing work with VLC, I was stunned how it turned my iPod Touch into a miniature iron, because it was working so hard to process the video. Manually processing video is very processor intensive and really is a drain. So your way or VLC could still have some additional issues.
I'm developing a virtual instrument app for iOS and am trying to implement a recording function so that the app can record and playback the music the user makes with the instrument. I'm currently using the CocosDenshion sound engine (with a few of my own hacks involving fades etc) which is based on OpenAL. From my research on the net it seems I have two options:
Keep a record of the user's inputs (ie. which notes were played at what volume) so that the app can recreate the sound (but this cannot be shared/emailed).
Hack my own low-level sound engine using AudioUnits & specifically RemoteIO so that I manually mix all the sounds and populate the final output buffer by hand and hence can save said buffer to a file. This will be able to be shared by email etc.
I have implemented a RemoteIO callback for rendering the output buffer in the hope that it would give me previously played data in the buffer but alas the buffer is always all 00.
So my question is: is there an easier way to sniff/listen to what my app is sending to the speakers than my option 2 above?
Thanks in advance for your help!
I think you should use remoteIO, I had a similar project several months ago and wanted to avoid remoteIO and audio units as much as possible, but in the end, after I wrote tons of code and read lots of documentations from third party libraries (including cocosdenshion) I end up using audio units anyway. More than that, it's not that hard to set up and work with. If you however look for a library to do most of the work for you, you should look for one written a top of core audio not open al.
You might want to take a look at the AudioCopy framework. It does a lot of what you seem to be looking for, and will save you from potentially reinventing some wheels.
I want to record two voices and compare them. I think there is some Apple sample code for voice recording. I have no idea about
comparing two audio files. What is the right approach for this? Is there any framework Apple provides for this purpose or is there any third party framework?
It's not in objective C, but it does contain some fantastic explanation about how audio is compared by Shazam, and includes sample code (and source for a working application) in Java:
Check this out
Additionally, This Question has a fantastic link to audio fingerprinting, which is essentially the same as the article above, but more in depth.
Hope this helps
I'm using Visqol for this purpose. If your audio files are generally not more than 10sek this could be something worth looking into. Also check ffmpeg library for converting the files into the desired format(Visqol will require certain sample rate depending if it is just music or speech).
https://github.com/google/visqol