I'm trying to create an audio visualizer and I'm using the accelerate framework functions to compute the root means square value of the buffer data to get a uniform scalar for my metal shader.
I have an onscreen midi controller where the users can press the buttons to make a sound.
The sounds I'm playing are very short. For example, I have sound buttons that the user presses which makes a beat, this only lasts for about 0.2-0.4s and I only get about 3-4 callbacks during the play.
My visualizer looks quite awful and choppy as it just snaps to 4 different sizes per button press rather than having a smooth transition.
I'm going for a look like this:
Do I install a second tap? Should I try and interpolate the points to "fake" the transitions?
Ideally, I'd like something along the lines of 10-15 frames/second.
Since there is is no way to increase the frequency, I resorted to interpolating the previous value with the current value to smoothen the transition.
Related
I read this from page on Tuning Your OpenGL ES App :
Redraw Scenes Only When the Scene Data Changes :
Your app should wait until something in the scene changes before rendering a new frame. Core Animation caches the last image presented to the user and continues to display it until a new frame is presented.
Even when your data changes, it is not necessary to render frames at the speed the hardware processes commands. A slower but fixed frame rate often appears smoother to the user than a fast but variable frame rate. A fixed frame rate of 30 frames per second is sufficient for most animation and helps reduce power consumption.
From what I understand, there is an event loop which keeps on running and re-rendering the scene. We just override the onDrawFrame method and put our rendering code there. I don't have any control on when this method gets called. How can then I "Redraw Scenes Only When the Scene Data Changes" ?
In my case, there is a change in scene only when user interacts (click, pinch etc.). Ideally I would like to not render when user is not interacting with my scene, but this function is getting called continuously. Am confused.
At the lowest exposed level, there is an OpenGL-containing type of CoreAnimation layer, CAEAGLLayer. That can supply a colour buffer that is usable to construct a framebuffer object, to which you can draw as and when you wish, presenting as and when you wish. So that's the full extent of the process for OpenGL in iOS: draw when you want, present when you want.
The layer then has a pixel copy of the scene. Normal Core Animation rules apply so you're never autonomously asked to redraw. Full composition, transitions, Core Animation animations, etc, can all occur without any work on your part.
It is fairly common practice to connect up a timer, usually a CADisplayLink, either manually or just by taking advantage of one of the GLKit constructs. In that case you're trying to produce one frame per timer tick. Apple is suggesting that running the timer at half the refresh rate is acceptable and, if you wake up to perform a draw and realise that you'd just be drawing exactly the same frame as you did last time, not bothering to draw at all. Ideally stopping the timer completely if you have sufficient foresight.
As per the comment, onDrawFrame isn't worded like an Objective-C or Swift method and isn't provided by any Apple-supplied library. Whomever is posting that — presumably to try to look familiar to Android authors — needs to take responsibility for appropriate behaviour.
I'm going to ask this at the risk of being too vague or asking too many things in one question, but I'm really just looking for a point in the right direction.
In my app I want to record audio, show a waveform while recording, and scroll through the waveform to record and playback from a specified time. For example, if I have 3 minutes of audio, I should be able to scroll back to 2:00 and start recording from there to fix a mistake.
In Voice Memos, this is accomplished instantaneously, without any delay or loading time. I'm trying to figure out how the did this, if anyone has a clue.
What I've tried:
EZAudio - This library is great, but doesn't do what I want. You can't scroll through the waveform. It deletes the waveform data at the beginning and begins appending it to the end once it reaches a certain length.
SCWaveformView - This waveform is nice, but it uses images. Once the waveform is too long, putting it in a scroll view causes really jittery scrolling. Also you can't build the waveform while recording, only afterward.
As far as appending, I've used this method: https://stackoverflow.com/a/11520553/1391672
But there is significant processing time, even when appending two very short clips of audio together (in my experience).
How does Voice Memos do what it does? Do you think the waveform is drawn in OpenGL or CoreGraphics? Are they using Core Audio or AVAudioRecorder? Has anyone built anything like this that can point me in the right direction?
When zoomed-in, a scrollview only needs to draw the small portion of the waveform that is visible. When zoomed-out, a graph view might only drawn every Nth point of the audio buffer, or use some other DSP down-sampling algorithm on the data before rendering. This likely has to be done using your own custom drawing or graphics rendering code inside a UIScrollView or similar custom controller. The waveform rendering code during and after recording don't have to be the same.
The recording API and the drawing API you use can be completely independent, and can be almost anything, from OpenGL to Metal to Core Graphics (on newer faster devices). On the audio end, Core Audio will help provide the lowest latency, but Audio Queues and the AVAudioEngine might also be suitable.
I am looking into making a drawing app with Sprite Kit in iOS, with either swift or objective C.
There is a tutorial here that shows how to draw a line with sprite kit. This is great, but for my app I want more. I want the app to play sound effects while the line is being drawn. The tone of the sound will depend on the speed at which the user is drawing the line. The faster the user moves their finger, the higher pitch the sound is. All i have found regarding playing sounds in sprite kit is background music and playing a single sound. Can someone point me in the right direction to accomplish my goal?
You'll probably want to have a chromatic set of samples, ranging from the lowest to the highest possible tone you want to play, at a very short duration (maybe 0.25 seconds? You'll need to experiment, but they will all have to be the same duration).
Let's say you have 30 different samples, line-0.wav .. line-29.wav. Then all you need to do is calculate the velocity of the user's drag appropriately, and use some function to map the possible range of drag velocity to the integer range of sample indices. As long as the drag is in progress, play the appropriate sound repeatedly.
I have a game where user controls a character (with his finger) and i would like to add functionality so the user can record his moves while playing and then play it back. The problem is that the game includes physics and i guess it would be very hard to replicate the exact same moves. How can i implement such a system that will perfectly replay all the users actions? Do i have to record every touch and then play back all the touches? Does any one have any experience with this? I am using Box2D for physics.
"We record replays by storing keystrokes and frame numbers" - box2d.org/forum/viewtopic.php?f=3&t=1982&view=next Seems as if it's the only way to do it. Write these to a PLIST or something and you'll have your replays. Also, if your physics isn't already deterministic (ie. random) then just take down the random values too)
From the comments:
"Just record all the position and rotation state for all the objects every frame (or every other possibly), then, when you want to play things back simply skip the physics engine entirely and just reposition your objects every frame from your recorded position/rotation states.
All you'll need to do is make sure your frames for the play-back are the same duration as they were when the physics was running."
When I capture camera images of projected patterns using openCV via 'cvQueryFrame', I often end up with an unintended artifact: the projector's scan line. That is, since I'm unable to precisely time when 'cvQueryFrame' captures an image, the image taken does not respect the constant 30Hz refresh of the projector. The result is that typical horizontal band familiar to those who have turned a video camera onto a TV screen.
Short of resorting to hardware sync, has anyone had some success with approximate (e.g., 'good enough') informal projector-camera sync in openCV?
Below are two solutions I'm considering, but was hoping this is a common enough problem that an elegant solution might exist. My less-than-elegant thoughts are:
Add a slider control in the cvWindow displaying the video for the user to control a timing offset from 0 to 1/30th second, then set up a queue timer at this interval. Whenever a frame is needed, rather than calling 'cvQueryFrame' directly, I would request a callback to execute 'cvQueryFrame' at the next firing of the timer. In this way, theoretically the user would be able to use the slider to reduce the scan line artifact, provided that the timer resolution is sufficient.
After receiving a frame via 'cvQueryFrame', examine the frame for the tell-tale horizontal band by looking for a delta in HSV values for a vertical column of pixels. Naturally this would only work when the subject being photographed contains a fiducial strip of uniform color under smoothly varying lighting.
I've used several cameras with OpenCV, most recently a Canon SLR (7D).
I don't think that your proposed solution will work. cvQueryFrame basically copies the next available frame from the camera driver's buffer (or advances a pointer in a memory mapped region, or blah according to your driver implementation).
In any case, the timing of the cvQueryFrame call has no effect on when the image was captured.
So as you suggested, hardware sync is really the only route, unless you have a special camera, like a point grey camera, which gives you explicit software control of the frame integration start trigger.
I know this has nothing to do with synchronizing but, have you tried extending the exposure time? Or doing so by intentionally "blending" two or more images into one?