Using wavesurfer to analyse bird songs data and getting negative amplitudes with the FFT analysis that I can not find the reason why.
I have been using wavesurfer for analysing my paper data on birds song. I open my data with the spectrogram and then in spectrum section is possible to find frequency and amplitude (using FFT analysis). My amplitude is negative but I can not find a way to justify that. I dont know why is that and I have significant results on my data, meaning that everything needs to be justified. The forum of the software does not work and there is literally no answer to my question on the internet. I have even emailed the creators asking for help. Find a screen attached to the windows.
dB are on a logarithmic scale. The log of a small enough positive FFT magnitude can be negative.
e.g. 20*log10(0.1) = -20
We express values in dB in order to say something like: it's n times as large. A dB quantity is the log10 of a ratio, it is used to make a comparison between two measurements, for example output vs. input. Sometimes there is a single measurement compared with a fixed value. In that case we do not use the dB unit but some variants, e.g. if the reference is 1mW, we use dBm. The logarithmic scale facilitates calculations of attenuation and gain along a processing chain. A negative value indicates the ratio is less than 1, and the signal is attenuated (or less than the other), conversely a positive value indicates the signal is amplified (or greater than the other).
A dB (deci+Bel) is 0.1 B, so -60.1dB is -6.01B.
Power ratio
Originally the dB scale is related to power. Assuming the value is a power, you have to find the number which log10 is -6.01, a matter of taking the exponential of both sides:
log(x) = -6.01 (after conversion to B)
10^(log(x)) = 10^-6.01
x = 9.7e-07 ≈ 1/1,000,00
So your measurement is a power 1e6 times smaller than another power. Which one? It's not told, but it's likely the spectral line with the maximum value.
Voltage ratio
When dB relate to a ratio of voltages, the number is doubled for the same ratio. The reason is:
It is assumed power is increased (or decreased) by managing to increase (decrease) some voltage. A change in the same proportion occurs for the current according to Ohm's law.
Power is voltage x current, therefore to change the power in the ratio k, it's sufficient to change the voltage in the ratio sqrt(k). Log (sqrt(k)) = 1/2*log(k).
E.g.
Power increase from 10W to 20W, a ratio of 2, is log(2) = 0.3B or 3dB.
Voltage increase from 5V to 10V, also a ratio of 2 but 0.3B x 2 or 6dB.
Root-power quantities vs. root-power quantities
The rule actually separates power quantities (like power, energy or acoustic intensity) and root-power quantity (like voltage, sound pressure and electric field).
Related
I'm currently working on a program in C++ in which I am computing the time varying FFT of a wav file. I have a question regarding plotting the results of an FFT.
Say for example I have a 70 Hz signal that is produced by some instrument with certain harmonics. Even though I say this signal is 70 Hz, it's a real signal and I assume will have some randomness in which that 70Hz signal varies. Say I sample it for 1 second at a sample rate of 20kHz. I realize the sample period probably doesn't need to be 1 second, but bear with me.
Because I now have 20000 samples, when I compute the FFT. I will have 20000 or (19999) frequency bins. Let's also assume that my sample rate in conjunction some windowing techniques minimize spectral leakage.
My question then: Will the FFT still produce a relatively ideal impulse at 70Hz? Or will there 'appear to be' spectral leakage which is caused by the randomness the original signal? In otherwords, what does the FFT look like of a sinusoid whose frequency is a random variable?
Some of the more common modulation schemes will add sidebands that carry the information in the modulation. Depending on the amount and type of modulation with respect to the length of the FFT, the sidebands can either appear separate from the FFT peak, or just "fatten" a single peak.
Your spectrum will appear broadened and this happens in the real world. Look e.g for the Voight profile, which is a Lorentizan (the result of an ideal exponential decay) convolved with a Gaussian of a certain width, the width being determined by stochastic fluctuations, e.g. Doppler effect on molecules in a gas that is being probed by a narrow-band laser.
You will not get an 'ideal' frequency peak either way. The limit for the resolution of the FFT is one frequency bin, (frequency resolution being given by the inverse of the time vector length), but even that (as #xvan pointed out) is in general broadened by the window function. If your window is nonexistent, i.e. it is in fact a square window of the length of the time vector, then you'll get spectral peaks that are convolved with a sinc function, and thus broadened.
The best way to visualize this is to make a long vector and plot a spectrogram (often shown for audio signals) with enough resolution so you can see the individual variation. The FFT of the overall signal is then the projection of the moving peaks onto the vertical axis of the spectrogram. The FFT of a given time vector does not have any time resolution, but sums up all frequencies that happen during the time you FFT. So the spectrogram (often people simply use the STFT, short time fourier transform) has at any given time the 'full' resolution, i.e. narrow lineshape that you expect. The FFT of the full time vector shows the algebraic sum of all your lineshapes and therefore appears broadened.
To sum it up there are two separate effects:
a) broadening from the window function (as the commenters 1 and 2 pointed out)
b) broadening from the effect of frequency fluctuation that you are trying to simulate and that happens in real life (e.g. you sitting on a swing while receiving a radio signal).
Finally, note the significance of #xvan's comment : phi= phi(t). If the phase angle is time dependent then it has a derivative that is not zero. dphi/dt is a frequency shift, so your instantaneous frequency becomes f0 + dphi/dt.
Let's say I have some data that corresponds to the average temperature in a city measured every minute for around 1 year. How can I determine if there's cyclical patterns from the data using an FFT?
I know how it works for sound... I do an FFT of a sound wave and now the magnitude is shown in the Y axis and the frequency in Hertz is shown in the X-axis because the sampling frequency is in Hertz. But in my previous example the sampling frequency would be... 1 sample every minute, right? So how should I change it to something meaningful? I would get cycles/minute instead of cycles per seconds? And what does cycles/minute would mean here?
I think your interpretation is correct - you are just scaling to different units. Once you've found the spectral peak you might find it more useful to take the reciprocal to express the value in minutes/cycle (ie the length of the periodic cycle). Effectively this is thinking in terms of wavelength rather than frequency.
I want to select an optimal window for STFT for different audio signals. For a signal with frequency contents from 10 Hz to 300 Hz what will be the appropriate window size ? similarly for a signal with frequency contents 2000 Hz to 20000 Hz, what will be the optimal window size ?
I know that if a window size is 10 ms then this will give you a frequency resolution of about 100 Hz. But if the frequency contents in the signal lies from 100 Hz to 20000 HZ then 10 ms will be appropriate window size ? or we should go for some other window size because of 20000 Hz frequency content in a signal ?
I know the classic "uncertainty principle" of the Fourier Transform. You can either have high resolution in time or high resolution in frequency but not both at the same time. The window lengths allow you to trade off between the two.
Windowed analysis is designed for quasi-stationary signals. Quasi-stationary signals are signals which change over time but on some short period of time they might be considered stable.
One example of quasi-stationary signal is speech. Frequency components of this signal change over time when position of tongue and mouth changes, but on a short period of time approximately 0.01s they might be considered stable because tongue does not move this fast. The range of 0.01s is determined by our biology, we just can't move tongue faster than that.
Another example is music. When you touch the string you might consider it produces more or less stable sound for some short period of time. Usually 0.05 seconds. Within this period you might consider sound stable.
There might be other types of signals, for example, it might have frequency 10Ghz and be quasi-stationary of 1ms of time.
Windowed analysis allows to capture both stationary properties of signal and change of signal over time. Here it does not matter what sample rate does signal have, what frequency resolution do you need or what are the main harmonics. Are main harmonics near 100Hz or near 3000Hz. It is important on what period of time the signal is stationary and on what it can be considered as changing.
So for speech 25ms window is good just because speech is quasi-stationary on that range. For music you usually take longer windows because our fingers are moving slower than our mouth. You need to study your signal to decide optimal window length or you need to provide more information about it.
You need to specify your "optimality" criteria.
For a desired frequency resolution, you need a length or window size of roughly Fs/df (or from a fraction to twice that length or more, depending on S/N and window). However the length also needs to be similar to or shorter than the length of time during which your signal is stationary within your desired frequency resolution bounds. This may not be possible or known, thus requiring you to specify which criteria (df vs. dt) is more important for your desired "optimality".
If multiple window lengths meet your criteria, then the shortest length that is a multiple of very small primes is likely to be the most computationally efficient for the following FFTs within an STFT computational sequence.
Based on the sampling theorem, the sampling frequency needs to be larger than twice the highest frequency of the signal. And based on DFT (discrete Fourier Transform), we also know that the frequency resolution is the inverse of the entire signal duration, and the the entire frequency span is the inverse of the time resolution. Note that the frequency is simply the inverse of the period, thus the relationships go inversely with each other.
Frequency resolution = 1 / (overall time duration)
Frequency span = 1 / (time resolution)
Having said that, to process 20kHz audio signal, we need to sample in 40kHz. And if we want to get the frequency resolution down, say to 10Hz, we will need to sample the entire duration as long as 0.1Sec, which is 1/10Hz.
This is the reason we normally see that audio files are said to be 44k. Because the human hearing range is limited to 20kHz. To add some margin to it, we use 44k sampling frequency in stead of 40kHz.
I think the uncertainty principle goes with the fact that more localized signal in one domain, actually spread out on the other. For example, a pulse in time domain goes from negative infinity to positive infinite, i.e the entire stretch of the spectrum. And vice versa that the a single frequency signal in spectrum stretches from negative infinity to positive infinite in time domain. This is simply because we had to go forever in order to know if a signal could be a pure sinusoidal signal or not.
But for DFT, we can always get the frequency span if we sample twice the highest frequency of the signal, and the resolution we want if we sample the signal duration long enough. So, not so uncertain as the uncertainty principle says, as long as we know how many samples to take and how fast and how long to take them.
I have some geographical trajectories sampled to analyze, and I calculated the histogram of data in spatial and temporal dimension, which yielded a time domain based feature for each spatial element. I want to perform a discrete FFT to transform the time domain based feature into frequency domain based feature (which I think maybe more robust), and then do some classification or clustering algorithms.
But I'm not sure using what descriptor as frequency domain based feature, since there are amplitude spectrum, power spectrum and phase spectrum of a signal and I've read some references but still got confused about the significance. And what distance (similarity) function should be used as measurement when performing learning algorithms on frequency domain based feature vector(Euclidean distance? Cosine distance? Gaussian function? Chi-kernel or something else?)
Hope someone give me a clue or some material that I can refer to, thanks~
Edit
Thanks to #DrKoch, I chose a spatial element with the largest L-1 norm and plotted its log power spectrum in python and it did show some prominent peaks, below is my code and the figure
import numpy as np
import matplotlib.pyplot as plt
sp = np.fft.fft(signal)
freq = np.fft.fftfreq(signal.shape[-1], d = 1.) # time sloth of histogram is 1 hour
plt.plot(freq, np.log10(np.abs(sp) ** 2))
plt.show()
And I have several trivial questions to ask to make sure I totally understand your suggestion:
In your second suggestion, you said "ignore all these values."
Do you mean the horizontal line represent the threshold and all values below it should be assigned to value zero?
"you may search for the two, three largest peaks and use their location and probably widths as 'Features' for further classification."
I'm a little bit confused about the meaning of "location" and "width", does "location" refer to the log value of power spectrum (y-axis) and "width" refer to the frequency (x-axis)? If so, how to combine them together as a feature vector and compare two feature vector of "a similar frequency and a similar widths" ?
Edit
I replaced np.fft.fft with np.fft.rfft to calculate the positive part and plot both power spectrum and log power spectrum.
code:
f, axarr = plt.subplot(2, sharex = True)
axarr[0].plot(freq, np.abs(sp) ** 2)
axarr[1].plot(freq, np.log10(np.abs(sp) ** 2))
plt.show()
figure:
Please correct me if I'm wrong:
I think I should keep the last four peaks in first figure with power = np.abs(sp) ** 2 and power[power < threshold] = 0 because the log power spectrum reduces the difference among each component. And then use the log spectrum of new power as feature vector to feed classifiers.
I also see some reference suggest applying a window function (e.g. Hamming window) before doing fft to avoid spectral leakage. My raw data is sampled every 5 ~ 15 seconds and I've applied a histogram on sampling time, is that method equivalent to apply a window function or I still need apply it on the histogram data?
Generally you should extract just a small number of "Features" out of the complete FFT spectrum.
First: Use the log power spec.
Complex numbers and Phase are useless in these circumstances, because they depend on where you start/stop your data acquisiton (among many other things)
Second: you will see a "Noise Level" e.g. most values are below a certain threshold, ignore all these values.
Third: If you are lucky, e.g. your data has some harmonic content (cycles, repetitions) you will see a few prominent Peaks.
If there are clear peaks, it is even easier to detect the noise: Everything between the peaks should be considered noise.
Now you may search for the two, three largest peaks and use their location and probably widths as "Features" for further classification.
Location is the x-value of the peak i.e. the 'frequency'. It says something how "fast" your cycles are in the input data.
If your cycles don't have constant frequency during the measuring intervall (or you use a window before caclculating the FFT), the peak will be broader than one bin. So this widths of the peak says something about the 'stability' of your cycles.
Based on this: Two patterns are similar if the biggest peaks of both hava a similar frequency and a similar widths, and so on.
EDIT
Very intersiting to see a logarithmic power spectrum of one of your examples.
Now its clear that your input contains a single harmonic (periodic, oscillating) component with a frequency (repetition rate, cycle-duration) of about f0=0.04.
(This is relative frquency, proprtional to the your sampling frequency, the inverse of the time beetween individual measurment points)
Its is not a pute sine-wave, but some "interesting" waveform. Such waveforms produce peaks at 1*f0, 2*f0, 3*f0 and so on.
(So using an FFT for further analysis turns out to be very good idea)
At this point you should produce spectra of several measurements and see what makes a similar measurement and how differ different measurements. What are the "important" features to distinguish your mesurements? Thinks to look out for:
Absolute amplitude: Height of the prominent (leftmost, highest) peaks.
Pitch (Main cycle rate, speed of changes): this is position of first peak, distance between consecutive peaks.
Exact Waveform: Relative amplitude of the first few peaks.
If your most important feature is absoulute amplitude, you're better off with calculating the RMS (root mean square) level of our input signal.
If pitch is important, you're better off with calculationg the ACF (auto-correlation function) of your input signal.
Don't focus on the leftmost peaks, these come from the high frequency components in your input and tend to vary as much as the noise floor.
Windows
For a high quality analyis it is importnat to apply a window to the input data before applying the FFT. This reduces the infulens of the "jump" between the end of your input vector ant the beginning of your input vector, because the FFT considers the input as a single cycle.
There are several popular windows which mark different choices of an unavoidable trade-off: Precision of a single peak vs. level of sidelobes:
You chose a "rectangular window" (equivalent to no window at all, just start/stop your measurement). This gives excellent precission of your peaks which now have a width of just one sample. Your sidelobes (the small peaks left and right of your main peaks) are at -21dB, very tolerable given your input data. In your case this is an excellent choice.
A Hanning window is a single cosine wave. It makes your peaks slightly broader but reduces side-lobe levels.
The Hammimg-Window (cosine-wave, slightly raised above 0.0) produces even broader peaks, but supresses side-lobes by -42 dB. This is a good choice if you expect further weak (but important) components between your main peaks or generally if you have complicated signals like speech, music and so on.
Edit: Scaling
Correct scaling of a spectrum is a complicated thing, because the values of the FFT lines depend on may things like sampling rate, lenght of FFT, window, and even implementation details of the FFT algorithm (there exist several different accepted conventions).
After all, the FFT should show the underlying conservation of energy. The RMS of the input signal should be the same as the RMS (Energy) of the spectrum.
On the other hand: if used for classification it is enough to maintain relative amplitudes. As long as the paramaters mentioned above do not change, the result can be used for classification without further scaling.
Assume a sequence of numbers (wave-like data). I perform then the DFT (or FFT) transform. Next step I want to achieve is to find the frequencies, that correspond to the real frequencies that are included in data. As we know, DFT output has real and imaginary part a[i] and b[i]. If we look at spectrum (sqrt(a[i]^2+b[i]^2) then the maximum in it corresponds to the frequency that is included to the data. The question is how to find all frequencies from DFT? The problem arises when there are many other peaks that can be falsely selected.
I had a similar problem when doing spectral analysis processing of data when I was writing my honours thesis.
You are right: To find dominant frequencies you generally only need to look at the magnitude of the complex value in the DFT.
Unfortunately, you pretty much have to write some sort of intelligent algorithm which will identify the peaks (frequencies). The way the algorithm works is highly dependent on what the DFT looks like for your application. My DFTs all had similar characteristics, so it wasn't too difficult to put together a heuristic algorithm. If your DFT can take on any form, then you will probably get a lot of false positives and/or false negatives.
The way I did it was to identify regions in the DFT with high magnitude (peaks) which were surrounded by low magnitude (troughs). You can define the minimum difference between peaks and troughs (the sensitivity) as a constant times the standard deviation of the data. Additionally, you can say that any peaks that fall below a certain magnitude (threshold) are ignored altogether, as they are just noise.
Of course, the above technique will only really work if you have relatively well defined frequencies in your data. If your DFT is highly random, then you will need to take extra care to set the sensitivity and threshold carefully.
Don't forget that the magnitude of your data is symmetric, so you only need to look at half of it.
Once you have identified the frequencies in your DFT, don't forget to convert it into the units you want. From memory, if you have n samples taken with time discretisation dt, then if you have a peak at data point 5 (for example), where the first data point is 1, then the frequency is 1/(n*dt) radians per time unit. (I haven't done this in a while, so that formula might be off by a factor of Pi or something)