I have captured both a transmitted signal and when there is no transmission (i.e. noise only). I would like to calculate the SNR of the signal. I would like to make sure the following GNURadio flowgraph is not wrong:
In summary, after the PSD of each is calculated, the "Integrate with Decimate over 2048" block sums up the power over the 2048 FFT bins. Then, the noise FFT sum is subtracted from the signal FFT sum. This is divided by the noise FFT sum and converted to dB.
This is the output of my flowgraph:
As calculated by my flowgraph, the power values are:
signal only, raw power: ~0.329
noise only, raw power: 0.000007
SNR in dB: ~46.6dB
I'm using a LoRa node to transmit the signal of interest; the modulation details are here: https://www.thethingsnetwork.org/docs/lorawan/#modulation-and-data-rate
The signal occupies the captured bandwidth (125k) and is sampled at 1 million samples per second.
Your flowgraph should give you the correct SNR value under the following conditions:
the signal and noise sources are uncorrelated
the "noise only" captured by the lower branch has the same characteristics (especially the same average power) as the noise included in the "signal + noise" captured by the upper branch
As an aside, unless you are also using intermediate signals for other purposes, there are a few simplifications that can be made to your flowgraph:
The multiplications up the upper and lower branches by the same constant factor will eventually cancel out in the divide block. You could save yourself the trouble of the scaling altogether.
From Parseval's theorem, the summation of the squared magnitudes in the frequency-domain is proportional to the summation of the squared samples in the time-domain. The FFT blocks would thus not be necessary.
That said, in your flowgraph you are using some intermediate signals for GUI output purposes. In this case, you could simply put the required constant scaling just before the Number Sink.
Related
I'm currently working on a program in C++ in which I am computing the time varying FFT of a wav file. I have a question regarding plotting the results of an FFT.
Say for example I have a 70 Hz signal that is produced by some instrument with certain harmonics. Even though I say this signal is 70 Hz, it's a real signal and I assume will have some randomness in which that 70Hz signal varies. Say I sample it for 1 second at a sample rate of 20kHz. I realize the sample period probably doesn't need to be 1 second, but bear with me.
Because I now have 20000 samples, when I compute the FFT. I will have 20000 or (19999) frequency bins. Let's also assume that my sample rate in conjunction some windowing techniques minimize spectral leakage.
My question then: Will the FFT still produce a relatively ideal impulse at 70Hz? Or will there 'appear to be' spectral leakage which is caused by the randomness the original signal? In otherwords, what does the FFT look like of a sinusoid whose frequency is a random variable?
Some of the more common modulation schemes will add sidebands that carry the information in the modulation. Depending on the amount and type of modulation with respect to the length of the FFT, the sidebands can either appear separate from the FFT peak, or just "fatten" a single peak.
Your spectrum will appear broadened and this happens in the real world. Look e.g for the Voight profile, which is a Lorentizan (the result of an ideal exponential decay) convolved with a Gaussian of a certain width, the width being determined by stochastic fluctuations, e.g. Doppler effect on molecules in a gas that is being probed by a narrow-band laser.
You will not get an 'ideal' frequency peak either way. The limit for the resolution of the FFT is one frequency bin, (frequency resolution being given by the inverse of the time vector length), but even that (as #xvan pointed out) is in general broadened by the window function. If your window is nonexistent, i.e. it is in fact a square window of the length of the time vector, then you'll get spectral peaks that are convolved with a sinc function, and thus broadened.
The best way to visualize this is to make a long vector and plot a spectrogram (often shown for audio signals) with enough resolution so you can see the individual variation. The FFT of the overall signal is then the projection of the moving peaks onto the vertical axis of the spectrogram. The FFT of a given time vector does not have any time resolution, but sums up all frequencies that happen during the time you FFT. So the spectrogram (often people simply use the STFT, short time fourier transform) has at any given time the 'full' resolution, i.e. narrow lineshape that you expect. The FFT of the full time vector shows the algebraic sum of all your lineshapes and therefore appears broadened.
To sum it up there are two separate effects:
a) broadening from the window function (as the commenters 1 and 2 pointed out)
b) broadening from the effect of frequency fluctuation that you are trying to simulate and that happens in real life (e.g. you sitting on a swing while receiving a radio signal).
Finally, note the significance of #xvan's comment : phi= phi(t). If the phase angle is time dependent then it has a derivative that is not zero. dphi/dt is a frequency shift, so your instantaneous frequency becomes f0 + dphi/dt.
I have some geographical trajectories sampled to analyze, and I calculated the histogram of data in spatial and temporal dimension, which yielded a time domain based feature for each spatial element. I want to perform a discrete FFT to transform the time domain based feature into frequency domain based feature (which I think maybe more robust), and then do some classification or clustering algorithms.
But I'm not sure using what descriptor as frequency domain based feature, since there are amplitude spectrum, power spectrum and phase spectrum of a signal and I've read some references but still got confused about the significance. And what distance (similarity) function should be used as measurement when performing learning algorithms on frequency domain based feature vector(Euclidean distance? Cosine distance? Gaussian function? Chi-kernel or something else?)
Hope someone give me a clue or some material that I can refer to, thanks~
Edit
Thanks to #DrKoch, I chose a spatial element with the largest L-1 norm and plotted its log power spectrum in python and it did show some prominent peaks, below is my code and the figure
import numpy as np
import matplotlib.pyplot as plt
sp = np.fft.fft(signal)
freq = np.fft.fftfreq(signal.shape[-1], d = 1.) # time sloth of histogram is 1 hour
plt.plot(freq, np.log10(np.abs(sp) ** 2))
plt.show()
And I have several trivial questions to ask to make sure I totally understand your suggestion:
In your second suggestion, you said "ignore all these values."
Do you mean the horizontal line represent the threshold and all values below it should be assigned to value zero?
"you may search for the two, three largest peaks and use their location and probably widths as 'Features' for further classification."
I'm a little bit confused about the meaning of "location" and "width", does "location" refer to the log value of power spectrum (y-axis) and "width" refer to the frequency (x-axis)? If so, how to combine them together as a feature vector and compare two feature vector of "a similar frequency and a similar widths" ?
Edit
I replaced np.fft.fft with np.fft.rfft to calculate the positive part and plot both power spectrum and log power spectrum.
code:
f, axarr = plt.subplot(2, sharex = True)
axarr[0].plot(freq, np.abs(sp) ** 2)
axarr[1].plot(freq, np.log10(np.abs(sp) ** 2))
plt.show()
figure:
Please correct me if I'm wrong:
I think I should keep the last four peaks in first figure with power = np.abs(sp) ** 2 and power[power < threshold] = 0 because the log power spectrum reduces the difference among each component. And then use the log spectrum of new power as feature vector to feed classifiers.
I also see some reference suggest applying a window function (e.g. Hamming window) before doing fft to avoid spectral leakage. My raw data is sampled every 5 ~ 15 seconds and I've applied a histogram on sampling time, is that method equivalent to apply a window function or I still need apply it on the histogram data?
Generally you should extract just a small number of "Features" out of the complete FFT spectrum.
First: Use the log power spec.
Complex numbers and Phase are useless in these circumstances, because they depend on where you start/stop your data acquisiton (among many other things)
Second: you will see a "Noise Level" e.g. most values are below a certain threshold, ignore all these values.
Third: If you are lucky, e.g. your data has some harmonic content (cycles, repetitions) you will see a few prominent Peaks.
If there are clear peaks, it is even easier to detect the noise: Everything between the peaks should be considered noise.
Now you may search for the two, three largest peaks and use their location and probably widths as "Features" for further classification.
Location is the x-value of the peak i.e. the 'frequency'. It says something how "fast" your cycles are in the input data.
If your cycles don't have constant frequency during the measuring intervall (or you use a window before caclculating the FFT), the peak will be broader than one bin. So this widths of the peak says something about the 'stability' of your cycles.
Based on this: Two patterns are similar if the biggest peaks of both hava a similar frequency and a similar widths, and so on.
EDIT
Very intersiting to see a logarithmic power spectrum of one of your examples.
Now its clear that your input contains a single harmonic (periodic, oscillating) component with a frequency (repetition rate, cycle-duration) of about f0=0.04.
(This is relative frquency, proprtional to the your sampling frequency, the inverse of the time beetween individual measurment points)
Its is not a pute sine-wave, but some "interesting" waveform. Such waveforms produce peaks at 1*f0, 2*f0, 3*f0 and so on.
(So using an FFT for further analysis turns out to be very good idea)
At this point you should produce spectra of several measurements and see what makes a similar measurement and how differ different measurements. What are the "important" features to distinguish your mesurements? Thinks to look out for:
Absolute amplitude: Height of the prominent (leftmost, highest) peaks.
Pitch (Main cycle rate, speed of changes): this is position of first peak, distance between consecutive peaks.
Exact Waveform: Relative amplitude of the first few peaks.
If your most important feature is absoulute amplitude, you're better off with calculating the RMS (root mean square) level of our input signal.
If pitch is important, you're better off with calculationg the ACF (auto-correlation function) of your input signal.
Don't focus on the leftmost peaks, these come from the high frequency components in your input and tend to vary as much as the noise floor.
Windows
For a high quality analyis it is importnat to apply a window to the input data before applying the FFT. This reduces the infulens of the "jump" between the end of your input vector ant the beginning of your input vector, because the FFT considers the input as a single cycle.
There are several popular windows which mark different choices of an unavoidable trade-off: Precision of a single peak vs. level of sidelobes:
You chose a "rectangular window" (equivalent to no window at all, just start/stop your measurement). This gives excellent precission of your peaks which now have a width of just one sample. Your sidelobes (the small peaks left and right of your main peaks) are at -21dB, very tolerable given your input data. In your case this is an excellent choice.
A Hanning window is a single cosine wave. It makes your peaks slightly broader but reduces side-lobe levels.
The Hammimg-Window (cosine-wave, slightly raised above 0.0) produces even broader peaks, but supresses side-lobes by -42 dB. This is a good choice if you expect further weak (but important) components between your main peaks or generally if you have complicated signals like speech, music and so on.
Edit: Scaling
Correct scaling of a spectrum is a complicated thing, because the values of the FFT lines depend on may things like sampling rate, lenght of FFT, window, and even implementation details of the FFT algorithm (there exist several different accepted conventions).
After all, the FFT should show the underlying conservation of energy. The RMS of the input signal should be the same as the RMS (Energy) of the spectrum.
On the other hand: if used for classification it is enough to maintain relative amplitudes. As long as the paramaters mentioned above do not change, the result can be used for classification without further scaling.
I am running some experiments on MATLAB, and I have noticed that, keeping the period fixed, increasing the sampling rate of a sine signal causes the different shifted waveforms in the Fourier transform to become more distinct. They get further apart, I think this makes sense because as the sampling rate increases, the difference between the Nyquist rate and the sampling rate increases too, which creates an effect opposed to aliasing. I have also noticed that the amplitude of the peaks of the transform also increase as the sampling rate increases. Even the DC component (frequency = 0) changes. It's shown as being 0 at some sampling rate, but when increasing the sampling rate it's not 0 anymore.
All the sampling rates are above the Nyquist rate. It seems odd to me that the Fourier transform changes its shape, since according to the sampling theorem, the original signal can be recovered if the sampling rate is above the Nyquist rate, no matter if it's 2 times the nyquist rate or 20 times. Wouldn't a different Fourier waveform mean a different recovered signal?
I am wondering, formally, what's the impact of the sampling rate
Thank you.
You're conflating conversion between time-discrete and time-continuous forms of a signal with reversibility of a transform.
The only guarantee is: For a given transform of some discrete signal, its inverse transform will yield the "same" discrete signal back. The discrete signal is abstract from any frequencies. All that the transform does is take some vector of complex values, and give the dimensionally matching vector of complex values back. You can then take this vector, run an inverse transform on it, and get the "original" vector. I use quotes since there may be some numerical errors that depend on the implementation. As you can see, nowhere does the word frequency appear because it's irrelevant.
So, your real question is then, how to get an FFT with values that are useful for something besides getting the original discrete signal back through an inverse transform. Say, how to get an FFT that will tell a human something nice about the frequency content of a signal. A transform "tweaked" for human usefulness, or for use in further signal processing such as automated music transcription, can't reproduce the original signal anymore after inversion. We're trading off veracity for usefulness. Detailed discussion of this can't really fit into one answer, and is off topic here anyway.
Another of your real questions is how to go between a continuous signal and a discrete signal - how to sample the continuous signal, and how to reconstruct it from its discrete representation. The reconstruction means a function (or process) that will yield the values the signal had at points in time between the samples. Again, this is a big topic.
You are seeing several things when you increase the sample rate:
most (forward) FFT implementations have an implicit scaling factor of N (sometimes sqrt(N)) - if you're increasing your FFT size as you increase the sample rate (i.e. keeping the time window constant) then the apparent magnitude of the peaks in the FFT will increase. When calculating absolute magnitude values you would normally need to take this scaling factor into account.
I'm guessing that you are not currently applying a window function prior to the FFT - this will result in "smearing" of the spectrum, due to spectral leakage, and the exact nature of this will be very dependent on the relationship between sample rate and the frequencies of the various components in your signal. Apply a window function and the spectrum should look a lot more consistent as you vary the sample rate.
I wish to use Stein Unbiased Estimate of Risk (Sure) for denoising signals.
I have a 1-Dimensional signal. I am using wavelets to decompose the signal into multiple levels of approximate and detail coefficients.
For denoising the original signal, do I need to do a thresholding for every level of detail coefficients or doing it on the last level of detail coeffcient will do the job ?
Thresholding is usually applied to all the frequencies of a signal because the procedure exploits the fact that the wavelet transform maps white noise (purely random, uncorrelated and constant power spectral density noise) in the signal domain to white noise in the transform domain and as such is spread across the different frequencies Thus, while signal energy becomes more concentrated into fewer coefficients in the transform domain, noise energy does not. Other noises given that have different spectrum properties will map differently and this is where the selection of the type of thresholding procedure becomes important.
In thresholding the highest decomposition level (lowest frequencies) while leaving the lower levels (higher frequencies) not denoised sounds a little extrange if you want to reconstruct the signal.
However you could also extract a level and denoise its related range of frequencies (e.g. from level 1 to level 2) if you have a range of frequencies you may have interest for.
Speaking about the thresholding function be aware in any case that Sure has different results depending on the type of noises the signal has. For example it will reduce the distribution of white noise in horizontal components but will only decrease large amplitudes. For signals where togueter with white you may have other noise colors like random walk and flicker noise sure is not an efective procedure.
Is there any formula to calculate the frequency (or frequencys) of a signal that is bad sampled?
For example, what's the output of an analog signal with F=22Khz when it's sampled at 25Khz, or 10Khz?
EDIT:
In this example, the sampled signal (on the right) have a different frequency than the original one, because it was bad sampled (Fs is minor than 2*F)
My question is: is there any formula to know what's the frequency of this 20kHz signal, sampled at 30kHz?
No any formula to know what's the frequency of 20kHz signal, sampled at 30kHz. But it is a fact that the frequency of undersampled signal will be reflected about Nyquist frequency. In your example 30 kHz means that Nyquist frequency is about 15 KHz, that is not enough to record original signal (20KHz) correctly, only 15 kHz of it distributed, another 5 KHz (reminder after distribution of 15 KHZ) during reflection about Nyquist frequency appear in position 15-5=10 KHz. This is final ansver. The frequency of sampled signal will be equal 10 kHz in your case
Unless the bandwidth of the signal is less than half the sampling rate, you lose information during sampling and generally can't distinguish frequencies after that due to aliasing.
See Undersampling for more details about sampling at rates lower than twice the maximum signal frequency.
There's no simple formula that can give you the spectral content of a signal or the main frequency. In general you need to calculate a Discrete Fourier Transform of the sampled signal to find that out. If you're interested in whether or not there's a specific frequency, or how strong it is, you can calculate DFT at that frequency. The Goertzel algorithm can be an option.
EDIT: a signal at frequency f such that fsample/2 <= f < fsample will alias to f* = fsample - f, hence a 20KHz sine wave sampled at 30KHz will appear as a 10KHz sine wave.
In general frequencies above the fsample/2 can be observed in the sampled signal, but their frequency is ambiguous. That is, a frequency component with frequency f cannot be distinguished from other components with frequencies N*fsample/2 + f and N*fsample/2 – f for nonzero integers N. This ambiguity is called aliasing*.
Assuming a constant sampling rate, any sampling will alias together spectral content from below and above the sampling rate. If you have frequency content on both sides of the sampling rate that you don't want combined, you will have to filter one or the other frequency band Out before the sampling, or you will have a problem. For instance a low-pass filter which only passes signals below Fs/2, or a bandpass filter that only passes signals strictly between n*Fs/2 and (n+1)*Fs/2 for some integer n, might be appropriate.
Note that the accuracy of the sampling rate must be higher (lower jitter) for n > 0. Lack of this lower jitter would be an example of bad sampling that would add random phase noise.