Let's say I have two frequencies sounding at the same time.
There's a harmonic series above each, right.
I need to know the frequency of the coincidental partial in order to apply a band pass filter around that frequency in real time.
Can this be done?
if you have two frequencies, then you have two sinus in superposition.
you have two option to extract one of the frequencies:
do a fft, then in the output you will get the two frequencies represented as two bins, then select the bin that represent the frequency you are interested in, (this is time consuming if you are doing the fft especially for this reason).
you can also implement an iir (or fir, iir is usually more efficient) band pass filter around the relevant frequency (if you have only two frequencies a low pass or high pass will be sufficient).
Related
I have time series data that I want to decompose into its intrinsic frequencies. After running FFT on the series, I am left with a periodogram with the main frequencies.
Is there any way to extract a specific frequency and convert it back into the original time-domain?
For example, if I find a 7-day frequency, I can understand that this is a weekly cycle, however, is there any way to understand on what day the peaks occur?
Use a complex FFT instead of a scalar periodogram. The magnitude of a complex FFT shows the peaks, but the complex inverse (complex IFFT) of a single complex FFT peak (might be more than one sample in width) will retain the phase information and result in the proper time relationship of a particular cyclic component to the original time series.
Note: The input to the complex IFFT needs to retain conjugate symmetry for the result to be real instead of complex (except for rounding/quantization noise).
Is it possible to decorrelate accelerometer data in real-time? If so, how is it done?
Background:
My application is receiving (X,Y,Z) accelerometer data in real-time (sample rate is 6.75Hz). The sensor is moving in a periodic motion but the motion is not necessarily along only one axis. The 3 signals x(t), y(t) and z(t) are therefore slightly correlated and I would like to know if I can find a rotation matrix (in real time) which can be used to rotate the measured (x,y,z) into a new vector (x*,y*,z*) so that the entire motion is along the z-axis?
I would like to implement the algorithm in C.
Thanks.
What you're trying to do is generally called "principal component analysis". The Wikipedia article is pretty good:
https://en.wikipedia.org/wiki/Principal_component_analysis
For static data you generally use the eigenvectors of the covariance matrix as your new coordinate basis.
PCA in real time is doable, but not super easy. See, for example: http://www.bio-conferences.org/articles/bioconf/pdf/2011/01/bioconf_skills_00055.pdf
I'd like to first of all emphasize that Matt Timmermans' answer has done exactly what people are actually doing when classifying accelerometer data from clinical studies (a project I worked on).
Then: you're observing a sampled signal. In general, if you have a sensor that gives you samples at a rate of 6.75Hz, the highest frequency of a signal you can detect is 6.75Hz/2 = 3.375Hz. Everything that has a frequency higher than that will inherently be aliased back and look like it was something with a frequency f with 0<=f<3.375Hz. If you've not considered this, please go and read up on the Nyquist–Shannon sampling theorem. Especially: shield your sensors (however you do that, e.g. by employing dampeners) from all input above that limit, otherwise your measurements might be worth very little or even nothing. If your sensor does this internally (that's absolutely possible, there are enough accelerometers with analog low pass filters), this has been taken care of. However, document that characteristics of your sensor.
Now, your case is a little bit easier because you know pretty well that your whole observation is going to be periodic, and it's measured along three orthogonal axis.
In this case, just doing three discrete Fourier transforms at once, extracting the "strongest" spectral component over all three channels, and finding the phase of that spectral component (which is but the complex argument of that DFT bin) in the two others would give you something that you can map to a periodic movement around a specific axis in 3D space. If you want to, remove these value (set the bins to 0), and search for strongest component again etc.
Discrete cosine transforms can be done in staggering speed nowadays. with 6.75Hz, no PC in this world will ever get into trouble when you try this while you receive further samples. It's a hilariously low sampling rate.
Another, more elegant (read: you need less samples to compute this) would be using a parametric estimator; in your case, a direction-of-arrival sensor from the world of RF technology with multiple antennas would, as far as I can think, map directly to detection of rotational axis. The classical algorithms here are MUSIC and ESPRIT, and for your case (limited, known amount of oscillating parts), ESPRIT might be the better choice.
I have some geographical trajectories sampled to analyze, and I calculated the histogram of data in spatial and temporal dimension, which yielded a time domain based feature for each spatial element. I want to perform a discrete FFT to transform the time domain based feature into frequency domain based feature (which I think maybe more robust), and then do some classification or clustering algorithms.
But I'm not sure using what descriptor as frequency domain based feature, since there are amplitude spectrum, power spectrum and phase spectrum of a signal and I've read some references but still got confused about the significance. And what distance (similarity) function should be used as measurement when performing learning algorithms on frequency domain based feature vector(Euclidean distance? Cosine distance? Gaussian function? Chi-kernel or something else?)
Hope someone give me a clue or some material that I can refer to, thanks~
Edit
Thanks to #DrKoch, I chose a spatial element with the largest L-1 norm and plotted its log power spectrum in python and it did show some prominent peaks, below is my code and the figure
import numpy as np
import matplotlib.pyplot as plt
sp = np.fft.fft(signal)
freq = np.fft.fftfreq(signal.shape[-1], d = 1.) # time sloth of histogram is 1 hour
plt.plot(freq, np.log10(np.abs(sp) ** 2))
plt.show()
And I have several trivial questions to ask to make sure I totally understand your suggestion:
In your second suggestion, you said "ignore all these values."
Do you mean the horizontal line represent the threshold and all values below it should be assigned to value zero?
"you may search for the two, three largest peaks and use their location and probably widths as 'Features' for further classification."
I'm a little bit confused about the meaning of "location" and "width", does "location" refer to the log value of power spectrum (y-axis) and "width" refer to the frequency (x-axis)? If so, how to combine them together as a feature vector and compare two feature vector of "a similar frequency and a similar widths" ?
Edit
I replaced np.fft.fft with np.fft.rfft to calculate the positive part and plot both power spectrum and log power spectrum.
code:
f, axarr = plt.subplot(2, sharex = True)
axarr[0].plot(freq, np.abs(sp) ** 2)
axarr[1].plot(freq, np.log10(np.abs(sp) ** 2))
plt.show()
figure:
Please correct me if I'm wrong:
I think I should keep the last four peaks in first figure with power = np.abs(sp) ** 2 and power[power < threshold] = 0 because the log power spectrum reduces the difference among each component. And then use the log spectrum of new power as feature vector to feed classifiers.
I also see some reference suggest applying a window function (e.g. Hamming window) before doing fft to avoid spectral leakage. My raw data is sampled every 5 ~ 15 seconds and I've applied a histogram on sampling time, is that method equivalent to apply a window function or I still need apply it on the histogram data?
Generally you should extract just a small number of "Features" out of the complete FFT spectrum.
First: Use the log power spec.
Complex numbers and Phase are useless in these circumstances, because they depend on where you start/stop your data acquisiton (among many other things)
Second: you will see a "Noise Level" e.g. most values are below a certain threshold, ignore all these values.
Third: If you are lucky, e.g. your data has some harmonic content (cycles, repetitions) you will see a few prominent Peaks.
If there are clear peaks, it is even easier to detect the noise: Everything between the peaks should be considered noise.
Now you may search for the two, three largest peaks and use their location and probably widths as "Features" for further classification.
Location is the x-value of the peak i.e. the 'frequency'. It says something how "fast" your cycles are in the input data.
If your cycles don't have constant frequency during the measuring intervall (or you use a window before caclculating the FFT), the peak will be broader than one bin. So this widths of the peak says something about the 'stability' of your cycles.
Based on this: Two patterns are similar if the biggest peaks of both hava a similar frequency and a similar widths, and so on.
EDIT
Very intersiting to see a logarithmic power spectrum of one of your examples.
Now its clear that your input contains a single harmonic (periodic, oscillating) component with a frequency (repetition rate, cycle-duration) of about f0=0.04.
(This is relative frquency, proprtional to the your sampling frequency, the inverse of the time beetween individual measurment points)
Its is not a pute sine-wave, but some "interesting" waveform. Such waveforms produce peaks at 1*f0, 2*f0, 3*f0 and so on.
(So using an FFT for further analysis turns out to be very good idea)
At this point you should produce spectra of several measurements and see what makes a similar measurement and how differ different measurements. What are the "important" features to distinguish your mesurements? Thinks to look out for:
Absolute amplitude: Height of the prominent (leftmost, highest) peaks.
Pitch (Main cycle rate, speed of changes): this is position of first peak, distance between consecutive peaks.
Exact Waveform: Relative amplitude of the first few peaks.
If your most important feature is absoulute amplitude, you're better off with calculating the RMS (root mean square) level of our input signal.
If pitch is important, you're better off with calculationg the ACF (auto-correlation function) of your input signal.
Don't focus on the leftmost peaks, these come from the high frequency components in your input and tend to vary as much as the noise floor.
Windows
For a high quality analyis it is importnat to apply a window to the input data before applying the FFT. This reduces the infulens of the "jump" between the end of your input vector ant the beginning of your input vector, because the FFT considers the input as a single cycle.
There are several popular windows which mark different choices of an unavoidable trade-off: Precision of a single peak vs. level of sidelobes:
You chose a "rectangular window" (equivalent to no window at all, just start/stop your measurement). This gives excellent precission of your peaks which now have a width of just one sample. Your sidelobes (the small peaks left and right of your main peaks) are at -21dB, very tolerable given your input data. In your case this is an excellent choice.
A Hanning window is a single cosine wave. It makes your peaks slightly broader but reduces side-lobe levels.
The Hammimg-Window (cosine-wave, slightly raised above 0.0) produces even broader peaks, but supresses side-lobes by -42 dB. This is a good choice if you expect further weak (but important) components between your main peaks or generally if you have complicated signals like speech, music and so on.
Edit: Scaling
Correct scaling of a spectrum is a complicated thing, because the values of the FFT lines depend on may things like sampling rate, lenght of FFT, window, and even implementation details of the FFT algorithm (there exist several different accepted conventions).
After all, the FFT should show the underlying conservation of energy. The RMS of the input signal should be the same as the RMS (Energy) of the spectrum.
On the other hand: if used for classification it is enough to maintain relative amplitudes. As long as the paramaters mentioned above do not change, the result can be used for classification without further scaling.
I'm working on signal processing issues. I'm extracting some features for feeding a classifier. Among these features, there is the sum of first 5 FFT coefficients. As you know primary FFT coefficients actually indicate how dominant low frequency components of a signal are. This is very close to what a low-pass filter gives.
Here I'm suspicious about whether computing FFT to take those first 5 coefficients is an unnecessary task. I think applying low-pass filter will just eliminate low-frequency components and it won't have a significant effect on primary FFT coefficients. However there may be some other way in combination with low-pass filter in order to extract same information (that is contained in first five FFT coefficients) without using FFT.
Do you have any ideas or suggestions regarding this issue?
Thanks in advance.
If you just need an indicator for the low freq part of a signal I suggest to do something really simple. Just take an ordinary lowpass filter, for instance a 2nd order butterworth, with the cutoff frequency set appropriately (5Hz in your case, if I understood right). Then compute the energy (sum over squared values) or rms-value over your window (length 100). Or perhaps take the ratio of the low-freq energy and the overall energy of the window, to get a relative measure. That should give you a pretty good indicator for low frequency contributions of your signal.
People tend to overuse the fft for all kinds of really simple tasks. In 90% of the use cases an fft can be replaced by a simpler algorithm.
I seems you should take a look at the Goertzel Algorithm, as for the seemingly limited number of frequencies you need, it should take less computation. After updating the feedback parts on each sample, you can select how often to generate your "feature metric" or a little additional weighting of the results, can yield a respectable low pass filter.
Assume a sequence of numbers (wave-like data). I perform then the DFT (or FFT) transform. Next step I want to achieve is to find the frequencies, that correspond to the real frequencies that are included in data. As we know, DFT output has real and imaginary part a[i] and b[i]. If we look at spectrum (sqrt(a[i]^2+b[i]^2) then the maximum in it corresponds to the frequency that is included to the data. The question is how to find all frequencies from DFT? The problem arises when there are many other peaks that can be falsely selected.
I had a similar problem when doing spectral analysis processing of data when I was writing my honours thesis.
You are right: To find dominant frequencies you generally only need to look at the magnitude of the complex value in the DFT.
Unfortunately, you pretty much have to write some sort of intelligent algorithm which will identify the peaks (frequencies). The way the algorithm works is highly dependent on what the DFT looks like for your application. My DFTs all had similar characteristics, so it wasn't too difficult to put together a heuristic algorithm. If your DFT can take on any form, then you will probably get a lot of false positives and/or false negatives.
The way I did it was to identify regions in the DFT with high magnitude (peaks) which were surrounded by low magnitude (troughs). You can define the minimum difference between peaks and troughs (the sensitivity) as a constant times the standard deviation of the data. Additionally, you can say that any peaks that fall below a certain magnitude (threshold) are ignored altogether, as they are just noise.
Of course, the above technique will only really work if you have relatively well defined frequencies in your data. If your DFT is highly random, then you will need to take extra care to set the sensitivity and threshold carefully.
Don't forget that the magnitude of your data is symmetric, so you only need to look at half of it.
Once you have identified the frequencies in your DFT, don't forget to convert it into the units you want. From memory, if you have n samples taken with time discretisation dt, then if you have a peak at data point 5 (for example), where the first data point is 1, then the frequency is 1/(n*dt) radians per time unit. (I haven't done this in a while, so that formula might be off by a factor of Pi or something)