How do I blend two images - thermal(80x60) and RGB(640x480) efficiently?
If I scale the thermal to 640x480 it doesn't scale up evenly or doesn't have enough quality to do any processing on it. Any ideas would be really helpful.
RGB image - http://postimg.org/image/66f9hnaj1/
Thermal image - http://postimg.org/image/6g1oxbm5n/
If you scale the resolution of the thermal image up by a factor of 8 and use Bilinear Interpolation you should get a smoother, less-blocky result.
When combining satellite images of different resolution, (I talk about satellite imagery because that is my speciality), you would normally use the highest resolution imagery as the Lightness or L channel to give you apparent resolution and detail in the shapes because the human eye is good at detecting contrast and then use the lower resolution imagery to fill in the Hue and Saturation, or a and b channels to give you the colour graduations you are hoping to see.
So, in concrete terms, I would consider converting the RGB to Lab or HSL colourspace and retaining the L channel. The take the thermal image and up-res it by 8 using bilinear interpolation and use the result as the a, or b or H or S and maybe fill in the remaining channel with the one from the RGB that has the most variance. Then convert the result back to RGB for a false-colour image. It is hard to tell without seeing the images or knowing what you are hoping to find in them. But in general terms, that would be my approach. HTH.
Note: Given that a of Lab colourspace controls the red/green relationship, I would probably try putting the thermal data in that channel so it tends to show more red the "hotter" the thermal channel is.
Updated Answer
Ok, now I can see your images and you have a couple more problems... firstly the images are not aligned, or registered, with each other which is not going to help - try using a tripod ;-) Secondly, your RGB image is very poorly exposed so it is not really going to contribute that much detail - especially in the shadows - to the combined image.
So, firstly, I used ImageMagick at the commandline to up-size the thermal image like this:
convert thermal.png -resize 640x480 thermal.png
Then, I used Photoshop to do a crude alignment/registration. If you want to try this, the easiest way is to put the two images into separate layers of the same document and set the Blending mode of the upper layer to Difference. Then use the Move Tool (shortcut v) to move the upper image around till the screen goes black which means that the details are on top of each other and when subtracted they come to zero, i.e. black. Then crop so the images are aligned and turn off one layer and save, then turn that layer back on and the other layer off and save again.
Now, I used ImageMagick again to separate the two images into Lab layers:
convert bigthermalaligned.png -colorspace Lab -separate thermal.png
convert rgbaligned.png -colorspace Lab -separate rgb.png
which gives me
thermal-0.png => L channel
thermal-1.png => a channel
thermal-2.png => b channel
rgb-0.png => L channel
rgb-1.png => a channel
rgb-2.png => b channel
Now I can take the L channel of the RGB image and the a and b channels of the thermal image and put them together:
convert rgba-0.png thermal-1.png thermal-2.png -normalize -set colorpsace lab -combine result.png
And you get this monstrosity! Obviously you can play around with the channels and colourpsaces and a tripod and proper exposures, but you should be able to see some of the details of the RGB image - especially the curtains on the left, the lights, the camera on the cellphone and the label on the water bottle - have come through into the final image.
Assuming that the images were not captured using a single camera, you need to note that the two cameras may have different parameters. Also, if it's two cameras, they are probably not located in the same world position (offset).
In order to resolve this, you need to get the intrinsic calibration matrix of each of the cameras, and find the offset between them.
Then, you can find a transformation between a pixel in one camera and the other. Unfortunately, if you don't have any depth information about the scene, the most you can do with the calibration matrix is get a ray direction from the camera position to the world.
The easy approach would be to ignore the offset (assuming the scene is not too close to the camera), and just transform the pixel.
p2=K2*(K1^-1 * p1)
Using this you can construct a new image that is a composite of both.
The more difficult approach would be to reconstruct the 3D structure of the scene by finding features that you can match between both images, and then triangulate the point with both rays.
Related
I have a picture with high and low contrast transitions.
I need to detect edges on the above picture. I need binary image. I can easily detect the black and "dark" blue edges with Sobel operator and thresholding.
However, the edge between "light" blue and "light" yellow color is problematic.
I start with smooth image with median filter for each channel to remove noise.
What I have tried already to detect edges:
Sobel operator
Canny operator
Laplace
grayscale, RGB, HSV, LUV color spaces (with multichannel spaces, edges are detected in each channel and then combined together to create one final edge image)
Preprocessing RGB image with gamma correction (the problem with preprocessing is the image compression. The source image is JPG and if I use preprocessing edge detection often ends with visible grid caused by JPG macroblocks.)
So far, Sobel on RGB works best but the low-contrast line is also low-contrast.
Further thresholding remove this part. I consider edge everything that is under some gray value. If I use high threshold vales like 250, the result for low contrast edge is better but the remaining edges are destroyed. Also I dont like gaps in low-contrast edge.
So, if I change the threshold further and say that all except white is edge, I have edges all over the place.
Do you have any other idea how to combine low and high contrast edge detection so that the edges are without gaps as much as possible and also not all over the place?
Note: For test I use mostly OpenCV and what is not available in OpenCV, I programm myself
IMO this is barely doable, if doable at all if you want an automated solution.
Here I used binarization in RGB space, by assigning every pixel to the closest color among two colors representative of the blue and yellow. (I picked isolated pixels, but picking an average over a region would be better.)
Maybe a k-means classifier could achieve that ?
Update:
Here is what a k-means classifier can give, with 5 classes.
All kudos and points to Yves please for coming up with a possible solution. I was having some fun playing around experimenting with this and felt like sharing some actual code, as much for my own future reference as anything. I just used ImageMagick in Terminal, but you can do the same thing in Python with Wand.
So, to get a K-means clustering segmentation with 5 colours, you can do:
magick edge.png -kmeans 5 result.png
If you want a swatch of the detected colours underneath, you can do:
magick edge.png \( +clone -kmeans 5 -unique-colors -scale "%[width]x20\!" \) -background none -smush +10 result.png
Keywords: Python, ImageMagick, wand, image processing, segmentation, k-means, clustering, swatch.
I would like to compare videos. To compare the quality (Non blurry) by coding a C program. Someone told me to learn about DFT (Discrete Fourier Transform) for image analysis and to use a FFT or DFT tool to learn the difference between blurred vs detailed (non-blurry) copies of same image.
(copied from other question):
Lets say we have different files with different video quality, one is extremely clear, other is blurred, one is having rough colors. Compare all files basically frame by frame and report to the user which has better quality.
So can anyone help me with this ??
Let's say we have various files having different video quality:
one is extremely clear, other is blurred, one is having rough colors.
Compare all files basically frame by frame and report to the user which has better quality.
(1) Color Quality detection...
To check which has better color, you analyze the histograms of the test images. The histogram will be a count of how many pixels have intensity X. Where X is a number ranging between 0 up to 255 (because each red, green and blue channels each holds any of those 256 possible intensities).
There are many tutorials online about how to create a histogram since it's a basic task in computer graphics.
Generally it goes like:
First make 3 arrays (eg: hist_Red) to hold data for red, green and blue channels.
Break up (using FOR loop) each pixel into individual R/G/B channel components:
example:
temp_Red = this_pixel >> 16 & 0x0ff;
temp_Grn = this_pixel >> 8 & 0x0ff;
temp_Blu = this_pixel >> 0 & 0x0ff;
Then add +1 to that specific red/green/blue intensity in relevant histogram.
example:
hist_Red[ temp_Red ] += 1;
hist_Grn[ temp_Grn ] += 1;
hist_Blu[ temp_Blu ] += 1;
By adding the totals of red, green and blue, you will have total intensities of RGB in an array that could build charts like below. Check with image's array has most values to find image with better quality of colors:
(2) Detailed vs Blurred detection...
You can try using a convolution filter to detect blur in image. Give the filter a kernel (eg: a matrix). The matrix (3x3) shown below gives an edge-detect filter, where blurred images give less edges (therefore gives more black pixels).
Use logic to assume that: more black pixels EQuals a more blurred image (less detail).
You can read about convolutions here
Lode's Computer Graphics Tutorial: Image Filtering
Image Convolution with C/C++ code
PDF Image Manipulation: Filters and Convolutions
PDF Read page 10 onwards : Convolution filters
I'm trying to show students how the RGB color model works to create a particular color (or moreover to convince them that it really does). So I want to take a picture and convert each pixel to an RGB representation so that when you zoom in, instead of a single colored pixel, you see the RGB colors.
I've done this but for some very obvious reasons the converted picture is either washed out or darker than the original (which is a minor inconvenience but I think it would be more powerful if I could get it to be more like the original).
Here are two pictures "zoomed out":
Here is a "medium zoom", starting to show the RGB artifacts in the converted picture:
And here is a picture zoomed in to the point that you can clearly see individual pixels and the RGB squares:
You'll notice the constant color surrounding the pixels; that is the average RGB of the picture. I put that there so that you could see individual pixels (otherwise you just see rows/columns of shades of red/green/blue). If I take that space out completely, the image is even darker and if I replace it with white, then the image looks faded (when zoomed out).
I know why displaying this way causes it to be darker: a "pure red" will come with a completely black blue and green. In a sense if I were to take a completely red picture, it would essentially be 1/3 the brightness of the original.
So my question is:
1: Are there any tools available that already do this (or something similar)?
2: Any ideas on how to get the converted image closer to the original?
For the 2nd question, I could of course just increase the brightness for each "RGB pixel" (the three horizontal stripes in each square), but by how much? I certainly can't just multiply the RGB ints by 3 (in apparent compensation for what I said above). I wonder if there is some way to adjust my background color to compensate for me? Or would it just have to be something that needs to be fiddled with for each picture?
You were correct to assume you could retain the brightness by multiplying everything by 3. There's just one small problem: the RGB values in an image use gamma correction, so the intensity is not linear. You need to de-gamma the values, multiply, then gamma correct them again.
You also need to lose the borders around each pixel. Those borders take up 7/16 of the final image which is just too much to compensate for. I tried rotating every other pixel by 90 degrees, and while it gives the result a definite zig-zag pattern it does make clear where the pixel boundaries are.
When you zoom out in an image viewer you might see the gamma problem too. Many viewers don't bother to do gamma correction when they resize. For an in-depth explanation see Gamma error in picture scaling, and use the test image supplied at the end. It might be better to forgo scaling altogether and simply step back from the monitor.
Here's some Python code and a crop from the resulting image.
from PIL import Image
im = Image.open(filename)
im2 = Image.new('RGB', (im.size[0]*3, im.size[1]*3))
ld1 = im.load()
ld2 = im2.load()
for y in range(im.size[1]):
for x in range(im.size[0]):
rgb = ld1[x,y]
rgb = [(c/255)**2.2 for c in rgb]
rgb = [min(1.0,c*3) for c in rgb]
rgb = tuple(int(255*(c**(1/2.2))) for c in rgb)
x2 = x*3
y2 = y*3
if (x+y) & 1:
for x3 in range(x2, x2+3):
ld2[x3,y2] = (rgb[0],0,0)
ld2[x3,y2+1] = (0,rgb[1],0)
ld2[x3,y2+2] = (0,0,rgb[2])
else:
for y3 in range(y2, y2+3):
ld2[x2,y3] = (rgb[0],0,0)
ld2[x2+1,y3] = (0,rgb[1],0)
ld2[x2+2,y3] = (0,0,rgb[2])
Don't waste so much time on this. You cannot make two images look the same if you have less information in one of them. You still have your computer that will subsample your image in weird ways while zooming out.
Just pass a magnifying glass through the class so they can see for themselves on their phones or other screens or show pictures of a screen in different magnification levels.
If you want to stick to software, triple the resolution of your image, don't use empty rows and columns or at least make them black to increase contrast and scale the RGB components to full range.
Why don't you keep the magnified image for the background ? This will let the two images look identical when zoomed out, while the RGB strips will remain clearly visible in the zoom-in.
If not, use the average color over the whole image to keep a similar intensity, but the washing effect will remain.
An intermediate option is to apply a strong lowpass filter on the image to smoothen all details and use that as the background, but I don't see a real advantage over the first approach.
Light Field captures the scene from slightly different points. This means I would have two images of the same scene with a slight shift, as shown in the following figure:
Assuming the red squares in the images above are pixels. I know that the spatial difference between those two pixels is a shift. Nevertheless, what other information do these two pixels give us in terms of scene radiance? I mean is there a way to find (or compute) the difference in image irradiance values between those two points?
Look for color space representations other than RGB. Some of them have explicit channel(s) carrying luminance information of a pixel.
A varaiant of the same idea is to convert to a Black and White image and examine the pixel values.
I have a customized camera, which contains 3 individual lens+filters arranged in a triangle so in every shot I get 3 single band grayscale images (r, g, b). I want to merge them to get an RGB.
The problem is, since the 3 lens are physically separated, the image captured by them are not aligned. As a result, when I use command qdal_merge in the software pack QGIS, the result looks weird. I may also need to adjust the weight of the r,g,b. I put the raw r,g,b images and the output I generated using qgis in this dropbox folder.
Is there existing open-source tool to do the alignment and merge? If not, how can I do it using opencv?
Combining R,G,B images is possible using a simple pixel intensity distance metric like Sum of Squared Distances (SSD). A better metric is the Normalized Cross-Correlation (NCC) (see Wikipedia) which first normalizes an image matrix into a unit vector, and computes the dot product of such unit vectors (from 2 input images). The higher the NCC value, the greater the similarity of the two input images.
However, NCC similarity may be insufficient for computing the best alignment of two high resolution images, such as the TIFF images you provide. One should therefore use a downsampling method as described below
to align two input images at a smaller size and then simply compute the offset as you rescale the images.
So for the input images, red, green and blue, there are two approaches to align them into a single RGB image:
Consider the blue image as the reference image for example, w.r.t. which we align the red and green images. Now consider red and blue images. Within a certain window, compute the best alignment offset of the red and blue images using the NCC similarity metric, and find the shifted_red image. Do the same for the green and blue images. Now combine the shifted_red, shifted_green and blue images to get the final RGB image.
For high-resolution images, decide a scale_count. Recursively, at each step resize the image by half, compute the offset of the red image w.r.t. the blue image, rescale the offset and apply it. The benefit of doing such a recursive multi-scale alignment is decrease in computation time and increase in accuracy of alignment (you don't know the best window size for searching for alignment offsets for solution (1), so this will work better). Repeat this approach for computing the alignment for green and blue channels, and then combine the final results as in (1).
Since this problem is common in assignments of computational photography courses, I am not going to share any code. I have, however implemented the two approaches and experimented with the images you provide. I don't know which of the input images is red, so I have two results (rescaled to decrease file size):
If IMG_0290_1.tif is Red, IMG_0290_2.tif is Green and IMG_0290_3.tif is blue:
RGB result if red:1, green:2, blue:3
If IMG_0290_3.tif is Red, IMG_0290_2.tif is Green and IMG_0290_1.tif is blue (this looks more correct to me):
RGB result if red:3, green:2, blue:1