I'm looking at the example: https://github.com/fchollet/keras/blob/master/examples/conv_lstm.py
This RNN is actually predicting the next frame of the movie, so the output should be a movie too (according to the test data fed in). I wonder if there are information lost due to the conv layers with padding.
For example, the underlying Tensorflow is padding bottom right, if there is a big padding: (n stands for numbers)
n n n n 0 0 0
n n n n 0 0 0
n n n n 0 0 0
n n n n 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
when we do the second conv, the bottom right corner will always be 0, which means the back propagation will never be able to capture anything there. As in this case a movie(a square moves on the whole screen), will it lost the information when the validation label is on the bottom right corner?
The answer is yes after asking a Ph.D. doing AI research.
Related
I've a binary image where removing green dot gets me separate line segments. I've tried using label_components() function from Julia but it labels only verticall joined pixels as one label.
I'm using
using Images
img=load("current_img.jpg")
img[findall(img.==RGB(0.0,0.1,0.0))].=0 # this makes green pixels same as background, i.e. black
labels = label_components(img)
I'm expecteing all lines which are disjoint to be given a unique label
(as was a funciton in connected component labeling in matlab, but i can't find something similar in julia)
Since you updated the question and added more details to make it clear, I decided to post the answer. Note that this answer utilizes some of the functions that I wrote here; so, if you didn't find documentation for any of the following functions, I refer you to the previous answer. I operated on several examples and brought the results in the continue.
Let's begin with an image similar to the one you brought in the question and perform the entire operation from the scratch. for this, I drew the following:
I want to perform a segmentation process on it and labelize each segment and highlight the segments using the achieved labels.
Let's define the functions:
using Images
using ImageBinarization
function check_adjacent(
loc::CartesianIndex{2},
all_locs::Vector{CartesianIndex{2}}
)
conditions = [
loc - CartesianIndex(0,1) ∈ all_locs,
loc + CartesianIndex(0,1) ∈ all_locs,
loc - CartesianIndex(1,0) ∈ all_locs,
loc + CartesianIndex(1,0) ∈ all_locs,
loc - CartesianIndex(1,1) ∈ all_locs,
loc + CartesianIndex(1,1) ∈ all_locs,
loc - CartesianIndex(1,-1) ∈ all_locs,
loc + CartesianIndex(1,-1) ∈ all_locs
]
return sum(conditions)
end;
function find_the_contour_branches(img::BitMatrix)
img_matrix = convert(Array{Float64}, img)
not_black = findall(!=(0.0), img_matrix)
contours_branches = Vector{CartesianIndex{2}}()
for nb∈not_black
t = check_adjacent(nb, not_black)
(t==1 || t==3) && push!(contours_branches, nb)
end
return contours_branches
end;
"""
HighlightSegments(img::BitMatrix, labels::Matrix{Int64})
Highlight the segments of the image with random colors.
# Arguments
- `img::BitMatrix`: The image to be highlighted.
- `labels::Matrix{Int64}`: The labels of each segment.
# Returns
- `img_matrix::Matrix{RGB}`: A matrix of RGB values.
"""
function HighlightSegments(img::BitMatrix, labels::Matrix{Int64})
colors = [
# Create Random Colors for each label
RGB(rand(), rand(), rand()) for label in 1:maximum(labels)
]
img_matrix = convert(Matrix{RGB}, img)
for seg∈1:maximum(labels)
img_matrix[labels .== seg] .= colors[seg]
end
return img_matrix
end;
"""
find_labels(img_path::String)
Assign a label for each segment.
# Arguments
- `img_path::String`: The path of the image.
# Returns
- `thinned::BitMatrix`: BitMatrix of the thinned image.
- `labels::Matrix{Int64}`: A matrix that contains the labels of each segment.
- `highlighted::Matrix{RGB}`: A matrix of RGB values.
"""
function find_labels(img_path::String)
img::Matrix{RGB} = load(img_path)
gimg = Gray.(img)
bin::BitMatrix = binarize(gimg, UnimodalRosin()) .> 0.5
thinned = thinning(bin)
contours = find_the_contour_branches(thinned)
thinned[contours] .= 0
labels = label_components(thinned, trues(3,3))
highlighted = HighlightSegments(thinned, labels)
return thinned, labels, highlighted
end;
The main function in the above is find_labels which returns
The thinned matrix.
The labels of each segment.
The highlighted image (Matrix, actually).
First, I load the image, and binarize the Gray scaled image. Then, I perform the thinning operation on the binarized image. After that, I find the contours and the branches using the find_the_contour_branches function. Then, I turn the color of contours and branches to black in the thinned image; this gives me neat segments. After that, I labelize the segments using the label_components function. Finally, I highlight the segments using the HighlightSegments function for the sake of visualization (this is the bonus :)).
Let's try it on the image I drew above:
result = find_labels("nU3LE.png")
# you can get the labels Matrix using `result[2]`
# and the highlighted image using `result[3]`
# Also, it's possible to save the highlighted image using:
save("nU3LE_highlighted.png", result[3])
The result is as follows:
Also, I performed the same thing on another image:
julia> result = find_labels("circle.png")
julia> result[2]
14×16 Matrix{Int64}:
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0
0 1 1 0 0 0 3 3 0 0 0 5 5 5 0 0
0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
As you can see, the labels are pretty clear. Now let's see the results of performing the procedure in some examples in one glance:
Original Image
Labeled Image
I'm developing a PdfParser and I want to print the text content of the pdf on a coordinate plane. Below is the text object and matrices that are used to render text. How can I isolate the scaling, rotation and translation and use for printing the text content on exact coordinates on a canvas?
//Decoded text stream containing text objects
S
Q
q
0.000 0.750 0.750 -0.000 15.000 301.890 cm
0.000 g
/F10 16.000 Tf
0 Tr
0.000 Tc
BT
1 0 0 -1 20.000 13.600 Tm
[<007a>]TJ
ET
Q
q
0.000 0.750 0.750 -0.000 15.000 301.890 cm
1.000 0.416 0.000 rg
/F10 6.667 Tf
0 Tr
0.000 Tc
BT
1 0 0 -1 136.667 13.600 Tm
[<0024>12<0046><0046><0058><0055>6<0048><0003><0032><0058><0057><0053><0058><0057><0003><0036>-4<0052><004f><0058><0057><004c><0052><0051><0003><0026>3<004f><0052><0058><0047><0003><0048><0051>18<0059><004c><0055>6<0052><0051><0050><0048><0051>3<0057>7<000f><0003><0027><0028><0030><0032><0003><0044><0046><0046><0058><0055>6<0048>]TJ
ET
Q
q
0.000 0.750 0.750 -0.000 15.000 301.890 cm
0.000 g
/F10 16.000 Tf
0 Tr
0.000 Tc
BT
1 0 0 -1 603.333 13.600 Tm
[<007a>]TJ
ET
Q
q
The initial S Q is a leftover of a previous instruction block ending in some path stroking and graphics state restoring. As we don't know anything to the contrary, let's assume that 'Q' restores to the initial graphics state, in particular to an unmodified current transformation matrix (CTM).
As we are interested in coordinates according to the default user space coordinate system, we can assume accordingly that the current CTM is the identity matrix,
Let's take a look at the block
q
0.000 0.750 0.750 -0.000 15.000 301.890 cm
0.000 g
/F10 16.000 Tf
0 Tr
0.000 Tc
BT
1 0 0 -1 20.000 13.600 Tm
[<007a>]TJ
ET
Q
As you implied yourself in a comment, the only relevant instructions for the total transformation matrix at the time the text rendering instruction [<007a>]TJ begins to be executed are
0.000 0.750 0.750 -0.000 15.000 301.890 cm
and
1 0 0 -1 20.000 13.600 Tm
setting the current transformation matrix to
0 0.75 0 1 0 0 0 0.75 0
0.75 0 0 * 0 1 0 = 0.75 0 0
15.00 301.89 1 0 0 1 15.00 301.89 1
and the text and text line matrices both to
1 0 0
0 -1 0
20.0 13.6 1
Thus, the effects of text matrix and current transformation matrix combine to:
1 0 0 0 0.75 0 0 0.75 0
0 -1 0 * 0.75 0 0 = -0.75 0 0
20.0 13.6 1 15.00 301.89 1 25.2 316.89 1
You can split up that combined matrix in a scaling, rotation, and translation like this:
0 0.75 0 0.75 0 0 0 1 0 1 0 0
-0.75 0 0 = 0 0.75 0 * -1 0 0 * 0 1 0
25.2 316.89 1 0 0 1 0 0 1 25.2 316.89 1
We have a scaling by .75, a rotation by 90° counterclockwise, and a translation by (25.2, 316.89).
(Of course this can still be subject to a page rotation...)
Recently I study the image processing.
When I go through the problem of filling the hole, it confuses me (I assume that the people able to answer the question is familiar with the step of doing this so I skip to the problem):
Let's say if I have a binary image like this:
0 0 0 0 0 0 0
0 0 1 1 0 0 0
0 1 0 0 1 0 0
0 1 0 0 1 0 0
0 0 1 0 1 0 0
0 0 1 0 1 0 0
0 1 0 0 0 1 0
0 1 0 0 0 1 0
0 1 1 1 1 0 0
0 0 0 0 0 0 0
And the book says to start form the region that is inside of the hole and perform the dilation operation and set the bound in case it fills the whole image.
I have no problem understanding the whole process, but if I try to code it, how can I only deal with a specific region (in the hole for this case)? Or the actual implement would be different method ?
If you can assume that the object with holes does not touch the border of the image, you can create an intermediate image where you call flood fill (with value e.g. 2) on the top left pixel. Any remaining '0' pixels have to be inside the contour. Take the position of the first encountered remaining '0' pixel and flood fill it in the original image.
I'm implementing Convolutions using Radix-2 Cooley-Tukey FFT/FFT-inverse, and my output is correct but shifted upon completion.
My solution is to zero-pad both input size and kernel size to 2^m for smallest possible m, tranforming both input and kernel using FFT, then multiply the two element-wise and transform the result back using FFT-inverse.
As an example on the resulting problem:
0 1 2 3 0 0 0 0
4 5 6 7 0 0 0 0
8 9 10 11 0 0 0 0
12 13 14 15 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
with identity kernel
0 0 0 0
0 1 0 0
0 0 0 0
0 0 0 0
becomes
0 0 0 0 0 0 0 0
0 0 1 2 3 0 0 0
0 4 5 6 7 0 0 0
0 8 9 10 11 0 0 0
0 12 13 14 15 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
It seems any sizes of inputs and kernels produces the same shift (1 row and 1 col), but I could be wrong. I've performed the same computations using the online calculator at this link! and get same results, so it's probably me missing some fundamental knowledge. My available litterature has not helped. So my question, why does this happen?
So I ended up finding the answer why this happens myself. The answered is given through the definition of the convolution and the indexing that happens there. So by definition the convolution of s and k is given by
(s*k)(x) = sum(s(k)k(x-k),k=-inf,inf)
The center of the kernel is not "known" by this formula, and thus an abstraction we make. Define c as the center of the convolution. When x-k = c in the sum, s(k) is s(x-c). So the sum containing the interesting product s(x-c)k(c) ends up at index x. In other words, shifted to the right by c.
FFT fast convolution does a circular convolution. If you zero pad so that both the data and kernel are circularly centered around (0,0) in the same size NxN arrays, the result will also stay centered. Otherwise any offsets will add.
I am trying to do image compression using DCT (Discrete Cosine Transform). Can someone please help me understand how masking affects bit per pixel in DCT? How is the bit allocation done in the masking?
PS: By masking, I mean multiplying the DCT coefficients with a matrix like the one below (element wise multiplication, not matrix multiplication).
mask = [1 1 1 1 0 0 0 0
1 1 1 0 0 0 0 0
1 1 0 0 0 0 0 0
1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0]
Background on "Masking"
Compression using the DCT calculates the DCT of blocks of an image, in this case 8x8 pixels. High frequency components of an image are less important for the human perception of an image and thus can be discarded in order to save space.
The mask Matrix selects which DCT coefficients are kept and which ones get discarded in order to save space. Coefficients towards the top left corner represent low frequencies.
For more information visit Discrete Cosine Transform.
This looks like a variation of quantization matrix.
Low frequencies are in top left, high frequencies are in bottom right. Eye is more sensitive to low frequencies, so removal of high frequency coefficients will remove less important details of the image.