Recently I came across "Parameter" layer in caffe.
It seems like this layer exposes its internal parameter blob to "top".
What is this layer using for?
Can you give a usage example?
This layer was introduced in the pull request #2079, with the following description:
This layer simply holds a parameter blob of user-defined shape, and shares it as its single top.
which is exactly what you expected. This was introduced in context of the issue #1474, which basically proposes to treat parameters like normal bottom blobs. To show why this can be useful, consider the following example (taken from issue #1474, by #longjon):
The inner product layer calculates C = A * B, where C is the top blob (output), B is a bottom blob, and A must be a parameter blob. This is very restrictive, as this makes it impossible to use the inner product layer for multiplying the inner product between two bottom blobs, e.g. multiplying two input matrices. The issue #1474 suggests to make a fundamental change: make parameters independent of the layer. Instead, treat them like a normal bottom blob.
As a first step in that direction, the Parameter layer was introduced. This allows you to define a new parameter, which you can then feed into the bottom of another layer.
The counterpart - a method for feeding parameters into a layer as a bottom blob - is proposed in pull request #2166, which isn't merged yet, as of Jan. 2017.
Though this is not merged yet, you can still use Parameter layer to define new learnable parameters to feed into other layers as bottom blob.
Related
Is it possible to place a port on a layer so that it is invisible when the layer is not visible?
I could not find an answer in GOJS documentation. I tested with this link template and it did not work.
myDiagram.linkTemplate =
$(CustomLink, // defined below
{ layerName: "blue",...
You can only put Parts in Layers -- either Nodes or Links or Adornments. In other words, you cannot split up a Node so that a piece of it appears in one layer and another piece of the same Node appears in a different layer.
The normal thing to do is to show or hide pieces such as ports, either by changing their visible property or their opacity property. The former causes the node's panels to be remeasured and rearranged; the latter does not. Several samples demonstrate this, including the Flow Chart sample.
So I have this line of code to load a dataset of images from two classes called "0" and "1" for simplicity:
train_data = torchvision.datasets.ImageFolder(os.path.join(TRAIN_DATA_DIR), train_transform)
and then I prepare the loader to be used with my model in this way:
train_loader = torch.utils.data.DataLoader(train_data, TRAIN_BATCH_SIZE, shuffle=True)
So for now each image is associated to a class, what I want to do is take each image and apply a transformation to it between those two lines of code, let's say a rotation of one of four degrees: 0, 90, 180, 270, and add that info as an additional label of four classes: 0, 1, 2, 3. In the end I want the dataset to contain the rotated images and as their labels a list of two values: the class of the image and the applied rotation.
I tried this and there is no error, but the dataset remains unchanged if then I try to print the labels:
for idx,label in enumerate(train_data.targets):
train_data.targets[idx] = [label, 1]
Is there a nice way to do it by modifying directly train_data without requiring a custom dataset?
Is there a nice way to do it by modifying directly train_data without requiring a custom dataset?
No, there isn't. If you want to use datasets.ImageFolder, you have to accept its limited flexibility. In fact, ImageFolder is simply a subclass of DatasetFolder, which is pretty much what a custom dataset is. You can see in its source code the following section of __getItem__:
if self.transform is not None:
sample = self.transform(sample)
if self.target_transform is not None:
target = self.target_transform(target)
This makes what you want impossible since your expected transform should modify both the image and the target at the same time, which is done independently here.
So, start by making your subclass of Dataset similar to DatasetFolder, and simply implement your own transform which takes in an image and a target at the same time and returns their transformed values. This is just an example of a transform class you could have, which would then need to be composed into a single function call:
class RotateTransform(object):
def __call__(self, image, target):
# Rotate the image randomly and adjust the target accordingly
#...
return image, target
If that's too much trouble for your case, then the best option you have is what #jchaykow mentionned, which is to simply modify your files prior to running your code.
Is there an optimal layout in JUNG for left-to-right placement of nodes in the graph? I'm looking for something that would model a data lineage and need to represent this somewhat linearly (left to right).
If your graph is a tree, then you can use the TreeLayout and do some post-processing to rotate it to have the root on the left instead of the top (see the L2RTreeLayoutDemo for an example of how to do this).
If your graph is only sort of tree-shaped, then you may want to extract a tree from your graph first (see MinimumSpanningTreeDemo for one way to do this automatically) and then follow the procedure above.
If neither of those works for you (for example, this won't work if your graph has multiple "roots"), then you will probably need to create your own layout, perhaps based on TreeLayout.
Has anyone been able to do spatial operations with #ApacheSpark? e.g. intersection of two sets that contain line segments?
I would like to intersect two sets of lines.
Here is a 1-dimensional example:
The two sets are:
A = {(1,4), (5,9), (10,17),(18,20)}
B = {(2,5), (6,9), (10,15),(16,20)}
The result intersection would be:
intersection(A,B) = {(1,1), (2,4), (5,5), (6,9), (10,15), (16,17), (18,20)}
A few more details:
- sets have ~3 million items
- the lines in a set cover the entire range
Thanks.
One approach to parallelize this would be to create a grid of some size, and group line segments by the grids they belong to.
So for a grid with sizes n, you could flatMap pairs of coordinates (segments of line segments), to create (gridId, ( (x,y), (x,y) )) key-value pairs.
The segment (1,3), (5,9) would be mapped to ( (1,1), ((1,3),(5,9) ) for a grid size 10 - that line segment only exists in grid "slot" 1,1 (the grid from 0-10,0-10). If you chose a smaller grid size, the line segment would be flatmapped to multiple key-value pairs, one for each grid-slot it belongs to.
Having done that, you can groupByKey, and for each group, calculation intersections as normal.
It wouldn't exactly be the most efficient way of doing things, especially if you've got long line segments spanning multiple grid "slots", but it's a simple way of splitting the problem into subproblems that'll fit in memory.
You could solve this with a full cartesian join of the two RDDs, but this would become incredibly slow at large scale. If your problem is smallish, sure, this is an easy and cheap approach. Just emit the overlap, if any, between every pair in the join.
To do better, I imagine that you can solve this by sorting the sets by start point, and then walking through both at the same time, matching one's current interval versus another and emitting overlaps. Details left to the reader.
You can almost solve this by first mapping each tuple (x,y) in A to something like ((x,y),'A') or something, and the same for B, and then taking the union and sortBy the x values. Then you can mapPartitions to encounter a stream of labeled segments and implement your algorithm.
This doesn't quite work though since you would miss overlaps between values at the ends of partitions. I can't think of a good simple way to take care of that off the top of my head.
I've ran in to an issue concerning generating floating point coordinates from an image.
The original problem is as follows:
the input image is handwritten text. From this I want to generate a set of points (just x,y coordinates) that make up the individual characters.
At first I used findContours in order to generate the points. Since this finds the edges of the characters it first needs to be ran through a thinning algorithm, since I'm not interested in the shape of the characters, only the lines or as in this case, points.
Input:
thinning:
So, I run my input through the thinning algorithm and all is fine, output looks good. Running findContours on this however does not work out so good, it skips a lot of stuff and I end up with something unusable.
The second idea was to generate bounding boxes (with findContours), use these bounding boxes to grab the characters from the thinning process and grab all none-white pixel indices as "points" and offset them by the bounding box position. This generates even worse output, and seems like a bad method.
Horrible code for this:
Mat temp = new Mat(edges, bb);
byte roi_buff[] = new byte[(int) (temp.total() * temp.channels())];
temp.get(0, 0, roi_buff);
int COLS = temp.cols();
List<Point> preArrayList = new ArrayList<Point>();
for(int i = 0; i < roi_buff.length; i++)
{
if(roi_buff[i] != 0)
{
Point tempP = bb.tl();
tempP.x += i%COLS;
tempP.y += i/COLS;
preArrayList.add(tempP);
}
}
Is there any alternatives or am I overlooking something?
UPDATE:
I overlooked the fact that I need the points (pixels) to be ordered. In the method above I simply do scanline approach to grabbing all the pixels. If you look at the 'o' for example, it would grab first the point on the left hand side, then the one on the right hand side. I would need them to be ordered by their neighbouring pixels since I want to draw paths with the points later on (outside of opencv).
Is this possible?
You should look into implementing your own connected components labelling. The concept is very simple: you scan the first line and assign unique labels to each horizontally connected strip of pixels. You basically check for every pixel if it is connected to its left neighbour and assign it either that neighbour's label or a new label. In the second row you do the same, but you also check against the pixels above it. Sometimes you need a label merge: two strips that were not connected in the previous row are joined in the current row. The way to deal with this is either to keep a list of label equivalences or use pointers to labels (so you can easily do a complete label change for an object).
This is basically what findContours does, but if you implement it yourself you have the freedom to go for 8-connectedness and even bridge a single-pixel or two-pixel gap. That way you get "almost-connected components labelling". It looks like you need this for the "w" in your example picture.
Once you have the image labelled this way, you can push all the pixels of a single label to a vector, and order them something like this. Find the top left pixel, push it to a new vector and erase it from the original vector. Now find the pixel in the original vector closest to it, push it to the new vector and erase from the original. Continue until all pixels have been transferred.
It will not be very fast this way, but it should be a start.