I was searching on Google and StackOverflow to see if anyone have solution for my problem, but didn't found anyone with same problems.
So, currently I'm running Debian machine with Mapserver installed on it. The server also run webserver for displaying map data over the browser. The generation of map is dynamic, based on layers definition in database I built mapfile in PHP and based on that generated PHP the map is shown to user. The data is defined in database and as a SHP files (both combined in single mapfile).
It is fully dynamic, what I mean with that is that user can enable/disable any of layers or click inside polygon (select some points on map) it color the selection (generate new mapfile based on selection and re-generate tiles).
So the execution of all that code from selecting some area to coloring selected items somtimes take too much time for good user experience.
For solution I'd like to use some kind of temporary tiles cache, that can be used for single user, and to be able to delete it's content when user select some items on map or enable/disable one of the layers.
P.S. I already did all the optimizations provided from Mapserver documentation.
Thanks for any help.
It sounds to me like your problem is not going to be helped by server-side caching. If all of the tiles depend on user selections, then you're going to be generating a bunch of new tiles every time there's an interaction.
I've been using MapCache to solve a similar problem, where I am rendering a tileset in response to a user query. But I've broken up my tiles into multiple logical layers, and I do the compositing on the browser side. This lets me cache, server-side, the tiles for various queries, and sped up performance immensely. I did seed the cache down to zoom level 12, and I needed to use the BerkeleyDB cache type to keep from running out of inodes.
I'm using Leaflet.js for the browser-side rendering, but you should also consider OpenLayers.
After looking at the source code, I have some other ideas.
It looks like you're drawing each layer the same way each time. Is that right? That is, the style and predicate of a particular layer never change. Each user sees the image for that layer the same way, if they have selected the layer. But the combination of layers you show does change, based on OpenLayers control? If that's the case, you don't need per-user caching on the server. Instead, use per-layer caching, and let the user's browser figure out the client side caching.
A quick technique for finding slow layers is to turn them all of. Then reenable them one by one to find the culprit. Invoke Mapserver from the command line, and time the runs, for greater precision than you'll get by running it from your webserver.
You mentioned you're serving the images in Google 3857 while the layers are in Gauss-Kruger/EPSG 3912. Reprojecting this on the fly is expensive. Reprojecting the rasters on the fly is very expensive. If you can, you should reproject them ahead of time, and store them in 3857 (add an additional geometry column).
I don't know what a DOF file is--maybe Digital Obstacle File? Perhaps preload the DOF file into PostGIS too? That would eliminate the two pieces you think are problematic.
Take a look at the SQL queries that PostGIS is performing, and make sure those are using indexes
In any case, these individual layers should go into MapCache, in my opinion. Here is a video of a September 2014 talk by the MapCache project leader.
Related
I have about 2,000 images of cars, most pointing right, but some pointing left.
I'd like to find a way of automatically tagging a car with it's direction (new images will be coming in continually).
I'm struggling to get started and wondered if this kind of image detection problem has a name that may help my searches. Is object orientation detection a thing?
I'm a software developer (not doing much ML or Image stuff) and have a ton of azure and gcc resources available, but I can't find anything to solve this. Azure Cognitive Service can tell us it's a car in the picture, but doesn't tell us the direction.
Could just do with a good starting point to get going.
Should add, the images are quite clean on white backgrounds, examples:
Thanks to Venkata for commenting, it was a bad dataset causing our issues (too many rights vs left).
Here's what we did to get it all working:
We set up a training and prediction instance in azure (using custom vision cognitive services in our portal).
We then used https://www.customvision.ai/ to set everything up and train the model (it's super simple).
We didn't actually need any left facing images in the end, we just took all the right facing images we had (about 500 in the final instance), we uploaded them all with the tag "Right". We then mirrored all the images with a photoshop script and then uploaded them all again with a "Left" tag. It trained for about 15 minutes and we ended up with a 100% prediction score. We tested it with a load of images that weren't contained in the training set to confirm it was all working.
We then did the same for a ton of van/truck images, these were taken from a different angle (cars were all side profile shots, the vans were all front 3 quarter so we weren't sure if we'd have the same success).
Again, we flipped the images ourselves to create the left images so we only needed to source right facing vans to create the whole model.
We ended up with a 99.8% score, which is totally acceptable for our use case and we can now detect all cars and van directions and it even detects car directions that are front 3 quarters and vans that are in profile (even though we only trained cars in profile and vans in 3 quarter).
The custom vision portal gives you an API endpoint and a key, now when we detect a new image in our system it goes via the API (using the custom image sdk/nuget in our .net site) and we check the tags to see if it needs flipping. If it does, we flip it and save it back to the disk and it's then cached so it doesn't keep hitting the API.
It's pretty amazing, it took us just two days to research the options, pick a provider and then implement the solution in to a production platform. It's probably a simple use case for ML, but 10 years ago (or even 5) we couldn't have dreamed that things would have come along so far.
tldr; If you need to detect if an object in an image is pointing left or right, just grab a lot of right facing examples and then flip them yourself to create a well balanced model. Obviously, this relies on the object looking the same from one side to the other.
First post on SO; hopefully I am doing it right :-)
Have a situation where users need to upload and view very high resolution files (they need to pan, tilt, zoom, and annotate images). A single file sometimes crosses 1 GB so loading complete file on client side is not an option.
We are thinking about letting the users upload files to the server (like everyone does), then apply some encryption on server side creating multiple, relatively small low resolution images with varying sizes. We then give users thumbnails with canvas size option on the webpage for them to pick and start their work.
Lets assume a user opens low grade image with 1280 x 1028 canvas size. Image will be broken into tiles before display, and when user clicks on a title it will be like zooming in to a specific tile. Client will send request to the server asking for higher resolution image for the title. Server will send the image which will be broken into titles again for the user to click and get another higher resolution image from server and so on ... Having multiple images with varying resolution will help us break images into tiles and serve user needs ('keep zooming in' or out using tiles).
Has anyone dealt with humongous image files? Is there a preferred technical design you can suggest? How to handle areas that have been split across tiles is bothering me a lot so not sure how above approach can be modified to address this issue.
We need to plan for 100 to 200 users connected to the website simultaneously, and ours is .NET environment if it matters
Thanks!
The question is a little vague. I assume you are looking for hints, so here are a few:
I see uploading the images is a problem in the firstplace. Where I come from, upload-speeds are way slower than download speeds. (But there is litte you can do if you need your user to upload gigabytes...) Perhaps offer some more stable upload than web. FTP if you must.
Converting in smaller pieces should be no big problem. Use one of the availabe tools. Perhaps imagemagick. I see there is a .net wrapper out: https://magick.codeplex.com/
More than converting alone I think it is important not to do it everytime on the fly (you would need a realy big machine) but only once the image is uploaded. If you want to scale you can outsource this to another box in the network.
For the viewer. This is the interessting part. There are some ready to use ones. Google has one. It's called 'Maps' :). But there is a free alternative: OpenLayers from the OpenStreetmap Project: http://wiki.openstreetmap.org/wiki/OpenLayers All you have to do is naming your generated files in the right way and a litte configuration.
Even if you must for some reasons create the tiles on the fly or can't use something like OpenLayers I would try to stick to its naming scheme. Having something working to start with is never a bad idea.
I'm working with a project its related to Offline map application.Because of that I searched for offline map which shows the defined area. I used MapBox for offline mapping. I can add annotation on this map and draw lines.
But my requirement is offline map with routing. I was fed up to find a offline routing library or offline routing engine to embedded to Xcode.
Appreciate if any of you have any clue or sample project/code to implement this
Note : This question is related to my one. No one replied to this as well
Thanks.
Offline implies no internet, the iPhone is still able in most cases to get the users current location from the GPS. That means that you can be quite confidant that you can find out the current location of the user whilst offline.
The problem with offline routing is that the Phone is dumb, it only remembers the x amount of MB of data in terms of tiles to display.
Routing is something completely different, it takes a point A and B and works out the shortest, fastest, cheapest or all of those between A and B.
This takes a lot more then tiles to accomplish, after all if you think in terms of MVC, tiles are just the dump views, they don't know much about what's around them except what's inside of them. It would be the "controller" who would calculate routes, and for that you would need to be in possession of all the data spanning the desired area for routing.
For each mapping service you will find a different route, maybe not in terms of actual path, but in estimated time, effort etc, what this means is that if you have your own maps (offline in a database), it's up to you to use that data, so you should make your own routing algorithm which I'm sure isn't what you want to do.
So what are your options? At the moment this just isn't possible in the scope you want. Even if you had an offline maps database, you still need a routing algorithm.
In offline case also you can get the current location by using only GPS and you can draw overlay lines from current location to the interesting point for that you have to do some calculations
You can make offline routing by using graphhopper library by making graph data which contains (Street names, routes,edges) . Graph data is taken by .pbf file which can be taken by (Use this:http://download.geofabrik.de) and use commands(in Terminal) that was given by (https://github.com/graphhopper/graphhopper-ios/tree/master/graphhopper-ios-sample) to convert .pbf into graph data. Then we can make offline routing with its instruction (All given in graphhopper iOS sample).please refer that carefully. because i have done and finish my project successfully.
I am working on a project in which i have a database created using mysql and php. I want to advance it by connecting an image to it. every object in my database is going to be linked to a certain part of the image that will be highlighted.
I am not really sure how to do this.
As of right now, i have a search feature on my database that gives me the result.
I am willing to try any programming language that would make this easier.
My database has about maybe 1000 entries. They are all numbers that correspond to a certain location on my image.
My image is basically a map.
I'm not entirely sure what you are trying to do, but if you want to plot a graph you can use the HTML 5 canvas as some JavaScript. Or better yet, use tool that already exists like Google Charts. Maybe a scatter plot is what you're looking for?
As the question states: how is it possible to process some dynamic videostream? By saying dynamic, i actually mean I would like to just process stuff on my screen. So the imagearray should be some sort of "continuous screenshot".
I'd like to process the video / images based on certain patterns. How would I go about this?
It would be perfect if there already was (and there probably is) existing components. I need to be able to use the location of the matches (or partial matches). A .NET component for the different requirements could also be useful I guess...
You will probably need to read up on Computer Visual before you attempt this. There is nothing really special about video that seperates it from still imgaes. The process you might want to look at is:
Acquire the data
Split the data into individual frames
Remove noise (Use a Gaussian filter)
Segment the image into the sections you want
Remove the connected components of the image
Find a way to quantize the image for comparison
Store/match the components to a database of previously found components
With this database/datastore you'll have information on matches later in the database. Do what you like with it.
As far as software goes:
Most of these algorithms are not too difficult. You can write them yourself. They do take a bit of work though.
OpenCV does a lot of the basic stuff, but it won't do everything for you
Java: JAI, JHLabs [for filters], Various other 3rd party libraries
C#: AForge.net