Our company has a lot of data that are issue which are stored in a database.We want to create a search engine so that people can check how the issues were previously dealt with.We cannot use any 3rd party api as there is sensitive data an we want to keep it as in house. Right now the approach is as following :-
Clean up the data and then use a DOC2VEC to represent each issue as a vector .
Find the closest 5 issue using some distance metric.
The problem is that the results are not at all useful.The problem is most of the data is one liner and some issue description.There are spelling mistakes and stack traces and other things.
Is this the right approch or should we switch to something else?
Right now we are testing on 200K data.
Thanks for the help.
I'm drawing a map of the US with D3. Problem is the design on the project has changed and I need to draw regions as a whole instead of the individual states. I can't for the life of me find any good info on how to combine state features into a single region feature.
I'm assuming the gdal is the right tool for the job, but I can't seem to find how to do this.
The tutorial Mike T posted is a great resource, but it didn't solve the specific problem I had.
What I finally discovered is that when you're using topojson there is a topojson.merge method to combine multiple paths into a single path.
Follow this tutorial to convert a Shapefile into GeoJSON, then to TopoJSON, then to a D3 project. There is even an example of US (counties?) included with the project.
I'm working on a project for college and I'm having great difficulty with part of it.
Simply put, I am looking to do the following 5 things:
download the open street map data for my city
store that data locally on the phone's harddrive.
view that data in my iOS application as a map
place markers on the map.
draw paths along roads between those paths.
I have been working on this particular part of the project for a number of weeks and I'm getting nowhere with it. I haven't even been able to figure out how to store the map on the phone let alone view the map data. I've tried using the "Route-Me" library but cannot get it working (although it seems to be one of the best libraries for using openstreetmap data so I am looking to learn how to use it). I feel pretty goddamned defeated.
If anyone has accomplished any of the tasks I am trying to do could you please link me to tutorials/guides/videos that you have used.
I'm not looking for people to give me code or do the work for me, I want to learn how to do this, but if anyone can point me in the right direction of sites that I could learn off I would be very grateful.
Any advice or feedback would be much appreciated
Here's how I ended up solving the problem.
Since Tilemill doesn't natively read .osm/.o5m/.pbf files I used Osmosis to convert a .osm file into .shp files.
I then created a new project in Tilemill and added the particular .shp files I wanted as layers to the new project. It takes a little bit of tinkering to get the map to look like you want it to but it's very similar to css and pretty easy to pick up as you go.
Once I had the map looking the way I wanted it I exported it as a .mbtiles file. This takes a long time to make and the files can be very large depending on how detailed the tiles are. I did one map of Ireland with zoom levels between 7-14 inclusive and I did one map of just Dublin city with zoom levels of 11-17 inclusive. Even though the map of just the city of Dublin had much less tiles, they were both ~200MB in size.
I then found this tutorial online which explains how to store the .mbtiles file in you application and how to read it: http://martinsikora.com/creating-mbtiles-db-for-ios-mapbox-from-hi-res-map-image
Here are a few other links that I found useful:
http://www.kindle-maps.com/blog/using-tilemill-with-openstreetmap-data.html
http://mapbox.com/developers/mbtiles/
http://mapbox.com/mapbox-ios-sdk/api/
http://mapbox.com/developers/api/#static_api
http://support.mapbox.com/discussions
I hope this is useful to someone
I would suggest trying the MapBox iOS SDK. It is actually forked from the Route-Me library and will allow you to accomplish everything on your list.
A key point to remember is that you have another step in between downloading the OSM data and storing it locally on the iOS device, that is, generating the map tiles and storing them in some sort of database.
Here is an example iOS app using the MapBox SDK that has both online and offline map sources and is a good place to start.
I have a list of 200 some addresses. I used geocommons to convert these addresses into long/lat coordinates in kml and gpx format. I need to generate a map by connecting all the points to create a path. What would be the best way to do this? Is there a service that automatically does this, or should I dabble into something like the google maps API?
You need links between the nodes to calculate some kind of path. To get a realistic path for say a car or a person you need a accurate road data. This is normaly something you pay for, at least where I come from, but maybe there some way to get it for free.
If you get the road data it will probably not in wgs84, rather in some local or national system and you probably have to convert your point in to that other coordinate system.
To do the calculations you could use some kind of "GIS-tool" like the Open Source QGIS. I'm not cure if they got a one-click solution for your problem but I believe you can solve it with this tool. There are also well-defined algoritms to solve it by hand, just search for Traveling Salesman Problem.
Regards
Given a latitude and longitude, what is the easiest way to find the name of the city and the US zip code of that location.
(This is similar to https://stackoverflow.com/questions/23572/latitude-longitude-database, except I want to convert in the opposite direction.)
Related question: Get street address at lat/long pair
Any of the online services mentioned and their competitors offer "reverse geocoding" which does what you ask--convert lon/lat coordinates into a street address of some-sort.
If you only need the zip codes and/or cities, then I would obtain the Zip Code database and urban area database from the US Census Bureau which is FREE (paid for by your tax dollars). http://www.census.gov/geo/www/cob/zt_metadata.html.
From there, you can either come up with your own search algorithm for the spatial data or make use of one of a spatial databases such as Microsoft SQL Server, PostGIS, Oracle Spatial, ArcSDE, etc.
Update: The 2010 Census data can be found at:
http://www2.census.gov/census_2010/
This is the web service to call.
http://developer.yahoo.com/search/local/V2/localSearch.html
This site has ok web services, but not exactly what you're asking for here.
http://www.usps.com/webtools/
You have two main options:
Use a Reverse Geocoding Service
Google's can only be used in conjunction with an embedded Google Map on the same page, so I don't recommend it unless that is what you are doing.
Yahoo has a good one, see http://developer.yahoo.com/search/local/V3/localSearch.html
I've not used OpenStreetMap's. Their maps look very detailed and thorough, and are always getting better, but I'd be worried about latency and reliability, and whether their address data is complete (address data is not directly visible on a map, and OpenStreetMap is primarily an interactive map).
Use a Map of the ZIP Codes
The US Census publishes a map of US ZIP codes here. They build this from their smallest statistical unit, a Census Block, which corresponds to a city block in most cases. For each block, they find what ZIP code is most common on that block (most blocks have only one ZIP code, but blocks near the border between ZIP codes might have more than one). They then aggregate all the blocks with a given ZIP code into a single area called a Zip Code Tabulation Area. They publish a map of those areas in ESRI shapefile format.
I know about this because I wrote a Java Library and web service that (among other things) uses this map to return the ZIP code for a given latitude and longitude. It is a commercial product, so it won't be for everyone, but it is fast, easy to use, and solves this specific problem without an API. You can read about this product here:
http://askgeo.com/database/UsZcta2010
And about all of your geographic offerings here:
http://askgeo.com
Unlike reverse geocoding solutions, which are only available as Web APIs because running your own service would be extremely difficult, you can run this library on your own server and not depend on an external resource.
If you call volume to the service gets up too high, you should definitely consider getting your own set of postal data. In most cases, that will provide all of the information that you need, and there are plenty of db tools for indexing location data (i.e. PostGIS for PostgreSQL).
You can buy a fairly inexpensive subscription to zipcodes with lat and long info here: http://www.zipcodedownload.com/
Or google's reverse geocoding
link
http://maps.google.com/maps/geo?output=xml&q={0},{1}&key={2}&sensor=true&oe=utf8
where 0 is latitude 1 is longitude
geonames has an extensive set of ws that can handle this (among others):
http://www.geonames.org/export/web-services.html#findNearbyPostalCodes
http://www.geonames.org/export/web-services.html#findNearbyPlaceName
Another reverse geocoding provider that hasn't been listed here yet is OpenStreetMap: you can use their Nominatim search service.
OSM has the (potentially?) added bonus of being entirely user editable (wiki-like) and thus having a very liberal licencing scheme of all this data. Think of this of open source map data.