Tableau has a limited set of geographic roles (outside the US) that can be displayed on a choropleth map. I have a list of (Country, City) or (Country, District) pairs and I want to have it converted into a format that Tableau can read as a Geographic role.
Any service that can do that?
TIA
Related
I have a .csv file with Twitter profiles including information such as username, name, description etc. One column is geolocation. In this text the user may have a country (i.e., UK), a city or town (i.e., Cambridge), an actual address (5 Tyrian Place, WR5 TY1), a state (i.e, California, CA) or something silly (i.e., West of Hell).
Is there an API/library/automatic way of taking this information and deriving the country? For example, if the location is Cambridge the output should be UK, if the address is in the UK, the output should be UK, etc.
Google has a reverse geocoding service which you can access through their Maps API:
https://developers.google.com/maps/documentation/geocoding/start
They let you make 2500 free requests per day. One nice feature is it will give you correct latitude, longitude, state, country, etc for things like "Golden Gate Bridge" and "The Big Apple." Twitter users enter all sorts of (sarcastic) phrases for their location -- like "West of Hell," "Mars," etc -- and Google will reverse geocode that as well. Though, that may not be very useful.
As another level of checking, you can compare the user's timezone ("utc_offset"), if it is present, to the place that Google returns. It's a bit involved and requires that you compare the timezone's latitude boundaries to the latitude and longitude in Google's response.
I have a list of strings (noun phrases) and I want to filter out all valid geographical locations from them. Most of these (unwanted location names) are country or city or state names. What would be a way to do this? Is there any open-source lookup table available which contains all country, states, cities of the world?
Example desired output:
TREC4: false, Vienna: true, Ministry: false, IBM: false, Montreal: true, Singapore: true
Unlike this post: Verify user input location string is a valid geographic location?
I have a high number of strings like these (~0.7 million) so google geolocation API is probably not an option for me.
You can use geoplanet data by Yahoo, or geonames data by geonames.org.
Here is a link to geoplanet TSV file containing 5 million geographical places of the world :
https://developer.yahoo.com/geo/geoplanet/data/
Moreover, geoplanet data will provide you type ( city,country,suburb etc) of the geographical place, along with a unique id.
https://developer.yahoo.com/geo/geoplanet/guide/concepts.html
You can do a lowercase, sanitized ( e.g. remove special characters and other anomalies) match of your needle string to the names present in this data.
If you do not want full file scans, first processing this data to store it in a fast lookup database like mongodb or redis will be beneficial.
I can suggest the following three options:
a) Using the Alchemy API: http://www.alchemyapi.com/
If you try their demo, places like France, Honolulu give the entity type as Country or City
b) Using TAGME: http://tagme.di.unipi.it/
TAGME connects every entity in a given text to the corresponding wikipedia page. Crawl the wikipedia page and check the infobox and filter
c) Using Wikipedia Miner: I was unable to find relevant links for this. However, this also works like TAGME.
Suggest you to try all three and do majority voting for each instance.
Are most web and smartphone applications that show you how far other users are from you essentially based on a user's latitude and longitude? That is, do these apps basically derive a user's latitude and longitude from their country and postal code and then use an algorithm to create a sorted list of all users who are near them, closest first? I believe the answer is "yes" but I want to make sure before I build this feature into my Django application.
For those countries that don't use postal codes, I would imagine the latitude and longitude are derived from the city/region/country tuple they reside in.
Yes, they map IP to lat/long, and lat/long to city, and city to country, unless they have access to a GPS device/sensor for the exact position (+/- 1m).
The thing is, if you have an ip, you can convert that into a number (biginteger)
74.88.21.55 ==> (a.b.c.d) ==> d * 255^0 + c * 255^1 + b * 255^2 +a * 255^3 = x
then you can query a database.
All you need is a lookup-table:
T_Lookup
IP_Range_Start IP_Range_End Latitute Longitue
Then you can query like this:
SELECT Latitude, Longitude FROM T_Lookup
WHERE x BETWEEN IP_Range_Start AND IP_Range_End
Then you can calculate the distance between the two points using the haversine formula.
You can grab C code here:
http://aimbots.net/tutorials/7680-getting-country-ip-address.html
And an IP to country csv you find here:
http://ip-to-country.webhosting.info/node/view/6
Apps that display the locations of users relative to other users generally gather their data from either GPS data or IP address location data.
This Wiki article provides a nice explanation of geolocation.
They use the Galactic Coordinate system, which does in fact use longitude, l, and latitude, b.
http://en.wikipedia.org/wiki/Galactic_coordinate_system
from a list of lat/lon coordinates, how does one return a list of streets nearby in the format of (streetname,State Abbreviation) ex. Oaktree ln, TX from open street map?
You should determine for yourself what exactly is 'nearby' and how many streets you want in your list. You could then look into some services based on OSM to get what you want exactly. Nominatim performs reverse geocoding and might be what you want.
Looking for a way to get a list of telephone area codes for a given latitude and longitude (and if necessary a given intl. code.) Note, I'm not talking about international dialing prefixes but the area codes within them.
For example, Denver Colorado is covered by the area codes 303 and 720. It's at 39.739 -104.985 and is in NANP 1. So given 39.739,-104.985,1 I'd like to get back [303,720].
Libraries, web services, DB's, or raw data that needs to be parsed into a DB, e.g., a web page of shape points, are all fine and the more global coverage the better, but just NANP 1 would be a great help.
Note I already use MaxMind and could turn the lat-lng into a fake IP and use that as the lookup key, but MaxMind claims only U.S. area codes (whether they truly mean U.S. or actually NANP I haven't tested) and seemingly only 1 per location (e.g. just 303 for Denver.) So it's a possibility, just not a great one.
UPDATE: I found some more relevant information, but no definitive solutions so I'm listing it here rather than in an answer:
I was able to find two U.S. databases http://www.area-codes.com/area-code-database.asp and http://www.nationalnanpa.com/area_codes/index.html (50% down the page, MS Access file.) The former includes lat/lng for $450 and the latter would require nearest-neighbor matching as KeithS talks about (it's probably the same DB underlying the NANPA City Query he found.)
Additionally I found information that implies Teleatlas has area code boundary maps and that ESRI includes area code shape files with copies of ArcGIS. Maponics seems to have data available: there's a Google Maps implementation of Maponics' data at http://www.usnaviguide.com/areacode.htm.
Wow. You'll definitely need some sort of pre-existing database of points. My first thought was ZIPList5 Geocode. It includes lat-long data for each active U.S. ZIP code, so you can throw this data in a DB table, index the hell out of it, and search by just about any geographic info you'd have access to. You can buy one copy for $40, with enterprise-level use for $100. Only problem is that this DB has only the "primary" area code for each ZIP code, so metro areas that have more than one (Dallas, Chicago, NYC) aren't going to show all of them.
You could try a two-pronged approach with some free data I found: for a given latitude and longitude, do a nearest-neighbors search of the data in the USGS Geographic Names Information System; it includes information on every human habitation center, and every named landmark feature, with lat/long coordinates of their centers. You now have your lat/long point mapped to the nearest town/city, ZIP code, county, and state. Now, you can compare that against this list of U.S. Area Codes, to find area codes matching any or all of the identifying information from the USGS. This is all free, and will eventually get you what you need, but you'll probably have to do some work to "massage" the two sets of data into something you can efficiently cross-reference, and/or you'll need to implement a good "search engine" that will accurately find nearest-neighbor named points, and then find area codes for locations matching the names.
One more thing to look at is NANPA, which administers area code assignment to begin with. I'm sure they have a more comprehensive downloadable DB, but the only free public access I could find was this search page, which will find area codes for any city with >20k people. You could turn your lat/long data into a city and state, and then hit this search page: NANPA City Query
Here is an option:
http://geocoder.ca/39.739,-104.985?geoit=xml
<TimeZone>America/Denver</TimeZone>
<AreaCode>720,303</AreaCode