We've been needing to implement some geo-location service integration for some of our products. There are a lot of 3rd party companies that offer both free and paid geo location services and databases that are updated sometimes constantly and sometimes once a month.
Where do these services get the geo location information for their databases they distribute?
depends on the type of geolocation. if you mean ip<>country/city [eg www.maxmind.com] - basic information can be found in the whois records maintained by the network operators/regional internet registries. for instance: http://tools.whois.net/whoisbyip/?host=64.34.119.12
i suppose it's possible to clean/normalize this data semi-automatically.
if you mean mac address<>geographical coordinates - most probably just spying on the users [with their consent.. or without] - either gathering the information from applications running on mobile devices with gps and wifi onboard or by 'war driving' around - like the google earth street view teams.
We get our data from public enities:
ISO: International Standards Organization
BGN: US Board on Geographic Names
UNGEGN: United Nations Group of Experts on Geographic Names
PCGN: UK Permanent Committee on Geographic Names
FAO: United Nations Food and Argriculture Organization
FIPS: US Federal Information Processing System
ITU: The International Teleccmmunication Union
Then update our MySQL database monthly. Our service is currently FREE!
http://nwstartups.com/api/geo/country.php
Related
I'm using OpenUri and RSS in Rails 5.2.3 and Ruby 2.6.1 to do this.
I'm trying to parse WeWorkRemotely's RSS feed, however, they have one field description that contains all the information in a string. For example, when I parse it in Rails it returns:
"<img src=\"https://we-work-remotely.imgix.net/logos/0015/9022/logo.gif?ixlib=rails-2.1.3&w=50&h=50&dpr=2&fit=fill&auto=compress\" alt=\"Logo.gif?ixlib=rails 2.1\" />\n\n<p>\n <strong>Headquarters:</strong> San Francisco \n <br /><strong>URL:</strong> http://www.loom.com\n</p>\n\n<h1><strong>About Loom</strong></h1><div>Loom is a new kind of work communication tool, already helping over a million people get their message across through instantly shareable videos. Our users work at companies like HubSpot, Square, Uber, GrubHub and LinkedIn. Our mission is to be the global leader in human workplace communication. Founded in 2016, Loom has raised $15 million from top-tier investors including Kleiner Perkins, General Catalyst and Slack Fund.</div><h1><strong>The Role</strong></h1><div>As a Technical Support Engineer, you will be a key part of Loom's support experience at scale and provide timely and effective resolution to customer issues by applying your technical and troubleshooting skills.</div><div><br></div><div>We are looking for support champions who are genuinely happy to help others. If this sounds like you, you came to the right place!<br><br><strong>As a Technical Support Engineer, you will…</strong>\n</div><ul>\n<li>Help customers through email to ensure they are successful with our product</li>\n<li>Leverage effective troubleshooting to quickly identify the source of customer issues and provide a prompt and appropriate solution</li>\n<li>Troubleshoot, investigate, and create detailed bug reports for our Engineering team</li>\n<li>Jump on ad-hoc calls with customers to troubleshoot issues live, as necessary</li>\n<li>Identify bugs, test, report, and working with our Engineering team to assist with a fix</li>\n<li>Actively collect insights from customers and focus on closing the communication loop by providing product feedback to the team</li>\n<li>Provide timely updates to the Support and Engineering Managers regarding new trends in issues</li>\n<li>Develop and document best practices to enhance SL2 troubleshooting processes</li>\n<li>Create technical documentation such as FAQs, guides, knowledge-base articles and how-to’s for Loom customers</li>\n<li>Help the Engineering team develop tools to help our Support team work quickly and efficiently</li>\n<li>Dive into the codebase and gaining domain knowledge of different parts of Loom</li>\n<li>Make efficient changes to the codebase to solve small and quick tasks/issues</li>\n</ul><div>\n<br><strong>You could be a good fit if you have..</strong>\n</div><ul>\n<li>Previous experience delivering excellent support experiences with respect, empathy and understanding</li>\n<li>A minimum of 4+ years of Technical Support and Customer Support experience</li>\n<li>Gained experience/proficiency in Saas solutions and electron apps (CSS, JavaScript, HTML) or have earned a degree in a technical field like computer science</li>\n<li>Technical understanding and ability to troubleshoot and resolve technical problems on your own</li>\n<li>The ability to handle high volume of support conversations</li>\n<li>Excellent written and spoken English</li>\n<li>Are available to work in the Central or Pacific Time Zone and on a full-time schedule that may span weekends and may include holidays as our customers need us</li>\n</ul><div>\n<br><strong>A bonus if you have experience with...</strong>\n</div><ul>\n<li>Installation, configuration, and troubleshooting of Windows and Mac</li>\n<li>Troubleshooting protocols like HTTP, HTTPS, WebSockets, DNS</li>\n<li>Understanding of TCP/IP and ARP to run packet traces and troubleshoot network issues</li>\n<li>Any of these certifications: Cisco CCNA, Microsoft Certified Solutions Expert (MCSE), Apple Certified System Administrator, CompTIAA+, CompTIA Network+</li>\n</ul><div><br></div><div><strong>Perks at Loom</strong></div><div><br></div><div>* Competitive compensation and equity package</div><div>* Medical, dental, and vision coverage (US-based team), healthcare reimbursement (non-US based team)</div><div>* Unlimited PTO</div><div>* Remote-first team</div><div>* Paid parental leave</div><div>* Yearly off-site retreats (this year we went to Costa Rica for a week!)</div><div>* Learning & Development reimbursement</div><div>* Wellness reimbursement</div><div> </div><div><strong>SF office perks</strong></div><div>* Remote weeks every other month</div><div>* Daily in-office lunch, unlimited snacks & drinks</div><div><br></div><div><strong>Remote-specific perks</strong></div><div>* Home office & technology stipends</div><div>* New Hire Onboarding in SF</div><div><br></div><div><strong>Loom is an equal opportunity employer.</strong></div><div>We are actively seeking to create a diverse work environment because teams are stronger with different perspectives and experiences.</div><div><br></div><div>We value a diverse workplace and encourage women, people of color, LGBTQIA individuals, people with disabilities, members of ethnic minorities, foreign-born residents, older members of society, and others from minority groups and diverse backgrounds to apply. We do not discriminate on the basis of race, gender, religion, color, national origin, sexual orientation, age, marital status, veteran status, or disability status. All employees and contractors of Loom are responsible for maintaining a work culture free from discrimination and harassment by treating others with kindness and respect.</div>\n\n<p><strong>To apply:</strong> https://jobs.lever.co/useloom/15398ec6-b2c1-4f95-9ef5-8fa2a62c1bed?lever-origin=applied&lever-source%5B%5D=WeWorkRemotely</p>\n"
What would be the best way for me to actually grab data from this block? Even if I try to pick things up like img src, head quarters, or a href links, it's a big string where I can't easily split that makes sense.
Don't treat it as a string, treat it as an HTML document. Then you can employ the full power of CSS or XPath selectors (or even manual traversal using Ruby methods).
require 'nokogiri'
doc = Nokogiri::HTML.fragment(str)
# img src
doc.at_css('img')["src"]
# => "https://we-work-remotely.imgix.net/logos/0015/9022/logo.gif?ixlib=rails-2.1.3&w=50&h=50&dpr=2&fit=fill&auto=compress"
# headquarters
doc.at_xpath('.//strong[contains(text(), "Headquarters")]/following-sibling::text()').text.strip
# => "San Francisco"
I'm working on a project that returns information based on the user's location. I also want to display the user's town in text (no map) so they can change it if it's not accurate.
If things go well I hope this will be more than a small experiment, so can anyone recommend a good reverse geocoding service with the least restrictions? I notice that Google/Yahoo have a limit to the number of daily queries along with other usage terms. I basically need to take latitude and longitude and convert them to a city/town (which I presume cannot be done using the HTML5 Geolocation API).
Geocoda just launched a geocoding and spatial database service and offers up to 1K queries a month free, with paid plans starting at $49 for 25,000 queries/month. SimpleGeo just closed their Context API so you may want to look at Geocoda or other alternatives.
You're correct, the browser geolocation API only provides coordinates.
I use SimpleGeo a lot and recommend them. They offer 10K queries a day free then 0.25USD per 1K calls after that. Their Context API is what you're going to want, it pretty much does what is says on the tin. Works server-side and client-side (without requiring you to draw a map, like Google.)
GeoNames can also do this and allows up to 30K "credits" a day, different queries expend different credit amounts. The free service has highly variable performance, the paid service is more consistent. I've used them in the past, but don't much anymore because of the difficulty of automatically dealing with their data, which is more "pure" but less meaningful to most people.
I'm failing at Google search today. Is there a page that lists geolocations of the various Amazon AWS server farms?
I want to use this data to pick the appropriate farm for a client on a web app, CDN-style.
(This isn't programming, but it's for the purpose of programming, and I thought it would be useful to have this question answered for public consumption.)
Very late to the party, but in case anyone else has the same question, we've done some work to map AWS regional data centers by the fastest connection to each country (and state in US):
http://www.turnkeylinux.org/blog/aws-datacenters
The code used to this has been open sourced:
https://github.com/turnkeylinux/aws-datacenters
Mashup of associations and worldwide underwater cables:
Regarding the GeoIP implementation, see:
http://www.turnkeylinux.org/blog/geoip-amazon-regions
Each service's detail page (e.g. http://aws.amazon.com/ec2/, http://aws.amazon.com/cloudfront/) provides this information.
WikiLeaks - AmazonAtlas
Confidential AWS datacenters location has been leaked:
11 October 2018, WikiLeaks publishes a "Highly Confidential" internal document from the cloud computing provider Amazon. The document from late 2015 lists the addresses and some operational details of over one hundred data centers spread across fifteen cities in nine countries
https://wikileaks.org/amazon-atlas/
https://wikileaks.org/amazon-atlas/map/
datacenters.com
The best reference I found for tracking down the individual data center locations is datacenters.com's locations page: https://www.datacenters.com/locations
From there you can see locations and details of individual data centers like AWS Ashburn
domenech's Google Maps overlay
The other reference I cam across is from this blog post from domenech: Amazon Web Services Google Maps: World Domination Map
Direct link to the map here
An updated list here:
https://gist.github.com/atyachin/a011edf76df66c5aa1eac0cdca412ea9
Compiled from various sources including those mentioned in other answers.
Coordinates represent the location of a specific datacenter in the zone. Most availability-zones have multiple datacenters so there's no single coordinate for each AZ.
Is there a good physical address to GeoLocation conversion database in the UK? I am trying to use this to build a globrix style search box http://www.globrix.com/ for a web application. Any pointers will be nice. I have been searching for hours. I have found several that convert UK Postcodes into Geolocation. But I need the addresses listed as on Globrix.
The Google Maps API provides a geocoder webservice that you can actually use independently of Google Maps itself. You send it the address/postcode, and it responds with a lat/long plus disambiguated addresses. We use it server-side in the UK to do address lookup. It's incredibly quick, too.
http://code.google.com/apis/maps/documentation/geocoding/index.html
http://www.postcodeanywhere.co.uk should be able to help with this. Alternatively, you can buy the "PAF" (Postcode Address File) from the Royal Mail, but it is expensive.
Update for information relating UK geolocations in 2020. Since 2009:
Google's Geocoder has gotten an order of magnitude more expensive in 2018. It's ~0.5c per search with no free tier
Office for National Statistics have released a free postcode directory called ONSPD. This means if you have the postcode of your address, you can resolve a geolocation accurate to the postcode centroid (this may be 10-100m or so out). There's a free public service API available at https://postcodes.io which allows you to forward or reverse geocode a postcode. There are also public docker data and application images which allow you to host this easily
If you're interested in Rooftop accurate geocodes, a change in Ordnance Survey licensing in 2020 has meant its much simpler and cheaper to access geolocations for almost every premise in Great Britain from Ordnance Survey by combining it with Royal Mail PAF (Postcode Address File). As of September 2020, I think https://ideal-postcodes.co.uk is currently the only company to offer complete and authoritative rooftop geolocations under these new rules. It's likely other PAF vendors will catch up over the coming years.
Disclaimer: I'm the author of postcodes.io and work for ideal-postcodes.co.uk
Given a latitude and longitude, what is the easiest way to find the name of the city and the US zip code of that location.
(This is similar to https://stackoverflow.com/questions/23572/latitude-longitude-database, except I want to convert in the opposite direction.)
Related question: Get street address at lat/long pair
Any of the online services mentioned and their competitors offer "reverse geocoding" which does what you ask--convert lon/lat coordinates into a street address of some-sort.
If you only need the zip codes and/or cities, then I would obtain the Zip Code database and urban area database from the US Census Bureau which is FREE (paid for by your tax dollars). http://www.census.gov/geo/www/cob/zt_metadata.html.
From there, you can either come up with your own search algorithm for the spatial data or make use of one of a spatial databases such as Microsoft SQL Server, PostGIS, Oracle Spatial, ArcSDE, etc.
Update: The 2010 Census data can be found at:
http://www2.census.gov/census_2010/
This is the web service to call.
http://developer.yahoo.com/search/local/V2/localSearch.html
This site has ok web services, but not exactly what you're asking for here.
http://www.usps.com/webtools/
You have two main options:
Use a Reverse Geocoding Service
Google's can only be used in conjunction with an embedded Google Map on the same page, so I don't recommend it unless that is what you are doing.
Yahoo has a good one, see http://developer.yahoo.com/search/local/V3/localSearch.html
I've not used OpenStreetMap's. Their maps look very detailed and thorough, and are always getting better, but I'd be worried about latency and reliability, and whether their address data is complete (address data is not directly visible on a map, and OpenStreetMap is primarily an interactive map).
Use a Map of the ZIP Codes
The US Census publishes a map of US ZIP codes here. They build this from their smallest statistical unit, a Census Block, which corresponds to a city block in most cases. For each block, they find what ZIP code is most common on that block (most blocks have only one ZIP code, but blocks near the border between ZIP codes might have more than one). They then aggregate all the blocks with a given ZIP code into a single area called a Zip Code Tabulation Area. They publish a map of those areas in ESRI shapefile format.
I know about this because I wrote a Java Library and web service that (among other things) uses this map to return the ZIP code for a given latitude and longitude. It is a commercial product, so it won't be for everyone, but it is fast, easy to use, and solves this specific problem without an API. You can read about this product here:
http://askgeo.com/database/UsZcta2010
And about all of your geographic offerings here:
http://askgeo.com
Unlike reverse geocoding solutions, which are only available as Web APIs because running your own service would be extremely difficult, you can run this library on your own server and not depend on an external resource.
If you call volume to the service gets up too high, you should definitely consider getting your own set of postal data. In most cases, that will provide all of the information that you need, and there are plenty of db tools for indexing location data (i.e. PostGIS for PostgreSQL).
You can buy a fairly inexpensive subscription to zipcodes with lat and long info here: http://www.zipcodedownload.com/
Or google's reverse geocoding
link
http://maps.google.com/maps/geo?output=xml&q={0},{1}&key={2}&sensor=true&oe=utf8
where 0 is latitude 1 is longitude
geonames has an extensive set of ws that can handle this (among others):
http://www.geonames.org/export/web-services.html#findNearbyPostalCodes
http://www.geonames.org/export/web-services.html#findNearbyPlaceName
Another reverse geocoding provider that hasn't been listed here yet is OpenStreetMap: you can use their Nominatim search service.
OSM has the (potentially?) added bonus of being entirely user editable (wiki-like) and thus having a very liberal licencing scheme of all this data. Think of this of open source map data.