We'd like to use the UPRN and Postcode lookup files that the UK's Office for National Statistics (ONS) provide through their web portal:
Postcode: https://geoportal.statistics.gov.uk/datasets/national-statistics-postcode-lookup-2021-census-november-2022/about
UPRN: https://geoportal.statistics.gov.uk/datasets/national-statistics-uprn-lookup-october-2022/about
However, these files change frequently, and the ONS offer no API to return the latest dataset for each source of data.
We've tried looking for an endpoint that remains static that could be used to get the latest iteration of the file each time, but we can't find one. We have also looked for a way of returning the latest endpoint as a URL, but there does not appear to be a programmatic way to do that.
Related
On daily basis I am getting an attachment in email. Basically that attachment is an excel spreadsheet. I am pushing that spreadsheet to google drive folder ./attachments using Microsoft Power Automate. The main purpose of pushing that spreadsheet into google drive is to load into PowerBi for Analytics purposes.
In PowerBi I am using the "Web" connector to import the file and it's working fine. The sample link is below.
https://docs.google.com/spreadsheets/u/2/d/1eBJR6wrcFrdjv4Lbf_Wq3MQOeUwBbgLw/export?format=xlsx
The above link exports the file into powerBi and hence I can load data into PowerBi.
The problem is, on daily basis I am getting new file in drive and the Unique Id of the file is not same. In the above example the unique Id 1eBJR6wrcFrdjv4Lbf_Wq3MQOeUwBbgLw will be different for the second file even though I am renaming the file with the same name using Microsoft Power Automate when pushing to Google Drive. eg: "PowerBi load file.xlsx". Is that possible that I can get a stable link for all the files with the same name?.
I have also shared the whole folder ./attachments and tried to get the link of the file but that doesn't work. eg:
https://drive.google.com/drive/folders/1h1VuPtXfWflgIQw7ecMTwweoLblADscq/PowerBi Analytics file.xlsx/export?format=xlsx
Any help, suggestions will be really appreciated.
Thanks everyone.
I believe your goals as follows.
You want to retrieve the file IDs from a filename.
You want to retrieve the file IDs from a shared folder.
Answer for question 1:
In order to retrieve the file IDs from a filename, I think that the method of "Files: list" of Drive API can be used.
The endpoint is as follows.
GET https://www.googleapis.com/drive/v3/files?q=name%3D%27{filename}%27
The search query is name='{filename}'.
In this case, the API key cannot be directly used because the file list tries to be retrieved from the while Google Drive including the file. In this case, the access token is required to be used.
By this, I thought that your goal 2 might be suitable.
Answer for question 2:
In order to retrieve the file IDs from a shared folder, I think that the method of "Files: list" of Drive API can be also used. In this case, at least, the file list is retrieved from the shared folder. By this, the API key can be used.
The endpoint is as follows.
GET https://www.googleapis.com/drive/v3/files?q=%271h1VuPtXfWflgIQw7ecMTwweoLblADscq%27%20in%20parents&key=[YOUR_API_KEY]
The search query is '1h1VuPtXfWflgIQw7ecMTwweoLblADscq' in parents.
In this case, the file list can be retrieved using the API key because the folder is publicly shared and the file list is directly retrieved from the publicly shared.
But, in order to use this, it is required to use the API key. Please be careful this.
Other pattern:
If you want to achieve your goal without the API key and the access token, I would like to propose to use the Web Apps created by Google Apps Script as a wrapper API. When this Web Apps is used, you can achieve above both goals without using the API key and the access token.
The official document of Web Apps is here.
The unofficial document of Web Apps including several sample situations is here.
References:
Files: list
Search for files and folders
How can I retrieve all Arabic speaking channels locally in Israel.
Using the API explorer, I don't see such an option.
Which fields are mandatory for search to call the search API?
Is there any way to fulfill this task?
Update:
API Explorer
I couldn't get results
There are lots of parameters in the API v3 Docs for Search that you can play around with, some in particular you might find helpful are:
location
locationRadius
regionCode
relevanceLanguage
and you'll need to set type = "channel" to return only channels.
The only mandatory parameter is part which is what data you want returned from your results. For this search call, you only have one option for part, which is "snippet".
This is the only way to do it simply within your multiple parameters, and it will take 100 quota per search, assuming you have an API key already.
If you press 'Try This API' on the right, it'll even give you code snippets you can copy in a variety of languages, since you didn't specify one in the prompt.
Depending on the language, a call could look like the following:
GET https://youtube.googleapis.com/youtube/v3/search?part=snippet&location=Israel®ionCode=IL&relevanceLanguage=ar&type=channel&key=[YOUR_API_KEY] HTTP/1.1
I'm trying to fetch recent files from SharePoint Online using Microsoft Graph.
The API I'm using is :
https://graph.microsoft.com/v1.0/me/drive/recent
But the results include files from all site collection. I need to restrict the results to only a single site collection.
After some research, I found that there is no direct API to restrict the site collection.
I also tried using adding filter like :
$filter=sharepointIds/siteId eq 'site-id'
but it was also not working. It gave a `501 (Not implemented) error.
Update:
I found another API from Graph which is
https://graph.microsoft.com/beta/me/insights/used
which is inside insights(beta) named "items viewed and modified by me"
But it again returns data across tenant and I need to restrict it to a single site collection. I guess filter conditions are not yet implemented for this API in Graph.
Is there any method to get recent files from a particular site collection? Any help is appreciated.
One correction, /me/drive/recent isn't pulling from all site collections, it's pulling from the current user's OneDrive. This includes recent files held within their drive and any recent changes to files that have been shared with them.
As for finding Recent Files from a SharePoint Document Library, you need to make this request within the context of the drive you want to pull from. For example:
https://graph.microsoft.com/v1.0/sites/root/drives/{documentLibraryId}/recent
I have some experience building ROR projects with APIs for Google Maps, Weather Underground, etc. All of these sites had a login section with an API key. I now have a project to complete that requires an API for Github Jobs API. I have looked and looked and there is no API key provided on that page or on the regular Github API page. I have searched online at length and watched about 3 tutorials, yet I am at a loss about how to accomplish this without an API key. I would really appreciate some guidance. Thanks so much!!
Github Jobs API it's open and doesn't need API key to use, you only need to make GET requests as described here
URL: https://jobs.github.com
REQUEST: GET /positions.json
You can send a lot of parameters as query params:
description — A search term, such as "ruby" or "java". This parameter is aliased to search.
location — A city name, zip code, or other location search term.
lat — A specific latitude. If used, you must also send long and must not send location.
long — A specific longitude. If used, you must also send lat and must not send location.
full_time — If you want to limit results to full time positions set this parameter to 'true'.
Some examples of requests:
https://jobs.github.com/positions.json?description=python&location=sf&full_time=true
https://jobs.github.com/positions.json?search=ruby
https://jobs.github.com/positions.json?lat=37.3229978&long=-122.0321823
https://jobs.github.com/positions.json?location=sp&full_time=true
For use pagination just add page= at the final of query params:
https://jobs.github.com/positions.json?description=ruby&page=1
You can check all the documentation of API here
I have a collection of URLs that I want to estimate the age of. Let me phrase the question this way:
How to estimate the earliest point in time in which querying a URL would be successful (let's say HTTP status code 200 for a GET request)?
The solution I'm currently thinking about is perhaps Google (or some other crawler) has some (publicly available) way of providing the timestamp when they first visited that URL (preferably API).
I know how to get the age of Google's cached version, e.g.: webcache.googleusercontent.com/search?q=cache:stackoverflow.com. However, because the cached versions are updated rather frequently, this isn't very useful.
Not possible in a reliable way. (Well, unless you have all access log files of the servers you are interested in.)
Internet Archive’s Wayback Machine shows the first time it crawled a webpage. Of course it may take time until their bots find and crawl a page for the first time, so most indexed pages probably are much older.
Also note: as soon as the crawler is blocked (e.g., via robots.txt), the history/copies will be removed (from the FAQ):
When a URL has been excluded at direct owner request from being archived, that exclusion is retroactive and permanent.