I tried to get Walgreen's number of full-time employees from Yahoo Finance using importxml like so:
=importxml("https://finance.yahoo.com/quote/WBA/profile", "/html/body/div[1]/div/div/div[1]/div/div[3]/div[1]/div/div[1]/div/div/section/div[1]/div/div/p[2]/span[6]/span")
I have used the function successfully in getting other figures from Yahoo Finance. Example (market cap):
=mid(importxml("https://finance.yahoo.com/quote/WBA", "/html/body/div[1]/div/div/div[1]/div/div[3]/div[1]/div/div[1]/div/div/div/div[2]/div[2]/table/tbody/tr[1]/td[2]/span"),1,6)+0
But with the number of employees (and, by the way, also the trailing twelve months' (ttm) revenues) - I get this error.
Without VBA, with which I am not familiar, how can this be fixed?
Thanks!
This site is built client side by javascript, not server side. Therefore, native functions are inoperative.
You have to extract the json inside the source and parse it.
The object is named root.App.main inside the source.
To get employees for instance
function fullTimeEmployees(url='https://finance.yahoo.com/quote/WBA/profile'){
var source = UrlFetchApp.fetch(url).getContentText()
var jsonString = source.match(/root.App.main = ([\s\S\w]+?);\n/)
if (!jsonString || jsonString.length == 1) return;
var data = JSON.parse(jsonString[1].trim())
Logger.log(data.context.dispatcher.stores.QuoteSummaryStore.assetProfile.fullTimeEmployees)
}
is it possible to scrape “likes number” and “post number” from this website and import data on google sheet?
Because when i try i get empty data since the span of those data are basically all the same…
thanks for help
Edited:
As you even want to push that data to google sheet and read by back from their I could come up with the below solution you could modify according to your need.
First you need to install gspread library and follow this tutorial https://gspread.readthedocs.io/en/latest/oauth2.html to get the credentials to access the google sheets via api and then follow the below updated code.
Your sheet should be like this:
Code:
import requests
import gspread
headers = {'Accept': 'application/json', 'app-token': '33d57ade8c02dbc5a333db99ff9ae26a'}
gc = gspread.service_account(filename="credentials.json")
sh = gc.open("data")
for rownumber,rowvalues in enumerate(sh.sheet1.get_all_values(),1):
if len(rowvalues)==2:
if rowvalues[1]=='':
cookies = requests.post("https://onlyfans.com/api2/v2/init", headers=headers)
data = requests.get(f"https://onlyfans.com/api2/v2/users/{rowvalues[0]}", headers=headers, cookies=cookies)
if data.status_code == 200:
data = data.json()
sh.sheet1.update_cell(rownumber, 2, data["postsCount"])
else:
print(f"Check : {rowvalues}")
else:
cookies = requests.post("https://onlyfans.com/api2/v2/init", headers=headers)
data = requests.get(f"https://onlyfans.com/api2/v2/users/{rowvalues[0]}", headers=headers, cookies=cookies)
if data.status_code == 200:
data = data.json()
sh.sheet1.update_cell(rownumber, 2, data["postsCount"])
print(f"{rownumber} Processed")
Once you run this code you see will data has been updated in google sheets but before running this script follow the URL provided or else you will end up having errors.
Updated Gsheets:
Old:
Seeing to the network logs of that website I was able to extract your desired data by requests library and some of their API calls you check the data.json() dictionary for other data if required.
Follow the below code.
import requests
headers={'Accept': 'application/json', 'app-token': '33d57ade8c02dbc5a333db99ff9ae26a'}
cookies=requests.post("https://onlyfans.com/api2/v2/init",headers=headers)
data=requests.get("https://onlyfans.com/api2/v2/users/elettra_pink",headers=headers,cookies=cookies)
if data.status_code==200:
data=data.json()
print(f'Posts:{data["postsCount"]}\nPhotosCount:{data["photosCount"]}\nVideosCount:{data["videosCount"]}\nFavoritedCount:{data["favoritedCount"]}\nSubscribersCount:{data["subscribersCount"]}')
Output:
Let me know if you have any questions :)
For months I've been using a url like this, from perl:
http://finance.yahoo.com/d/quotes.csv?s=$s&f=ynl1 #returns yield, name, price;
Today, 11/1/17, it suddenly returns a 999 error.
Is this a glitch, or has Yahoo terminated the service?
I get the error even if I enter the URL directly into a browser as, eg:
http://finance.yahoo.com/d/quotes.csv?s=INTC&f=ynl1
so it doesn't seem to be a 'crumb' problem.
Note: This is NOT a question which has been answered in the past!
It was working yesterday.That it happened on the first of the month is suspicious.
As noted in the other answers and elsewhere (e.g. https://stackoverflow.com/questions/47076404/currency-helper-of-yahoo-sorry-unable-to-process-request-at-this-time-erro/47096766#47096766), Yahoo has indeed ceased operation of the Yahoo Finance API. However, as a workaround, you can access a trove of financial information, in JSON format, for a given ticker symbol, by doing a HTTPS GET request to: https://finance.yahoo.com/quote/SYMBOL (e.g. https://finance.yahoo.com/quote/MSFT). If you do a GET request to the above URL, you'll see that the financial data is contained within the response in JSON format. The following python3 script shows how you can parse individual values that you may be interested in:
import requests
import json
symbol = 'MSFT'
url ='https://finance.yahoo.com/quote/' + symbol
resp = requests.get(url)
# parse the section from the html document containing the raw json data that we need
# you can write jsonstr to a file, then open the file in a web browser to browse the structure of the json data
r = str(resp.content, 'utf-8')
i1 = 0
i1 = r.find('root.App.main', i1)
i1 = r.find('{', i1)
i2 = r.find("\n", i1)
i2 = r.rfind(';', i1, i2)
jsonstr = r[i1:i2]
# load the raw json data into a python data object
data = json.loads(jsonstr)
# pull the values that we are interested in
name = data['context']['dispatcher']['stores']['QuoteSummaryStore']['price']['shortName']
price = data['context']['dispatcher']['stores']['QuoteSummaryStore']['price']['regularMarketPrice']['raw']
change = data['context']['dispatcher']['stores']['QuoteSummaryStore']['price']['regularMarketChange']['raw']
shares_outstanding = data['context']['dispatcher']['stores']['QuoteSummaryStore']['defaultKeyStatistics']['sharesOutstanding']['raw']
market_cap = data['context']['dispatcher']['stores']['QuoteSummaryStore']['summaryDetail']['marketCap']['raw']
trailing_pe = data['context']['dispatcher']['stores']['QuoteSummaryStore']['summaryDetail']['trailingPE']['raw']
earnings_per_share = data['context']['dispatcher']['stores']['QuoteSummaryStore']['defaultKeyStatistics']['trailingEps']['raw']
forward_annual_dividend_rate = data['context']['dispatcher']['stores']['QuoteSummaryStore']['summaryDetail']['dividendRate']['raw']
forward_annual_dividend_yield = data['context']['dispatcher']['stores']['QuoteSummaryStore']['summaryDetail']['dividendYield']['raw']
# print the values
print('Symbol:', symbol)
print('Name:', name)
print('Price:', price)
print('Change:', change)
print('Shares Outstanding:', shares_outstanding)
print('Market Cap:', market_cap)
print('Trailing PE:', trailing_pe)
print('Earnings Per Share:', earnings_per_share)
print('Forward Annual Dividend Rate:', forward_annual_dividend_rate)
print('Forward_annual_dividend_yield:', forward_annual_dividend_yield)
Yahoo confirmed that they terminated the service:
It has come to our attention that this service is being used in violation of the Yahoo Terms of Service. As such, the service is being discontinued. For all future markets and equities data research, please refer to finance.yahoo.com .
There is still a way to get this data by querying some APIs used by the finance.yahoo.com page. Not sure if Yahoo will be supporting it long term as the previous API was (hopefully they will).
I adapted the method used by https://github.com/pstadler/ticker.sh into the following python hack that takes a list of symbols from the command line and outputs some of the variables as a csv:
#!/usr/bin/env python
import sys
import time
import requests
if len(sys.argv) < 2:
print("missing parameters: <symbol> ...")
exit()
apiEndpoint = "https://query1.finance.yahoo.com/v7/finance/quote"
fields = [
'symbol',
'regularMarketVolume',
'regularMarketPrice',
'regularMarketDayHigh',
'regularMarketDayLow',
'regularMarketTime',
'regularMarketChangePercent']
fields = ','.join(fields)
symbols = sys.argv[1:]
symbols = ','.join(symbols)
payload = {
'lang': 'en-US',
'region': 'US',
'corsDomain': 'finance.yahoo.com',
'fields': fields,
'symbols': symbols}
r = requests.get(apiEndpoint, params=payload)
for i in r.json()['quoteResponse']['result']:
if 'regularMarketPrice' in i:
a = []
a.append(i['symbol'])
a.append(i['regularMarketPrice'])
a.append(time.strftime(
'%Y-%m-%d %H:%M:%S', time.localtime(i['regularMarketTime'])))
a.append(i['regularMarketChangePercent'])
a.append(i['regularMarketVolume'])
a.append("{0:.2f} - {1:.2f}".format(
i['regularMarketDayLow'], i['regularMarketDayHigh']))
print(",".join([str(e) for e in a]))
Sample Run:
$ ./getquotePy.py AAPL GOOGL
AAPL,174.5342,2017-11-07 17:21:28,0.1630961,19905458,173.60 - 173.60
GOOGL,1048.6753,2017-11-07 17:21:22,0.5749836,840447,1043.00 - 1043.00
var API = "https://query1.finance.yahoo.com/v7/finance/quote?symbols=AAPL";
$.getJSON(API, function (json) {...});call throws this error: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://www.microplan.at/sar' is therefore not allowed access.
Please forgive me, as I have limited knowledge of scraperwiki and twitter mining.
I have the following code to scrape twitter data. However, I want to edit the code to only give me results that are geotagged for New York on a particular date (let's say, April 1, 2013). Do you know how I should do this?
###############################################################################
# Twitter srcaper for the term 'hello'.
###############################################################################
import scraperwiki
import simplejson
# retrieve a page
base_url = 'http://search.twitter.com/search.json?q='
q = 'hello'
options = '&rpp=10&page='
page = 1
while 1:
try:
url = base_url + q + options + str(page)
html = scraperwiki.scrape(url)
#print html
soup = simplejson.loads(html)
for result in soup['results']:
data = {}
data['id'] = result['id']
data['text'] = result['text']
data['from_user'] = result['from_user']
data['created_at'] = result['created_at']
# save records to the datastore
scraperwiki.datastore.save(["id"], data)
page = page + 1
except:
print str(page) + ' pages scraped'
break
In addition to q, use the query parameters geocode and until. See this page of the Twitter API documentation. Please note that you cannot use the Search API to find Tweets older than about a week.
Besides, it's easier to use urllib.urlencode() to construct your query, like for example
query_dict = {'q':'search term(s)', 'geocode':'37.781157,-122.398720,25mi', 'until':'2013-05-10'}
query = urllib.urlencode(query_dict)
response = urllib.urlopen(basic_url + query).read()
Update: Please see this example scraper that you can copy and adapt to your needs.
How do I query the contents of a specific collection using the Python client for Google Docs API?
This is how far I've come:
client = gdata.docs.service.DocsService()
client.ClientLogin('myuser', 'mypassword')
FOLDER_FEED1 = "/feeds/documents/private/full/-/folder"
FOLDER_FEED2 = "/feeds/default/private/full/folder%3A"
feed = client.Query(uri=FOLDER_FEED1 + "?title=MyFolder&title-exact=true")
full_id = feed.entry[0].resourceId.text
(res_type, res_id) = full_id.split(":")
feed = client.Query(uri=FOLDER_FEED2 + res_id + "/contents")
for entry in feed.entry:.
print entry.title.text
The first call to Client.Query succeeds and seems to provide a valid resource ID. The second call, however, returns:
{'status': 400, 'body': 'Invalid request URI', 'reason': 'Bad Request'}
How can I correct this to get it working?
It is much easier once you have a folder entry, to call client.GetResources(entry.content.src) rather than generating the URI by yourself and using a Query.
In your case, client.GetResources(feed.entry[0].content.src).