is it possible to scrape “likes number” and “post number” from this website and import data on google sheet?
Because when i try i get empty data since the span of those data are basically all the same…
thanks for help
Edited:
As you even want to push that data to google sheet and read by back from their I could come up with the below solution you could modify according to your need.
First you need to install gspread library and follow this tutorial https://gspread.readthedocs.io/en/latest/oauth2.html to get the credentials to access the google sheets via api and then follow the below updated code.
Your sheet should be like this:
Code:
import requests
import gspread
headers = {'Accept': 'application/json', 'app-token': '33d57ade8c02dbc5a333db99ff9ae26a'}
gc = gspread.service_account(filename="credentials.json")
sh = gc.open("data")
for rownumber,rowvalues in enumerate(sh.sheet1.get_all_values(),1):
if len(rowvalues)==2:
if rowvalues[1]=='':
cookies = requests.post("https://onlyfans.com/api2/v2/init", headers=headers)
data = requests.get(f"https://onlyfans.com/api2/v2/users/{rowvalues[0]}", headers=headers, cookies=cookies)
if data.status_code == 200:
data = data.json()
sh.sheet1.update_cell(rownumber, 2, data["postsCount"])
else:
print(f"Check : {rowvalues}")
else:
cookies = requests.post("https://onlyfans.com/api2/v2/init", headers=headers)
data = requests.get(f"https://onlyfans.com/api2/v2/users/{rowvalues[0]}", headers=headers, cookies=cookies)
if data.status_code == 200:
data = data.json()
sh.sheet1.update_cell(rownumber, 2, data["postsCount"])
print(f"{rownumber} Processed")
Once you run this code you see will data has been updated in google sheets but before running this script follow the URL provided or else you will end up having errors.
Updated Gsheets:
Old:
Seeing to the network logs of that website I was able to extract your desired data by requests library and some of their API calls you check the data.json() dictionary for other data if required.
Follow the below code.
import requests
headers={'Accept': 'application/json', 'app-token': '33d57ade8c02dbc5a333db99ff9ae26a'}
cookies=requests.post("https://onlyfans.com/api2/v2/init",headers=headers)
data=requests.get("https://onlyfans.com/api2/v2/users/elettra_pink",headers=headers,cookies=cookies)
if data.status_code==200:
data=data.json()
print(f'Posts:{data["postsCount"]}\nPhotosCount:{data["photosCount"]}\nVideosCount:{data["videosCount"]}\nFavoritedCount:{data["favoritedCount"]}\nSubscribersCount:{data["subscribersCount"]}')
Output:
Let me know if you have any questions :)
Related
I am attempting to update a table using a python library to iterate through the table rows.
I get this error: "Error Message: The API you are trying to use could not be found. It may be available in a newer version of Excel."
Adding rows succeeds, but any APIs on the rows endpoint doesn't work, I can't get range or update a row. I even tried going directly to requests to have more control over what gets passed. I tried both the v1.0 and beta endpoints as well.
https://learn.microsoft.com/en-us/graph/api/tablerow-update?view=graph-rest-1.0&tabs=http
Here is the URL Endpoint I am calling:
https://{redacted}/items/{file_id}/workbook/tables/Table1/rows/0
Any help is appreciated.
Update to add code (you have to have an existing authenticated requests session to run it in python):
data = {'values': [5, 6, 7]}
kwargs = {
'data': json.dumps(data),
'headers': {
'workbook-session-id': workbook.session.session_id,
'Content-type': 'application/json'}}
# Works
sharepoint = 'onevmw.sharepoint.com,***REDACTED***'
drive = '***REDACTED***'
item = '****REDACTED***'
base_url = f'https://graph.microsoft.com/v1.0//sites/{sharepoint}/drives/{drive}/items/{item}'
get_url = f"{base_url}/workbook/tables/{test_table.name}/rows"
session = office_connection.account.connection.get_session(load_token=True)
get_response: requests.Response = session.request(method='get', url=get_url)
print(get_response.text)
# Doesn't work
url = f"{base_url}/workbook/tables/{test_table.name}/rows/1"
response: requests.Response = session.request(method='patch', url=url, **kwargs)
print(response.text)
That's an issue. Unfortunately it not documented at the moment in the offical documentation.
I could make it work by changing the url from ".../rows/1" as ".../rows/itemAt(index=1)"
Posting the C# solution for others since the Msft Docs are incorrect and the actual solution is similar to #Amandeep's answer for javascript.
The docs (incorrectly) say:
...Tables["table_name"].Rows["row_num"].Request().UpdateAsync(); // incorrect!
Correct way:
...Tables["table_name"].Rows.ItemAt(123).Request().PatchAsync(wbRow); // correct!
Note the .ItemAt method takes an int, not a string.
I need to connect the api "https://coronavirus-19-api.herokuapp.com/countries" to google sheets and want to update it hourly or as soon as the api page refreshes. Went through various tutorials but need help in parsing and executing it as a first timer.
Please try and implement by using the above mentioned api link. Thanks in advance!
What you're asking is too broad. But I can show you how to get the data and parse it using python. The code goes as follows:
import json
import requests
from pprint import pprint
data = requests.get("https://coronavirus-19-api.herokuapp.com/countries")
data = json.loads(data.text)
pprint(data[0])
print("\n\n",data[0]["country"])
Here I am retrieving the data provided by the API and and parsing it with json. And then I printed the first index of the list followed by the name of the country in that index.
The output look like this-
{'active': 809939,
'cases': 1002529,
'casesPerOneMillion': 0,
'country': 'North America',
'critical': 16346,
'deaths': 57017,
'deathsPerOneMillion': 0,
'recovered': 135573,
'testsPerOneMillion': 0,
'todayCases': 6192,
'todayDeaths': 822,
'totalTests': 0}
North America
This site may you to figure out how the data is stored in the json file.
And follow this to get started with the sheets api.
For months I've been using a url like this, from perl:
http://finance.yahoo.com/d/quotes.csv?s=$s&f=ynl1 #returns yield, name, price;
Today, 11/1/17, it suddenly returns a 999 error.
Is this a glitch, or has Yahoo terminated the service?
I get the error even if I enter the URL directly into a browser as, eg:
http://finance.yahoo.com/d/quotes.csv?s=INTC&f=ynl1
so it doesn't seem to be a 'crumb' problem.
Note: This is NOT a question which has been answered in the past!
It was working yesterday.That it happened on the first of the month is suspicious.
As noted in the other answers and elsewhere (e.g. https://stackoverflow.com/questions/47076404/currency-helper-of-yahoo-sorry-unable-to-process-request-at-this-time-erro/47096766#47096766), Yahoo has indeed ceased operation of the Yahoo Finance API. However, as a workaround, you can access a trove of financial information, in JSON format, for a given ticker symbol, by doing a HTTPS GET request to: https://finance.yahoo.com/quote/SYMBOL (e.g. https://finance.yahoo.com/quote/MSFT). If you do a GET request to the above URL, you'll see that the financial data is contained within the response in JSON format. The following python3 script shows how you can parse individual values that you may be interested in:
import requests
import json
symbol = 'MSFT'
url ='https://finance.yahoo.com/quote/' + symbol
resp = requests.get(url)
# parse the section from the html document containing the raw json data that we need
# you can write jsonstr to a file, then open the file in a web browser to browse the structure of the json data
r = str(resp.content, 'utf-8')
i1 = 0
i1 = r.find('root.App.main', i1)
i1 = r.find('{', i1)
i2 = r.find("\n", i1)
i2 = r.rfind(';', i1, i2)
jsonstr = r[i1:i2]
# load the raw json data into a python data object
data = json.loads(jsonstr)
# pull the values that we are interested in
name = data['context']['dispatcher']['stores']['QuoteSummaryStore']['price']['shortName']
price = data['context']['dispatcher']['stores']['QuoteSummaryStore']['price']['regularMarketPrice']['raw']
change = data['context']['dispatcher']['stores']['QuoteSummaryStore']['price']['regularMarketChange']['raw']
shares_outstanding = data['context']['dispatcher']['stores']['QuoteSummaryStore']['defaultKeyStatistics']['sharesOutstanding']['raw']
market_cap = data['context']['dispatcher']['stores']['QuoteSummaryStore']['summaryDetail']['marketCap']['raw']
trailing_pe = data['context']['dispatcher']['stores']['QuoteSummaryStore']['summaryDetail']['trailingPE']['raw']
earnings_per_share = data['context']['dispatcher']['stores']['QuoteSummaryStore']['defaultKeyStatistics']['trailingEps']['raw']
forward_annual_dividend_rate = data['context']['dispatcher']['stores']['QuoteSummaryStore']['summaryDetail']['dividendRate']['raw']
forward_annual_dividend_yield = data['context']['dispatcher']['stores']['QuoteSummaryStore']['summaryDetail']['dividendYield']['raw']
# print the values
print('Symbol:', symbol)
print('Name:', name)
print('Price:', price)
print('Change:', change)
print('Shares Outstanding:', shares_outstanding)
print('Market Cap:', market_cap)
print('Trailing PE:', trailing_pe)
print('Earnings Per Share:', earnings_per_share)
print('Forward Annual Dividend Rate:', forward_annual_dividend_rate)
print('Forward_annual_dividend_yield:', forward_annual_dividend_yield)
Yahoo confirmed that they terminated the service:
It has come to our attention that this service is being used in violation of the Yahoo Terms of Service. As such, the service is being discontinued. For all future markets and equities data research, please refer to finance.yahoo.com .
There is still a way to get this data by querying some APIs used by the finance.yahoo.com page. Not sure if Yahoo will be supporting it long term as the previous API was (hopefully they will).
I adapted the method used by https://github.com/pstadler/ticker.sh into the following python hack that takes a list of symbols from the command line and outputs some of the variables as a csv:
#!/usr/bin/env python
import sys
import time
import requests
if len(sys.argv) < 2:
print("missing parameters: <symbol> ...")
exit()
apiEndpoint = "https://query1.finance.yahoo.com/v7/finance/quote"
fields = [
'symbol',
'regularMarketVolume',
'regularMarketPrice',
'regularMarketDayHigh',
'regularMarketDayLow',
'regularMarketTime',
'regularMarketChangePercent']
fields = ','.join(fields)
symbols = sys.argv[1:]
symbols = ','.join(symbols)
payload = {
'lang': 'en-US',
'region': 'US',
'corsDomain': 'finance.yahoo.com',
'fields': fields,
'symbols': symbols}
r = requests.get(apiEndpoint, params=payload)
for i in r.json()['quoteResponse']['result']:
if 'regularMarketPrice' in i:
a = []
a.append(i['symbol'])
a.append(i['regularMarketPrice'])
a.append(time.strftime(
'%Y-%m-%d %H:%M:%S', time.localtime(i['regularMarketTime'])))
a.append(i['regularMarketChangePercent'])
a.append(i['regularMarketVolume'])
a.append("{0:.2f} - {1:.2f}".format(
i['regularMarketDayLow'], i['regularMarketDayHigh']))
print(",".join([str(e) for e in a]))
Sample Run:
$ ./getquotePy.py AAPL GOOGL
AAPL,174.5342,2017-11-07 17:21:28,0.1630961,19905458,173.60 - 173.60
GOOGL,1048.6753,2017-11-07 17:21:22,0.5749836,840447,1043.00 - 1043.00
var API = "https://query1.finance.yahoo.com/v7/finance/quote?symbols=AAPL";
$.getJSON(API, function (json) {...});call throws this error: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://www.microplan.at/sar' is therefore not allowed access.
For months I've been using a url like this, from perl:
http://finance.yahoo.com/d/quotes.csv?s=$s&f=ynl1 #returns yield, name, price;
Today, 11/1/17, it suddenly returns a 999 error.
Is this a glitch, or has Yahoo terminated the service?
I get the error even if I enter the URL directly into a browser as, eg:
http://finance.yahoo.com/d/quotes.csv?s=INTC&f=ynl1
so it doesn't seem to be a 'crumb' problem.
Note: This is NOT a question which has been answered in the past!
It was working yesterday.That it happened on the first of the month is suspicious.
As noted in the other answers and elsewhere (e.g. https://stackoverflow.com/questions/47076404/currency-helper-of-yahoo-sorry-unable-to-process-request-at-this-time-erro/47096766#47096766), Yahoo has indeed ceased operation of the Yahoo Finance API. However, as a workaround, you can access a trove of financial information, in JSON format, for a given ticker symbol, by doing a HTTPS GET request to: https://finance.yahoo.com/quote/SYMBOL (e.g. https://finance.yahoo.com/quote/MSFT). If you do a GET request to the above URL, you'll see that the financial data is contained within the response in JSON format. The following python3 script shows how you can parse individual values that you may be interested in:
import requests
import json
symbol = 'MSFT'
url ='https://finance.yahoo.com/quote/' + symbol
resp = requests.get(url)
# parse the section from the html document containing the raw json data that we need
# you can write jsonstr to a file, then open the file in a web browser to browse the structure of the json data
r = str(resp.content, 'utf-8')
i1 = 0
i1 = r.find('root.App.main', i1)
i1 = r.find('{', i1)
i2 = r.find("\n", i1)
i2 = r.rfind(';', i1, i2)
jsonstr = r[i1:i2]
# load the raw json data into a python data object
data = json.loads(jsonstr)
# pull the values that we are interested in
name = data['context']['dispatcher']['stores']['QuoteSummaryStore']['price']['shortName']
price = data['context']['dispatcher']['stores']['QuoteSummaryStore']['price']['regularMarketPrice']['raw']
change = data['context']['dispatcher']['stores']['QuoteSummaryStore']['price']['regularMarketChange']['raw']
shares_outstanding = data['context']['dispatcher']['stores']['QuoteSummaryStore']['defaultKeyStatistics']['sharesOutstanding']['raw']
market_cap = data['context']['dispatcher']['stores']['QuoteSummaryStore']['summaryDetail']['marketCap']['raw']
trailing_pe = data['context']['dispatcher']['stores']['QuoteSummaryStore']['summaryDetail']['trailingPE']['raw']
earnings_per_share = data['context']['dispatcher']['stores']['QuoteSummaryStore']['defaultKeyStatistics']['trailingEps']['raw']
forward_annual_dividend_rate = data['context']['dispatcher']['stores']['QuoteSummaryStore']['summaryDetail']['dividendRate']['raw']
forward_annual_dividend_yield = data['context']['dispatcher']['stores']['QuoteSummaryStore']['summaryDetail']['dividendYield']['raw']
# print the values
print('Symbol:', symbol)
print('Name:', name)
print('Price:', price)
print('Change:', change)
print('Shares Outstanding:', shares_outstanding)
print('Market Cap:', market_cap)
print('Trailing PE:', trailing_pe)
print('Earnings Per Share:', earnings_per_share)
print('Forward Annual Dividend Rate:', forward_annual_dividend_rate)
print('Forward_annual_dividend_yield:', forward_annual_dividend_yield)
Yahoo confirmed that they terminated the service:
It has come to our attention that this service is being used in violation of the Yahoo Terms of Service. As such, the service is being discontinued. For all future markets and equities data research, please refer to finance.yahoo.com .
There is still a way to get this data by querying some APIs used by the finance.yahoo.com page. Not sure if Yahoo will be supporting it long term as the previous API was (hopefully they will).
I adapted the method used by https://github.com/pstadler/ticker.sh into the following python hack that takes a list of symbols from the command line and outputs some of the variables as a csv:
#!/usr/bin/env python
import sys
import time
import requests
if len(sys.argv) < 2:
print("missing parameters: <symbol> ...")
exit()
apiEndpoint = "https://query1.finance.yahoo.com/v7/finance/quote"
fields = [
'symbol',
'regularMarketVolume',
'regularMarketPrice',
'regularMarketDayHigh',
'regularMarketDayLow',
'regularMarketTime',
'regularMarketChangePercent']
fields = ','.join(fields)
symbols = sys.argv[1:]
symbols = ','.join(symbols)
payload = {
'lang': 'en-US',
'region': 'US',
'corsDomain': 'finance.yahoo.com',
'fields': fields,
'symbols': symbols}
r = requests.get(apiEndpoint, params=payload)
for i in r.json()['quoteResponse']['result']:
if 'regularMarketPrice' in i:
a = []
a.append(i['symbol'])
a.append(i['regularMarketPrice'])
a.append(time.strftime(
'%Y-%m-%d %H:%M:%S', time.localtime(i['regularMarketTime'])))
a.append(i['regularMarketChangePercent'])
a.append(i['regularMarketVolume'])
a.append("{0:.2f} - {1:.2f}".format(
i['regularMarketDayLow'], i['regularMarketDayHigh']))
print(",".join([str(e) for e in a]))
Sample Run:
$ ./getquotePy.py AAPL GOOGL
AAPL,174.5342,2017-11-07 17:21:28,0.1630961,19905458,173.60 - 173.60
GOOGL,1048.6753,2017-11-07 17:21:22,0.5749836,840447,1043.00 - 1043.00
var API = "https://query1.finance.yahoo.com/v7/finance/quote?symbols=AAPL";
$.getJSON(API, function (json) {...});call throws this error: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://www.microplan.at/sar' is therefore not allowed access.
I'd like to understand how to create a new ticket in JIRA using REST API from Jenkins. Is there any limitations or special things I should be aware of?
I'm going to write a Python script, which will parse the build log and then create a new ticket in JIRA project.
I checked the plugins, but most of them only can update the existing tickets.
Thanks
There's documentation here about the JSON schema and some example JSON which needs to go in the body of your POST request to /rest/api/2/issue
https://docs.atlassian.com/jira/REST/cloud/#api/2/issue-createIssue
Here's a basic python3 script to make a post request
import requests, json
from requests.auth import HTTPBasicAuth
base_url = "myjira.example.com" # The base_url of the Jira insance.
auth_user = "simon" # Jira Username
auth_pass = "N0tMyRe3lP4ssw0rd" # Jira Password
url = "https://{}/rest/api/2/issue".format(base_url)
# Set issue fields in python dictionary. See docs and comment below regarding available fields
fields = {
"summary": "something is wrong"
}
payload = {"fields": fields}
headers = {"Content-Type": "application/json"}
response = requests.post(
url,
auth=(auth_user, auth_pass),
headers=headers,
data=json.dumps(payload))
print("POST {}".format(url))
print("Response {}: {}".format(response.status_code, response.reason))
_json = json.loads(response.text)
Using this HTTP requests library for python http://docs.python-requests.org/en/master/
You can make a GET request to /rest/api/2/issue/{issueIdOrKey}/editmeta using the id or key of existing issue in the same project as the issue's you will be creating via the API will go to in order to get a list of all the fields you can set and which ones are required.
https://docs.atlassian.com/jira/REST/cloud/#api/2/issue-getEditIssueMeta