We are using Nylas api to get Access token for different type of Email account like Gmail, Outlook.. But We couldn't authenticate for Gmail.
let myURL = URL(string: getNylasAuthUrl())
let userAgent = getUserAgentParams()
webView.customUserAgent = userAgent
let myRequest = URLRequest(url: myURL!)
webView.load(myRequest)
got below error
Finally found a way, by setting User-Agent, we could do authentication for gmail from post
Tried below User-agents but didn't help
let userAgent = "Mozilla/5.0 (Apple \(Utils.getDeviceModel()) ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"
let userAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"
let userAgent = "Mozilla/5.0 (Google) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"
Finally, I found the working user-agent.
let userAgent = "Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_2 like Mac OS X)
AppleWebKit/603.1.30 (KHTML, like Gecko) Mobile/14F89 Safari/602.1"
If you want to Google auth via Webview, use this user-agent especially for getting access token.
Related
I am trying to scrape this website
Scraping information is possible manually
However, I cannot access information within p...p tags and ul...ul tags with one loop. These two tags are in a similar division. However, the loop breaks whenever p replaces ul or vice-versa.
Is this possible with just one loop??
import requests
from bs4 import BeautifulSoup as bs
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36",
"Accept-Encoding":"gzip, deflate, br",
"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"DNT":"1",
"Connection":"close",
"Upgrade-Insecure-Requests":"1"}
source = requests.get('https://insights.blackcoffer.com/how-small-business-can-survive-the-coronavirus-crisis/',
headers=headers)
page = source.content
soup = bs(page, 'html.parser')
information = ''
for section in soup.find('div', class_='td-post-content').find_all('p'):
if information != '':
information = information + '\n' + section.text
else:
information = section.text
print(information)
import requests
from bs4 import BeautifulSoup as bs
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36",
"Accept-Encoding":"gzip, deflate, br",
"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"DNT":"1",
"Connection":"close",
"Upgrade-Insecure-Requests":"1"}
source = requests.get('https://insights.blackcoffer.com/how-small-business-can-survive-the-coronavirus-crisis/',
headers=headers)
page = source.content
soup = bs(page, 'html.parser')
information = ''
for section in soup.find('div', class_='td-post-content').find_all(['p', 'li']):
information += '\n\n' + section.text
print(information.strip())
I noticed something strange with WKWebView user agents.
If I try setting it as
webViewConfiguration.defaultWebpagePreferences.preferredContentMode = .mobile
webViewConfiguration.applicationNameForUserAgent = "MYIOSAPP/\(appVersion)"
I get this:
Mozilla/5.0 (iPad; CPU OS 15_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) MYIOSAPP/1.0
However if I set the customUserAgent like so:
webView.customUserAgent = (webView.value(forKey: "userAgent") ?? "") + "MYIOSAPP/\(appVersion)"
I get this:
Mozilla/5.0 (iPad; CPU OS 15_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MYIOSAPP/1.0
What is causing it to lose the Mobile part of the user agent? Also why does it even need it if it already has the iOS version number in the string?
import scrapy
class oneplus_spider(scrapy.Spider):
name='one_plus'
page_number=0
start_urls=[
'https://www.amazon.com/s?k=samsung+mobile&page=3&qid=1600763713&ref=sr_pg_3'
]
def parse(self,response):
all_links=[]
total_links=[]
domain='https://www.amazon.com'
href=[]
link_set=set()
href=response.css('a.a-link-normal.a-text-normal').xpath('#href').extract()
for x in href:
link_set.add(domain+x)
for x in link_set:
next_page=x
yield response.follow(next_page, callback=self.parse_page1)
def parse_page1(self, response):
title=response.css('span.a-size-large product-title-word-break::text').extract()
print(title)
Error after running the code - (failed 2 times): 503 Service Unavailable.
I tried many ways but failed. Please help me. Thanks in advance!
Check url by "curl" first. like,
curl -I "https://www.amazon.com/s?k=samsung+mobile&page=3&qid=1600763713&ref=sr_pg_3"
then, you can see 503 response.
HTTP/2 503
In other words, your request is wrong.
you have to find proper request.
Chrome DevTools will help you. like
I think that user-agent ( like browser ) must be needed.
curl 'https://www.amazon.com/s?k=samsung+mobile&page=3&qid=1600763713&ref=sr_pg_3' \
-H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36' \
--compressed
so... It may work,
import scrapy
class oneplus_spider(scrapy.Spider):
name='one_plus'
page_number=0
user_agent = "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36"
start_urls=[
'https://www.amazon.com/s?k=samsung+mobile&page=3&qid=1600763713&ref=sr_pg_3'
]
def parse(self,response):
all_links=[]
total_links=[]
domain='https://www.amazon.com'
href=[]
link_set=set()
href=response.css('a.a-link-normal.a-text-normal').xpath('#href').extract()
for x in href:
link_set.add(domain+x)
for x in link_set:
next_page=x
yield response.follow(next_page, callback=self.parse_page1)
def parse_page1(self, response):
title=response.css('span.a-size-large product-title-word-break::text').extract()
print(title)
I am trying to decipher this information from user-agent string on a node.js server based on Sails.js framework.
I have access to user-agent in req.headers["user-agent"]
Currently I am using this function to segregate iPhone, iPad and Android devices.
function findPlatform(userAgent){
var iphoneReg = /\biphone\b/gi;
var ipadReg = /\bipad\b/gi;
var androidReg = /\bandroid\b/gi;
if(!userAgent){
sails.log.error("cant infer user agent");
return "others";
}
if(userAgent.search(androidReg) > -1){
return "android";
}
else if(userAgent.search(iphoneReg) > -1){
return "iphone";
}
else if(userAgent.search(ipadReg) > -1){
return "ipad";
}
else {
return "others";
}
}
However, I also need to segregate between mobile app and mobile browser for both android and iOS. I was looking at certain requests and could see that user-agent from mobile app looks like this:
"Mozilla/5.0 (Linux; Android 5.0; Lenovo A7000-a Build/LRX21M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/45.0.2454.95 Mobile Safari/537.36"
While from mobile browser, it looked like this:
"Mozilla/5.0 (Linux; Android 4.4.4; MI 4W Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36"
Can I user a regex to match keyword "Version" to identify the request as coming from app ? What about iOS ?
I want to detect what operating system the client is using. I can detect whether it is mobile or desktop. How can I detect whether it is using IOS or Android?
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/28.0.1500.52 Chrome/28.0.1500.52 Safari/537.36
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_2 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko)
Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.10
Use a gem like 'browser' or one like it.
https://github.com/fnando/browser
browser = Browser.new(:ua => "some string", :accept_language => "en-us")
browser.name # readable browser name
browser.version
browser.safari?
browser.opera?
browser.chrome?
browser.mobile?
browser.tablet?
browser.firefox?
browser.ie?
browser.ie6? # this goes up to 10
browser.modern? # Webkit, Firefox 17+, IE 9+ and Opera 12+
browser.platform # return :mac, :windows, :linux or :other
browser.mac?
browser.windows?
browser.linux?
browser.blackberry?
browser.meta # an array with several attributes
browser.to_s # the meta info joined by space