How to Scrape a website without disabling ssl - ruby-on-rails

I have to scrape a website without disabling SSL. I tried by using Nokogiri gem
require 'httparty'
require 'nokogiri'
require 'open-uri'
page = open("https://mywebsiteurl.com",{ssl_verify_mode: OpenSSL::SSL::VERIFY_NONE})
doc = Nokogiri::HTML(page)
puts doc
This code works by disabling SSL. but I want it to work without disabling SSL.
when I tried without disabling SSL I got this error
SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (OpenSSL::SSL::SSLError)
when I do curl https://mywebsiteurl.com I got this result.
* Hostname was NOT found in DNS cache
* Trying xxx.xxx.xxx.xxx...
* Connected to wxxxxxxxxx.com (xxx.xxx.xxx.xxx) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS alert, Server hello (2):
* SSL certificate problem: certificate has expired
* Closing connection 0
curl: (60) SSL certificate problem: certificate has expired
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.

Related

Blackfire profiling error - exit status 60

I am running a vagrant box with Centos 7 as its OS. I installed blackfire without error and then tried to profile from the web browser. It started profile, but then just hung and hung and never finished. I then tried it via curl in the command line and got the following error:
$ blackfire curl https://gitlist.demo.blackfire.io/
Profiling: [####------------------------------------] 1/10
Error while running command: exit status 60
Use the option '--ignore-exit-status' to ignore command exit status
* About to connect() to gitlist.demo.blackfire.io port 443 (#0)
* Trying 54.76.137.79...
* Connected to gitlist.demo.blackfire.io (54.76.137.79) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* Server certificate:
* subject: CN=gitlist.demo.blackfire.io
* start date: Jul 07 14:09:16 2019 GMT
* expire date: Oct 05 14:09:16 2019 GMT
* common name: gitlist.demo.blackfire.io
* issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
* NSS error -8181 (SEC_ERROR_EXPIRED_CERTIFICATE)
* Peer's Certificate has expired.
* Closing connection 0
curl: (60) Peer's Certificate has expired.
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a
"bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
Can someone explain to me what this error is? I have used Blackfire successfully on other projects (albeit, those were Ubuntu boxes) and have never seen this error. I can't find much on the web about this as well.
cURL is not sending the HTTP request because the "Peer's Certificate has expired".
As suggested, you could have added the "--insecure" option.
But it seems the certificate has been renewed.

solace pub sub + REST consumer Authentication schema client-certificate

we recently upgraded our solace VMR to pub sub+, I am trying to configure rest delivery point using SSL. I created a rest consumer and set authentication schema as client-certificate, it is not taking and displaying as none. I did all this from CLI.The ssl settings looks good default cipher and trusted CN. the rest consumer was down with failure reason " Remote SSL handshake failed: sslv3 alert handshake failure"
I suspect solace is not sending certificate when it is trying to connect. Any thoughts on setting Authentication scheme as client-certificate?
Check that the rest-consumer is configured correctly to use SSL, and to authenticate with client-certificate. The CLI commands are:
show message-vpn <vpn_name> rest rest-consumer <rest-consumer_name> authentication
show message-vpn <vpn_name> rest rest-consumer <rest-consumer_name> detail
Also, check that the CA of the remote host certificate is trusted with CLI command:
show certificate-authority ca-name * cert
At the minimum, the root issuer of the remote host certificate must be one of the configured certificate-authorities in that CLI command.
If those check out, ensure that the remote endpoint is indeed requesting client-certificates in the first place. You can check the raw packets on the wire with Wireshark:
Internet Protocol Version 4, Src: <remote-host-ip>, Dst: <solace-ip>
...
Secure Sockets Layer
TLSv1.2 Record Layer: Handshake Protocol: Server Hello
Content Type: Handshake (22)
...
TLSv1.2 Record Layer: Handshake Protocol: Certificate
Content Type: Handshake (22)
...
TLSv1.2 Record Layer: Handshake Protocol: Server Key Exchange
Content Type: Handshake (22)
...
TLSv1.2 Record Layer: Handshake Protocol: Multiple Handshake Messages
Content Type: Handshake (22)
...
Handshake Protocol: Certificate Request
Handshake Type: Certificate Request (13)
Then, you should see a client certificate response from the Solace container:
Internet Protocol Version 4, Src: <solace-ip>, Dst: <remote-host-ip>
...
Secure Sockets Layer
TLSv1.2 Record Layer: Handshake Protocol: Certificate
Content Type: Handshake (22)
...
Handshake Protocol: Certificate
Handshake Type: Certificate (11)
...
Certificates (xxx bytes)
...
Certificate (...,id-at-commonName=...)

getting SSLCACertificateError in rails

I have a following single line code in rails:
Curl::Easy.new("https://www.uts.edu.au/" ).perform
Running this is giving following error:
Curl::Err::SSLCACertificateError: Peer certificate cannot be authenticated with given CA certificates
Can anyone please suggest how to fix it ?
For further help:
curl -v https://www.uts.edu.au/
* Adding handle: conn: 0x16a3a40
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x16a3a40) send_pipe: 1, recv_pipe: 0
* About to connect() to www.uts.edu.au port 443 (#0)
* Trying 54.79.20.73...
* Connected to www.uts.edu.au (54.79.20.73) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
For some reason, Curl is failing to use trusted CAs. By any chance, are you running this inside a Docker Container without CAs?
Please provide the output of curl -v https://www.uts.edu.au/
Update:
It seems there is a problem with the Chain of Certificates for the host you are making requests.
The certificate might lack necessary meta-data or its certificate
authority might be malfunctioning
https://whatsmychaincert.com/generate?host=www.uts.edu.au

NSURLSession Fails With SSL Page Using TLS 1.2

I'm using NSULConnection to send data to a webservice over SSL. The server is signed with a wildcard certificate (*.mydomain.com). The certificate is using RES SHA256 and TLS 1.2 and is signed by a CA. I'm trying to send my data using the following code:
NSMutableURLRequest *req = [[NSMutableURLRequest alloc] init];
[req setTimeoutInterval:60];
[req setHTTPMethod:#"POST"];
// ... Set content type and add data to body ... //
[req setURL:#"https://subdomain.mydomain.com/service/"];
NSURLSession *session = [NSURLSession sessionWithConfiguration:[NSURLSessionConfiguration defaultSessionConfiguration]];
NSLog(#"Starting Upload");
NSURLSessionDataTask *task = [session dataTaskWithRequest:req
completionHandler:^(NSData *data, NSURLResponse *response, NSError *error){
// .. Handle Completion .. //
}];
[task resume];
When I run the above code I get the following output.
CFNetwork SSLHandshake failed (-9801)
NSURLSession/NSURLConnection HTTP load failed (kCFStreamErrorDomainSSL, -9801)
So, I tried adding the exceptions explained in this post: CFNetwork SSLHandshake failed iOS 9
But had no luck in getting the error to resolve itself. I then switched to using http instead of https, while leaving in the configuration changes explained above and it did work. However, that is far from an ideal solution. The http is fine for testing, but this app will be handling data where SSL is required. How can I get the SSL working?
Edit
Here are the results of running nscurl --ats-diagnostics on the https version of my service url
================================================================================
Default ATS Secure Connection
---
ATS Default Connection
Result : PASS
---
================================================================================
Allowing Arbitrary Loads
---
Allow All Loads
Result : PASS
---
================================================================================
Configuring TLS exceptions for ****
---
TLSv1.2
Result : PASS
---
---
TLSv1.1
Result : PASS
---
---
TLSv1.0
Result : PASS
---
================================================================================
Configuring PFS exceptions for ****
---
Disabling Perfect Forward Secrecy
Result : PASS
---
================================================================================
Configuring PFS exceptions and allowing insecure HTTP for ****
---
Disabling Perfect Forward Secrecy and Allowing Insecure HTTP
Result : PASS
---
================================================================================
Configuring TLS exceptions with PFS disabled for ****
---
TLSv1.2 with PFS disabled
Result : PASS
---
---
TLSv1.1 with PFS disabled
Result : PASS
---
---
TLSv1.0 with PFS disabled
Result : PASS
---
================================================================================
Configuring TLS exceptions with PFS disabled and insecure HTTP allowed for ****
---
TLSv1.2 with PFS disabled and insecure HTTP allowed
Result : PASS
---
---
TLSv1.1 with PFS disabled and insecure HTTP allowed
Result : PASS
---
---
TLSv1.0 with PFS disabled and insecure HTTP allowed
Result : PASS
---
================================================================================
Edit 2
Here are the exceptions I added:
<dict>
<key>mydomain.com</key>
<dict>
<key>NSIncludesSubdomains</key>
<true/>
<key>NSTemporaryExceptionAllowsInsecureHTTPLoads</key>
<true/>
<key>NSTemporaryExceptionMinimumTLSVersion</key>
<string>1.0</string>
<key>NSTemporaryExceptionRequiresForwardSecrecy</key>
<false/>
</dict>
</dict>
Result of curl -v
* CAfile: /opt/local/share/curl/curl-ca-bundle.crt
CApath: none
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-SHA384
* Server certificate:
* subject: C=XXX; ST=XXX; L=XXX; O=XXX; CN=*.mydomain.com
* start date: 2015-08-17 00:00:00 GMT
* expire date: 2018-11-14 12:00:00 GMT
* subjectAltName: subdomain.mydomain.com matched
* issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
* SSL certificate verify ok.
Result of openssl s_client -connect
CONNECTED(00000003)
depth=1 C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
0 s:/C=XXX/ST=XXX/L=XXX/O=XXX/CN=*.mydomain.com
i:/C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
1 s:/C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
.
.
.
-----END CERTIFICATE-----
subject=/C=XXX/ST=XXX/L=XXX/O=XXX/CN=*.mydomain.com
issuer=/C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
---
No client certificate CA names sent
---
SSL handshake has read 3033 bytes and written 490 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-SHA384
Session-ID: XXX
Session-ID-ctx:
Master-Key: XXX
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1454971371
Timeout : 300 (sec)
Verify return code: 20 (unable to get local issuer certificate)
---
This answer saved my day! Turns out that ATS requires that server must support ciphers listed in cipher suite here. I can confirm that suggested fix works.
Verify return code: 20 (unable to get local issuer certificate)
It looks like the DigiCert Root CA is not properly loaded on your local machine. Open the KeyChain Access App, and check if it is either in the Login or System Roots section.

Why can't I get Twitter's OAuth Test to work?

I'm at the bottom of the learning curve for using the REST API v1.1. I am attempting to use cURL to test my Twitter App.
Here's what I've done so far on Twitter's dev apps webpages (starting at https://dev.twitter.com/apps), and a description of the stumbling block which has completely defeated me:
I created an App, then given it "read, write and direct messages" permissions. Then I regenerated the API keys and OAuth tokens.
I went to the app management page for my app and clicked "Test OAuth".
I want to test it with something simple, so under Request Settings, I choose Request Type "GET" and Request Query "account/verify_credentials".
Then I click "see OAuth Signature for this request".
Among the data that is presented to me, I see the curl command to use. It looks like this (edited to hide the keys and the signature):
curl --get 'https://api.twitter.com/1.1/' --data 'account%2Fverify_credentials=' --header 'Authorization: OAuth oauth_consumer_key="XXXXXXXXXXXXXXXXXXXX", oauth_nonce="f187b1a421600e88ffebeed19b6c1fbf", oauth_signature="XXXXXXXXXXXXXXXXXX", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1399538823", oauth_token="XXXXXXXXXXXXXXXXXXXXXXXX", oauth_version="1.0"' --verbose
So, I copy-and-paste this IMMEDIATELY (i.e. so that the timestamp does not expire) to my terminal (I'm using Debian), and I get the following response (again, edited to remove keys and signature), where you can see down the bottom it says, once again, "Invalid or expired token":
* About to connect() to api.twitter.com port 443 (#0)
* Trying 199.16.156.40... connected
* Connected to api.twitter.com (199.16.156.40) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-SHA
* Server certificate:
* subject: C=US; ST=California; L=San Francisco; O=Twitter, Inc.; OU=Twitter Security; CN=api.twitter.com
* start date: 2014-04-08 00:00:00 GMT
* expire date: 2014-10-10 23:59:59 GMT
* subjectAltName: api.twitter.com matched
* issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)10; CN=VeriSign Class 3 Secure Server CA - G3
* SSL certificate verify ok.
> GET /1.1/?account%2Fverify_credentials= HTTP/1.1
> User-Agent: curl/7.21.0 (i486-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.15 libssh2/1.2.6
> Host: api.twitter.com
> Accept: /
> Authorization: OAuth oauth_consumer_key="XXXXXXXXXXXXXXXXXX", oauth_nonce="f187b1a421600e88ffebeed19b6c1fbf", oauth_signature="XXXXXXXXXXXXXXXXXXXXXX", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1399538823", oauth_token="XXXXXXXXXXXXXXXXXXXXXXXXXXX", oauth_version="1.0"
>
< HTTP/1.1 401 Unauthorized
< content-length: 25
< content-type: text/plain
< date: Thu, 08 May 2014 08:47:38 UTC
< server: tfe
< set-cookie: guest_id=v1%3A139953885863095383; Domain=.twitter.com; Path=/; Expires=Sat, 07-May-2016 08:47:38 UTC
< strict-transport-security: max-age=631138519
<
Invalid or expired token
* Connection #0 to host api.twitter.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
What confuses me completely is that I have submitted Twitter's own curl command back to Twitter and it has rejected it!!!!
Any ideas??? I am completely stuck at this point.
Firstly, You will need to read the verify_credentials API document
In the link above, choose your created application and hit "Generate Oauth Signature" button.
You will see that while perform an OAuth request, you will need following 4 things:
Consumer key
Consumer secret
Access token
Access token secret
Also, do not put "account/verify_credentials" to Request query field, instead fill Request URI
with "https://api.twitter.com/1.1/account/verify_credentials.json".
This time the displayed curl should work fine :)

Resources