I'm attempting to download a .zip file from a BingAds URL and am struggling to bypass this error:
Connection reset by peer - SSL_connect
I have this code currently running in production through another app but am utilizing nitrous.io for new application on a chromebook and running off their default rails install (the nitrous box).
The code currently working in the other app:
class BingApi
def self.get_data(request_params={})
require 'zip'
#Acquire Bing report download URL
report_url = BingApi.acquire_report_url(report_request_id, request_params)
zip_file = open(report_url)
unzippedxml = Zip::File.open(zip_file) # open zip
entry = unzippedxml.entries.reject(&:directory?).first # take first non-directory
entry.get_input_stream{|is| is.read } # read file contents
end
The report_url will look something like: https://download.api.bingads.microsoft.com/ReportDownload/Download.aspx?q=cWmkJ72lVlzGEG%2fouLL8Xes2j6I5qVhLrnTqNIrW....
When visited, it will prompt the download of a .zip file which I unzip and then parse.
However, this same code on the chromebook utilizing the nitrous box gives me the Connection reset by peer - SSL_connect error
Errno::ECONNRESET - Connection reset by peer - SSL_connect:
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/net/http.rb:920:in `block in connect'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/timeout.rb:76:in `timeout'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/net/http.rb:920:in `connect'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/net/http.rb:863:in `do_start'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/net/http.rb:852:in `start'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/open-uri.rb:313:in `open_http'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/open-uri.rb:724:in `buffer_open'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/open-uri.rb:210:in `block in open_loop'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/open-uri.rb:208:in `open_loop'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/open-uri.rb:149:in `open_uri'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/open-uri.rb:704:in `open'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/open-uri.rb:34:in `open'
lib/bing_api.rb:25:in `get_data'
lib/bing_api.rb:224:in `get_and_parse'
I've been trying various different solutions:
zip_file = open(report_url, :ssl_verify_mode => OpenSSL::SSL::VERIFY_NONE)
OR
zip_file = Faraday.get(report_url, :ssl => false)
Finally I've bypassed it by using:
uri = URI.parse(report_url)
https = Net::HTTP.new(uri.host, uri.port)
https.open_timeout = 5
https.use_ssl = true
https.ssl_version = 'SSLv3'
request = Net::HTTP::Get.new(uri.request_uri)
zip_file = https.request(request)
but I can't pass the zip_file to unzippedxml = Zip::File.open(zip_file) or I get a no implicit conversion of Net::HTTPOK into String TypeError.
Am I missing something simple here? Should I do something different with that zip_file.class => Net::HTTPOK object?
I tried calling zip_file.body but what is returned looks like what you'd see if you tried to open an zipped file in an editor prior to unzipping.
The server is broken.
It only supports explicit TLS1.0 and SSL3.0 handshakes and does not support the commonly used and most compatible SSLv23 handshake. And even with explicit TLS1.0 handshake it can fail if you include the wrong or too much ciphers. The relevant output from analyze.pl:
* version SSLv23, no verification, ciphers= -> FAIL! SSL wants a read first
* version SSLv23, no verification, ciphers=HIGH:ALL -> FAIL! SSL wants a read first
* version TLSv1_2, no verification, ciphers= -> FAIL! SSL wants a read first
* version TLSv1_2, no verification, ciphers=HIGH:ALL -> FAIL! SSL wants a read first
* version TLSv1_1, no verification, ciphers= -> FAIL! SSL connect attempt failed error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
* version TLSv1_1, no verification, ciphers=HIGH:ALL -> FAIL! SSL wants a read first
* version TLSv1 no verification, ciphers= -> TLSv1,AES256-SHA
* version TLSv1, no verification, ciphers=HIGH:ALL -> FAIL! SSL wants a read first
* version SSLv3 no verification, ciphers= -> SSLv3,AES256-SHA
* version SSLv3 no verification, ciphers=HIGH:ALL -> SSLv3,AES256-SHA
* supported SSL versions with handshake used and preferred cipher(s):
* handshake protocols ciphers
* SSLv23 FAILED: SSL wants a read first
* TLSv1_2 FAILED: SSL wants a read first
* TLSv1_1 FAILED: SSL connect attempt failed error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number SSL wants a read first
* TLSv1 TLSv1 AES256-SHA
* SSLv3 SSLv3 AES256-SHA
As can be seen here SSLv23, TLSv1_2 and TLSv1_1 handshakes do not work and TLSv1 handshake does work but not if ciphers are HIGH:ALL (maybe too much ciphers or maybe unexpected ciphers are included). SSLv3 handshake then works stable.
Browsers work around this kind of behavior by trying multiple times while slowly downgrading the SSL/TLS protocol version used in the handshake. But apart from browsers practically nobody else is doing this. So any other application will usually fail unless they are specifically configured to use TLS1.0 or SSL3.0 handshakes with this server.
but I can't pass the zip_file to unzippedxml = Zip::File.open(zip_file) or I get a no implicit conversion of Net::HTTPOK into String TypeError.
At least the URL you've given only returns 404 not found. Please check that you've actually got a ZIP file as result. I don't know about this server, but often these kind of download links get dynamically created and are only valid if you've visited another site before and got a cookie there or a similar thing to tie the URL to your browser session.
Related
I have small Rails app that performs various checks on our platform and sends me an email in case of an issue. Everything was running fine until today i started getting alerts about the following error:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired)
Now the problem is the certificate in question is valid, it gets automatically renewed (Let's encrypt) and this code has been untouched for a couple of years and never had any issues before and suddenly this started to happen.
The code that throws the exception:
def get_request url
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
#more than 10 seconds this is too slow
http.open_timeout = 10
http.read_timeout = 10
request = Net::HTTP::Get.new(uri.request_uri)
response = http.request(request)
if response.code.to_i == 200
return true
else
puts "Failed to GET #{url}: #{response.code.to_i}"
return false
end
end
If i open the site in the browser, it shows the secure connection without issues and shows that is using a valid certificate, furthermore if i check with certbot i get the following: Expiry Date: 2021-11-22 17:48:58+00:00 (VALID: 52 days) so clearly the certificate is valid, why suddenly rails is throwing a tantrum about it?
Note that i have restarted Nginx just in case, that didn't help.
Additional info: Ubuntu 16.04.5, OpenSSL 1.0.2g 1 Mar 2016, Rails 4.2, Ruby 2.6.5
EDIT:
This error also happens with a different url, which also has a valid certificate.
EDIT 2:
I've isolated the problem, it is related to Let's Encrypt DST Root CA X3 that has expired. A lot of people are dealing with this issue, i'll report my solution once i find one.
So after reading through this long thread of the Let's Encrypt community, the solution for my case ended up being to remove the DST Root CA X3 certificate:
sudo rm /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt
sudo update-ca-certificates
After that no more errors from openssl.
I'm trying to connect a webcrawler that accesses a certain site via SSL and queries my data on that site. The authentication of this site is via a self-signed Digital Certificate. At the moment I want to access the site, I upload this certificate in .pfx format to my api, convert it to .pem, and when I try to access the site with this certificate, the response comes with status 403 (forbidden ).
However, when I try to access the site through a browser with the certificate in .pfx format I usually get it.
I already tried using Mechanize, and it worked for a while (until a few months ago it worked), but then it started to give the error:
SSL_connect returned = 1 errno = 0 state = SSLv3 read finished A: sslv3 alert bad certificate
The site is old, it does not receive updates frequently.
After that I already tried to use the net / http lib and the error persisted, I tried to use the httprb gem and lastly I tried with Faraday. All attempts ended either in that error quoted above or with the response status == 403.
What can I do to be able to connect? Is there something wrong with my script? Is it missing any information I need to get through?
Code:
# Faraday customs method:
class FaradayHttp
def with_openssl
system "openssl pkcs12 -in my-certificate-path -out certificate-output-path -nodes -password pass:certificate-password"
def cert_object
OpenSSL::X509::Certificate.new File.read("certificate-output-path")
end
# create PKey
def key_object
OpenSSL::PKey.read File.read("certificate-output-path")
end
faraday = Faraday::Connection.new 'https://example-site.com',
:ssl => {
certificate: cert_object,
private_key: key_object,
version: :SSLv3,
verify: false
}
faraday
end
end
# Controller that try to connect with the ssl server:
agent = FaradayHttp.new.with_openssl
page = agent.get '/login_path'
# mypki will prompt you for certificates
require 'mypki'
# faraday will use certificates from mypki
require 'faraday'
faraday = Faraday::Connection.new 'https://example-site.com'
faraday.get '/login_path'
Recently, I was trying to use rubygem grpc version 1.3.2 as a clinet and connect to a grpc server which is built from golang. I went through the documentation at GRPC.IO and used it in my code as it.
irb(main):017:0> GRPC::Core::Credentials.new(File.read(CA_FILE_PATH))
NameError: uninitialized constant GRPC::Core::Credentials
from (irb):17
from /usr/local/share/gems/gems/railties-4.2.1/lib/rails/commands/console.rb:110:in `start'
from /usr/local/share/gems/gems/railties-4.2.1/lib/rails/commands/console.rb:9:in `start'
from /usr/local/share/gems/gems/railties-4.2.1/lib/rails/commands/commands_tasks.rb:68:in `console'
from /usr/local/share/gems/gems/railties-4.2.1/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
from /usr/local/share/gems/gems/railties-4.2.1/lib/rails/commands.rb:17:in `<top (required)>'
from bin/rails:4:in `require'
from bin/rails:4:in `<main>'
However their documentation specifically says,
creds = GRPC::Core::Credentials.new(load_certs) # load_certs typically loads a CA roots file
stub = Helloworld::Greeter::Stub.new('myservice.example.com', creds)
Then I came across ChannelCredentials and the creds is supposed to be either ChannelCredentials object or a symbol(e.g. :this_channel_is_insecure). Hence, I gave it a try as well.
I've taken the following function from the grpc gem's source code itself. This function was called in rspec test cases for loading the certs:
def load_certs
data_dir = "#{Rails.root}/certs"
files = ['ca.pem', 'server.key', 'server.pem']
files.map { |f| File.open(File.join(data_dir, f)).read }
end
Then I gave it a try with,
channel_creds = GRPC::Core::ChannelCredentials.new(load_certs)
stub = Helloworld::Greeter::Stub.new('myservice.example.com', channel_creds)
But the above failed with
E0619 09:59:10.410575570 14208 ssl_transport_security.c:601] Could not load any root certificate.
E0619 09:59:10.410604954 14208 ssl_transport_security.c:1315] Cannot load server root certificates.
E0619 09:59:10.410622519 14208 security_connector.c:837] Handshaker factory creation failed with TSI_INVALID_ARGUMENT.
I also tried:
channel_creds = GRPC::Core::ChannelCredentials.new(File.read(CA_FILE_PATH))
stub = Helloworld::Greeter::Stub.new('myservice.example.com', creds)
But all I got was error from the logs or rpc server:
2017/06/16 10:52:34 transport: http2Server.HandleStreams failed to receive the preface from client: EOF
2017/06/16 10:53:35 transport: http2Server.HandleStreams failed to receive the preface from client: EOF
2017/06/16 10:53:59 transport: http2Server.HandleStreams failed to receive the preface from client: EOF
2017/06/16 10:55:06 transport: http2Server.HandleStreams failed to receive the preface from client: EOF
Has anyone successfully tried this Ruby client Golang server combination with SSL/TLS enabled?
creds is supposed to be either ChannelCredentials object or a symbol
Yes the second argument of a client stub constructor (the creds argument), should be either a GRPC::Core::ChannelCredentials object or specifically the ::this_channel_is_insecure symbol (if the latter is passed, an insecure connection will be used).
I'd note that the test that uses
def load_certs
data_dir = "#{Rails.root}/certs"
files = ['ca.pem', 'server.key', 'server.pem']
files.map { |f| File.open(File.join(data_dir, f)).read }
end
might actually be misleading, since it only makes sense to construct channel credentials with the client's private key and certificate chain (that specific test I believe doesn't use the key and cert chain).
On the GRPC::Core::ChannelCredentials constructor:
There are three forms that can be used, (there's a comment above the constructor code in https://github.com/grpc/grpc/blob/master/src/ruby/ext/grpc/rb_channel_credentials.c#L128 that goes over them), but the options are:
Credentials.new()
Credentials.new(pem_root_certs)
Credentials.new(pem_root_certs, pem_private_key, pem_cert_chain)
In all cases, the roots file, private key, and cert chain parameters are pem encoded strings.
Note that if no arguments are passed (Credentials.new() is used), then the server root certificates will be found as described in this header comment (see the description for the behavior when server root certs parameter is null). And the last constructor is only needed if you want the client to use a private key and cert chain.
I can confirm this works.
channel_creds = GRPC::Core::ChannelCredentials.new(File.read("/home/user/.lnd/tls.cert"))
stub = Lnrpc::Lightning::Stub.new("127.0.0.1:10009", channel_creds)
obj = Lnrpc::GetInfoRequest.new
pp stub.get_info(obj)
I am connecting to an external API using Ruby SSL two way authentication.
My latest script is here:
namespace :rnif_message do
# With Proxy
task send_test_index: :environment do
our_cert = File.read(File.join(Rails.root, 'ssl', 'invoice', 'test', 'cert20190116_ourcert.der'))
their_test_cert = File.read(File.join(Rails.root, 'ssl', 'invoice', 'test', 'testcert2016_theircert.der'))
cert_store = OpenSSL::X509::Store.new
# Contains their intermediate CA files
cert_store.add_path File.join(Rails.root, 'ssl', 'invoice', 'test', 'ca')
cert_store.add_cert OpenSSL::X509::Certificate.new(their_test_cert)
uri = URI("https://xml.digital.com/wm.rn/receive")
proxy_host = "us-static-02.qg.com"
proxy_port = "port"
proxy_user = "user"
proxy_pass = "pass"
proxy_request = Net::HTTP.new(uri.hostname, '443', proxy_host, proxy_port, proxy_user, proxy_pass)
proxy_request.verify_mode = OpenSSL::SSL::VERIFY_PEER
proxy_request.use_ssl = true
proxy_request.ssl_version = :TLSv1_2
proxy_request.ciphers = ["AES256-SHA:AES128-SHA:DES-CBC3-SHA"]
proxy_request.cert = OpenSSL::X509::Certificate.new(our_cert)
proxy_request.cert_store = cert_store
post_request = Net::HTTP::Post.new(uri)
response = proxy_request.request(post_request)
end
Response back (since I updated the ciphers) is now
OpenSSL::SSL::SSLError: SSL_connect SYSCALL returned=5 errno=0 state=unknown state
Instead of the older from my two previous questions
OpenSSL::SSL::SSLError: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read server hello A
# /Users/me/projects/proj/lib/tasks/rnif_message_builder.rake:217:in `block (2 levels) in <top (required)>'
Here is my latest wireshark
In the initial configuration of my certificate and IP on THEIR server configuration, I may have given them the wrong IP address, so I may be getting blocked by their firewall. Is there ways using openssl s_client I can test this?
So far i've been trying
openssl s_client -showcerts -connect xml.digitaloilfield.com:https
But I am not very familiar with using openssl s_client
Any help on troubleshooting this would be greatly appreciated!
Update
Thanks you very much for your help so far. I am experimenting with those commands you sent me and trying to see what info I can get from them to help me with this. Currently, after they changed my IP address and allowed me through the firewall, I am getting this
EOFError: end of file reached /Users/me/projects/xtiri/xtiri.com/lib/tasks/rnif_message_builder.rake:219:in `block (2 levels) in <top (required)>'
This will usually connect to nearly all servers. It uses TLS 1.2 and SNI. That should establish the TCP connection and start the TLS handshake. The handshake may fail later, but that's a different problem.
$ openssl s_client -connect xml.digitaloilfield.com:443 -tls1_2 \
-servername xml.digitaloilfield.com -debug
<hang>
connect: Connection timed out
connect:errno=110
However, while s_client is hanging, jump over to another terminal and issue:
$ sudo netstat -a | grep openssl
$
Netstat does not show you the SYN_SEND state, so use tcptrack:
$ sudo tcptrack -i eth0
# <next, use s_client>
172.16.2.4:43302 208.38.22.37:443 SYN_SENT 15s 0 B/s
You are in TCP's wait timer. The other side did not perform the three-way handshake with you. In fact, they did not acknowledge your SYN. There could be a few reasons for it, but ...
Given the target, it looks like you encountered a firewall. Rather than Reject'ing connections, it is Drop'ing connections. Its sometimes called "Stealth Mode"; it makes it appear there's no server running on the machine. That's consistent with OpenSSL's connect: Connection timed out message.
The problem could be with the proxy. You really want to run the tests from there, but you probably won't be able to. It could be you are using the ciphers, protocols and ports as specified by the remote site; but the proxy is doing its own thing. Also see Jarmock's SSL Interception Proxies and Transitive Trust.
Here are a couple of references:
How can I monitor network connections for an app
Is it better to set -j REJECT or -j DROP in iptables?
TCP 3-way handshake on the Wireshark Wiki
I am using https://github.com/ruby-ldap/ruby-net-ldap (net-ldap) gem to verify the authenticity of a user in my rails app. But before passing data to the ldap server, I need to verify that I am talking with the same secure server.
Is there a workaround which allows me to verify the certificate in ruby
Additional details: (things I have tried)
The certificate which is passed on to me is same as the one I see when I run
openssl s_client -showcerts -connect "<host>:<port>" </dev/null 2>/dev/null|openssl x509 -outform PEM
I used http://www.ldapsoft.com/ to connect to client's server
Unless I add the certificate file given to me in Security > Manage server certificates, I get a warning saying unknown security certificate
I tried do it manually first in plain ruby (without gem)
But i get following error
test-ssl.rb:23:in `connect': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (OpenSSL::SSL::SSLError)
Code:
cert_store = OpenSSL::X509::Store.new
cert_store.add_file "server-wildcard.crt"
io = TCPSocket.new("SECURELDAP.MYSITE.EDU","636")
ctx = OpenSSL::SSL::SSLContext.new
#ctx.cert = OpenSSL::X509::Certificate.new(File.read("server-wildcard.crt"))
#ctx.client_ca = OpenSSL::X509::Certificate.new(File.read("server-wildcard.crt"))
#ctx.ca_file = "server-wildcard.crt"
#ctx.ca_path = "./"
ctx.verify_mode = OpenSSL::SSL::VERIFY_PEER | OpenSSL::SSL::VERIFY_FAIL_IF_NO_PEER_CERT
ctx.cert_store = cert_store
conn = OpenSSL::SSL::SSLSocket.new(io, ctx)
conn.connect
I am posting my solution here for the sake of completeness.
net-ldap gem override to support certificate validation
https://gist.github.com/mintuhouse/9931865
Ideal Solution:
Maintain list of trusted root CAs on your server
(If you are lazy like me, have a cron job which will download (weekly maintained by curl) copy from http://curl.haxx.se/ca/cacert.pem)
Override Net::HTTP to always use this trusted certificate list
As of today (late 2016), ruby-net-ldap supports this upstream! However, tls_options needs to be passed with verify_mode set to a value other than the default VERIFY_NONE.
# optional: create/pass your own cert_store
cert_store = OpenSSL::X509::Store.new
cert_store.set_default_paths # or add your own CAdir, &c.
# attributes documented for OpenSSL::SSL::SSLContext are valid here
tls_options = {
verify_mode: OpenSSL::SSL::VERIFY_PEER
cert_store: cert_store
}
ldap = Net::LDAP.new(
:host => host,
:port => port,
:encryption => {
:method => :simple_tls, # could also be :start_tls
:tls_options => tls_options
}
)