Ruby SSL handshake not receiving Server Hello back - using proxy Net::HTTP - ruby-on-rails

I am connecting to an external API using Ruby SSL two way authentication.
My latest script is here:
namespace :rnif_message do
# With Proxy
task send_test_index: :environment do
our_cert = File.read(File.join(Rails.root, 'ssl', 'invoice', 'test', 'cert20190116_ourcert.der'))
their_test_cert = File.read(File.join(Rails.root, 'ssl', 'invoice', 'test', 'testcert2016_theircert.der'))
cert_store = OpenSSL::X509::Store.new
# Contains their intermediate CA files
cert_store.add_path File.join(Rails.root, 'ssl', 'invoice', 'test', 'ca')
cert_store.add_cert OpenSSL::X509::Certificate.new(their_test_cert)
uri = URI("https://xml.digital.com/wm.rn/receive")
proxy_host = "us-static-02.qg.com"
proxy_port = "port"
proxy_user = "user"
proxy_pass = "pass"
proxy_request = Net::HTTP.new(uri.hostname, '443', proxy_host, proxy_port, proxy_user, proxy_pass)
proxy_request.verify_mode = OpenSSL::SSL::VERIFY_PEER
proxy_request.use_ssl = true
proxy_request.ssl_version = :TLSv1_2
proxy_request.ciphers = ["AES256-SHA:AES128-SHA:DES-CBC3-SHA"]
proxy_request.cert = OpenSSL::X509::Certificate.new(our_cert)
proxy_request.cert_store = cert_store
post_request = Net::HTTP::Post.new(uri)
response = proxy_request.request(post_request)
end
Response back (since I updated the ciphers) is now
OpenSSL::SSL::SSLError: SSL_connect SYSCALL returned=5 errno=0 state=unknown state
Instead of the older from my two previous questions
OpenSSL::SSL::SSLError: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read server hello A
# /Users/me/projects/proj/lib/tasks/rnif_message_builder.rake:217:in `block (2 levels) in <top (required)>'
Here is my latest wireshark
In the initial configuration of my certificate and IP on THEIR server configuration, I may have given them the wrong IP address, so I may be getting blocked by their firewall. Is there ways using openssl s_client I can test this?
So far i've been trying
openssl s_client -showcerts -connect xml.digitaloilfield.com:https
But I am not very familiar with using openssl s_client
Any help on troubleshooting this would be greatly appreciated!
Update
Thanks you very much for your help so far. I am experimenting with those commands you sent me and trying to see what info I can get from them to help me with this. Currently, after they changed my IP address and allowed me through the firewall, I am getting this
EOFError: end of file reached /Users/me/projects/xtiri/xtiri.com/lib/tasks/rnif_message_builder.rake:219:in `block (2 levels) in <top (required)>'

This will usually connect to nearly all servers. It uses TLS 1.2 and SNI. That should establish the TCP connection and start the TLS handshake. The handshake may fail later, but that's a different problem.
$ openssl s_client -connect xml.digitaloilfield.com:443 -tls1_2 \
-servername xml.digitaloilfield.com -debug
<hang>
connect: Connection timed out
connect:errno=110
However, while s_client is hanging, jump over to another terminal and issue:
$ sudo netstat -a | grep openssl
$
Netstat does not show you the SYN_SEND state, so use tcptrack:
$ sudo tcptrack -i eth0
# <next, use s_client>
172.16.2.4:43302 208.38.22.37:443 SYN_SENT 15s 0 B/s
You are in TCP's wait timer. The other side did not perform the three-way handshake with you. In fact, they did not acknowledge your SYN. There could be a few reasons for it, but ...
Given the target, it looks like you encountered a firewall. Rather than Reject'ing connections, it is Drop'ing connections. Its sometimes called "Stealth Mode"; it makes it appear there's no server running on the machine. That's consistent with OpenSSL's connect: Connection timed out message.
The problem could be with the proxy. You really want to run the tests from there, but you probably won't be able to. It could be you are using the ciphers, protocols and ports as specified by the remote site; but the proxy is doing its own thing. Also see Jarmock's SSL Interception Proxies and Transitive Trust.
Here are a couple of references:
How can I monitor network connections for an app
Is it better to set -j REJECT or -j DROP in iptables?
TCP 3-way handshake on the Wireshark Wiki

Related

Ruby - Connect with SSL and authenticate via client certificate - sslv3 alert bad certificate

I'm trying to connect a webcrawler that accesses a certain site via SSL and queries my data on that site. The authentication of this site is via a self-signed Digital Certificate. At the moment I want to access the site, I upload this certificate in .pfx format to my api, convert it to .pem, and when I try to access the site with this certificate, the response comes with status 403 (forbidden ).
However, when I try to access the site through a browser with the certificate in .pfx format I usually get it.
I already tried using Mechanize, and it worked for a while (until a few months ago it worked), but then it started to give the error:
SSL_connect returned = 1 errno = 0 state = SSLv3 read finished A: sslv3 alert bad certificate
The site is old, it does not receive updates frequently.
After that I already tried to use the net / http lib and the error persisted, I tried to use the httprb gem and lastly I tried with Faraday. All attempts ended either in that error quoted above or with the response status == 403.
What can I do to be able to connect? Is there something wrong with my script? Is it missing any information I need to get through?
Code:
# Faraday customs method:
class FaradayHttp
def with_openssl
system "openssl pkcs12 -in my-certificate-path -out certificate-output-path -nodes -password pass:certificate-password"
def cert_object
OpenSSL::X509::Certificate.new File.read("certificate-output-path")
end
# create PKey
def key_object
OpenSSL::PKey.read File.read("certificate-output-path")
end
faraday = Faraday::Connection.new 'https://example-site.com',
:ssl => {
certificate: cert_object,
private_key: key_object,
version: :SSLv3,
verify: false
}
faraday
end
end
# Controller that try to connect with the ssl server:
agent = FaradayHttp.new.with_openssl
page = agent.get '/login_path'
# mypki will prompt you for certificates
require 'mypki'
# faraday will use certificates from mypki
require 'faraday'
faraday = Faraday::Connection.new 'https://example-site.com'
faraday.get '/login_path'

APNS_CERTIFICATE - Push Notification does not send in production

I've had this issue for about 2 weeks, when I suddenly stopped sending notifications in production. I am using the django-push-notifications library and by django admin I can send a test message, but it does not send messages through the system.
On my local computer, everything works flawlessly. I discovered a command to test the certificate:
openssl s_client -connect gateway.push.apple.com:2195 -cert apns-cert.pem
With this one I had the return: Timeout: 7200 (sec) Verify return
code: 20 (unable to get local issuer certificate) Extended master
secret: yes
So with a lot of research, I discovered that I needed to put the path of "CA":
openssl s_client -CApath /etc/ssl/certs/ -connect gateway.push.apple.com:2195 -cert apns-cert.pem
Who was taking me to: Verify return code: 0 (ok)
However, for use in the library, I needed to put the full path of a .pem file. Then I found this command:
ls /etc/ssl/certs/Entrust*
I tested all the .pem files that were there, until I reached what appeared to have worked perfectly:
openssl s_client -CAfile /etc/ssl/certs/Entrust.net_Premium_2048_Secure_Server_CA.pem -connect gateway.push.apple.com:2195 -cert apns-cert.pem
Soon, I formatted my PUSH_NOTIFICATIONS_SETTINGS:
PUSH_NOTIFICATIONS_SETTINGS = {
"GCM_API_KEY": "xxxx",
"APNS_CERTIFICATE": os.path.join(BASE_DIR, "apns-cert.pem"),
"APNS_CA_CERTIFICATES": "/etc/ssl/certs/Entrust.net_Premium_2048_Secure_Server_CA.pem",
"APNS_ERROR_TIMEOUT": 3,
}
IOS_VERIFY_RECEIPT_API = 'https://buy.itunes.apple.com/verifyReceipt'
ANDROID_VERIFY_RECEIPT_API = 'https://www.googleapis.com/androidpublisher/v2/applications/{packageName}/purchases/subscriptions/{subscriptionId}/tokens/{token}'
Unfortunately it still does not send PUSH, and no error because I have configured it to pop errors to send by email.
PS: Remembering that by sending a test text via django admin: OK. Sending via sandbox (debug): OK.
In fact it was not an SSL issue, it was a bulk upload error by the library.
The tokens registered in the system were expired and the library does not know how to work with it and canceled the action, causing no other token to be attempted. I corrected the problem by looping and ignoring the individual error by sending a test to my email:
def send_push(self):
errors = []
# IOS
queryset_ios = APNSDevice.objects.filter(user=self.authentication)
for device in queryset_ios:
try:
device.send_message(self.subject, badge=1, sound=self.kind.sound)
except APNSServerError as e:
errors.append(APNS_ERROR_MESSAGES[e.status])
except Exception:
pass
# ANDROID
queryset_android = GCMDevice.objects.filter(user=self.authentication)
extra = {'notification': self.pk, 'kind': self.kind.kind, 'sound': self.kind.sound}
for device in queryset_android:
try:
queryset_android.send_message(self.subject, badge=1, extra=extra)
except GCMError as e:
errors.append(str(e))
except Exception:
pass
if errors:
send_mail("Push Error",
"Push: %s \n User: %s \n\n Errors: %s" % (self.subject, self.authentication.full_name, errors),
settings.DEFAULT_FROM_EMAIL, ["my#mail.com"])

Connection reset by peer - SSL_connect when opening a URL

I'm attempting to download a .zip file from a BingAds URL and am struggling to bypass this error:
Connection reset by peer - SSL_connect
I have this code currently running in production through another app but am utilizing nitrous.io for new application on a chromebook and running off their default rails install (the nitrous box).
The code currently working in the other app:
class BingApi
def self.get_data(request_params={})
require 'zip'
#Acquire Bing report download URL
report_url = BingApi.acquire_report_url(report_request_id, request_params)
zip_file = open(report_url)
unzippedxml = Zip::File.open(zip_file) # open zip
entry = unzippedxml.entries.reject(&:directory?).first # take first non-directory
entry.get_input_stream{|is| is.read } # read file contents
end
The report_url will look something like: https://download.api.bingads.microsoft.com/ReportDownload/Download.aspx?q=cWmkJ72lVlzGEG%2fouLL8Xes2j6I5qVhLrnTqNIrW....
When visited, it will prompt the download of a .zip file which I unzip and then parse.
However, this same code on the chromebook utilizing the nitrous box gives me the Connection reset by peer - SSL_connect error
Errno::ECONNRESET - Connection reset by peer - SSL_connect:
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/net/http.rb:920:in `block in connect'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/timeout.rb:76:in `timeout'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/net/http.rb:920:in `connect'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/net/http.rb:863:in `do_start'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/net/http.rb:852:in `start'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/open-uri.rb:313:in `open_http'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/open-uri.rb:724:in `buffer_open'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/open-uri.rb:210:in `block in open_loop'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/open-uri.rb:208:in `open_loop'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/open-uri.rb:149:in `open_uri'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/open-uri.rb:704:in `open'
/home/action/.parts/packages/ruby2.1/2.1.1/lib/ruby/2.1.0/open-uri.rb:34:in `open'
lib/bing_api.rb:25:in `get_data'
lib/bing_api.rb:224:in `get_and_parse'
I've been trying various different solutions:
zip_file = open(report_url, :ssl_verify_mode => OpenSSL::SSL::VERIFY_NONE)
OR
zip_file = Faraday.get(report_url, :ssl => false)
Finally I've bypassed it by using:
uri = URI.parse(report_url)
https = Net::HTTP.new(uri.host, uri.port)
https.open_timeout = 5
https.use_ssl = true
https.ssl_version = 'SSLv3'
request = Net::HTTP::Get.new(uri.request_uri)
zip_file = https.request(request)
but I can't pass the zip_file to unzippedxml = Zip::File.open(zip_file) or I get a no implicit conversion of Net::HTTPOK into String TypeError.
Am I missing something simple here? Should I do something different with that zip_file.class => Net::HTTPOK object?
I tried calling zip_file.body but what is returned looks like what you'd see if you tried to open an zipped file in an editor prior to unzipping.
The server is broken.
It only supports explicit TLS1.0 and SSL3.0 handshakes and does not support the commonly used and most compatible SSLv23 handshake. And even with explicit TLS1.0 handshake it can fail if you include the wrong or too much ciphers. The relevant output from analyze.pl:
* version SSLv23, no verification, ciphers= -> FAIL! SSL wants a read first
* version SSLv23, no verification, ciphers=HIGH:ALL -> FAIL! SSL wants a read first
* version TLSv1_2, no verification, ciphers= -> FAIL! SSL wants a read first
* version TLSv1_2, no verification, ciphers=HIGH:ALL -> FAIL! SSL wants a read first
* version TLSv1_1, no verification, ciphers= -> FAIL! SSL connect attempt failed error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
* version TLSv1_1, no verification, ciphers=HIGH:ALL -> FAIL! SSL wants a read first
* version TLSv1 no verification, ciphers= -> TLSv1,AES256-SHA
* version TLSv1, no verification, ciphers=HIGH:ALL -> FAIL! SSL wants a read first
* version SSLv3 no verification, ciphers= -> SSLv3,AES256-SHA
* version SSLv3 no verification, ciphers=HIGH:ALL -> SSLv3,AES256-SHA
* supported SSL versions with handshake used and preferred cipher(s):
* handshake protocols ciphers
* SSLv23 FAILED: SSL wants a read first
* TLSv1_2 FAILED: SSL wants a read first
* TLSv1_1 FAILED: SSL connect attempt failed error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number SSL wants a read first
* TLSv1 TLSv1 AES256-SHA
* SSLv3 SSLv3 AES256-SHA
As can be seen here SSLv23, TLSv1_2 and TLSv1_1 handshakes do not work and TLSv1 handshake does work but not if ciphers are HIGH:ALL (maybe too much ciphers or maybe unexpected ciphers are included). SSLv3 handshake then works stable.
Browsers work around this kind of behavior by trying multiple times while slowly downgrading the SSL/TLS protocol version used in the handshake. But apart from browsers practically nobody else is doing this. So any other application will usually fail unless they are specifically configured to use TLS1.0 or SSL3.0 handshakes with this server.
but I can't pass the zip_file to unzippedxml = Zip::File.open(zip_file) or I get a no implicit conversion of Net::HTTPOK into String TypeError.
At least the URL you've given only returns 404 not found. Please check that you've actually got a ZIP file as result. I don't know about this server, but often these kind of download links get dynamically created and are only valid if you've visited another site before and got a cookie there or a similar thing to tie the URL to your browser session.

Ruby verify the certificate of secure ldap server

I am using https://github.com/ruby-ldap/ruby-net-ldap (net-ldap) gem to verify the authenticity of a user in my rails app. But before passing data to the ldap server, I need to verify that I am talking with the same secure server.
Is there a workaround which allows me to verify the certificate in ruby
Additional details: (things I have tried)
The certificate which is passed on to me is same as the one I see when I run
openssl s_client -showcerts -connect "<host>:<port>" </dev/null 2>/dev/null|openssl x509 -outform PEM
I used http://www.ldapsoft.com/ to connect to client's server
Unless I add the certificate file given to me in Security > Manage server certificates, I get a warning saying unknown security certificate
I tried do it manually first in plain ruby (without gem)
But i get following error
test-ssl.rb:23:in `connect': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (OpenSSL::SSL::SSLError)
Code:
cert_store = OpenSSL::X509::Store.new
cert_store.add_file "server-wildcard.crt"
io = TCPSocket.new("SECURELDAP.MYSITE.EDU","636")
ctx = OpenSSL::SSL::SSLContext.new
#ctx.cert = OpenSSL::X509::Certificate.new(File.read("server-wildcard.crt"))
#ctx.client_ca = OpenSSL::X509::Certificate.new(File.read("server-wildcard.crt"))
#ctx.ca_file = "server-wildcard.crt"
#ctx.ca_path = "./"
ctx.verify_mode = OpenSSL::SSL::VERIFY_PEER | OpenSSL::SSL::VERIFY_FAIL_IF_NO_PEER_CERT
ctx.cert_store = cert_store
conn = OpenSSL::SSL::SSLSocket.new(io, ctx)
conn.connect
I am posting my solution here for the sake of completeness.
net-ldap gem override to support certificate validation
https://gist.github.com/mintuhouse/9931865
Ideal Solution:
Maintain list of trusted root CAs on your server
(If you are lazy like me, have a cron job which will download (weekly maintained by curl) copy from http://curl.haxx.se/ca/cacert.pem)
Override Net::HTTP to always use this trusted certificate list
As of today (late 2016), ruby-net-ldap supports this upstream! However, tls_options needs to be passed with verify_mode set to a value other than the default VERIFY_NONE.
# optional: create/pass your own cert_store
cert_store = OpenSSL::X509::Store.new
cert_store.set_default_paths # or add your own CAdir, &c.
# attributes documented for OpenSSL::SSL::SSLContext are valid here
tls_options = {
verify_mode: OpenSSL::SSL::VERIFY_PEER
cert_store: cert_store
}
ldap = Net::LDAP.new(
:host => host,
:port => port,
:encryption => {
:method => :simple_tls, # could also be :start_tls
:tls_options => tls_options
}
)

How can I use a PEM certificate with password?

I'm having plenty of trouble trying to use a certificate that has a password on ruby. I'm using rest-client, but that's not a requirement.
This is the cURL-equivalent of the call I need to make:
curl -E certificate.pem:PASSWORD -d ident=language -d data="test" "https://theurl"
I tried many things, but I can't get the password part working. Here's what I have so far:
cert = OpenSSL::X509::Certificate.new(File.read("#{RAILS_ROOT}/certificate.pem"))
reply = RestClient.post("https://theurl", {:ident => 'language', :data => 'test'}, {:ssl_client_cert => cert})
I've tried putting the password everywhere, as :password and :ssl_client_key, I've looked through all documentation I could find but there's nowhere that will accept this password.
This is the error I always get:
SSL_connect returned=1 errno=0 state=SSLv3 read finished A: sslv3 alert handshake failure
What am I missing?
The way you uses curl with option -E, you are specifying a private key with a certificate.
(from the cURL man page)
-E/--cert
(SSL) Tells curl to use the specified
client certificate file when getting a
file with HTTPS, FTPS or another
SSL-based protocol. The certificate
must be in PEM format. If the optional
password isn't specified, it will be
queried for on the terminal. Note that
this option assumes a "certificate"
file that is the private key and the
private certificate concatenated! See
--cert and --key to specify them independently.
So in order to do the samething with RestClient, you can try using the ssl_client_key option. Like:
:ssl_client_key => OpenSSL::PKey::RSA.new(File.read("key.pem"), "passphrase, if any"),

Resources