I am trying to interact with third party real time Web messaging System created and maintained by Pusher.com. Now, i cannot send anything through the API unless i produce an HMAC SHA256 hex digest of my data. A sample source code written in ruby could try to illustrate this:
# Dependencies
# gem install ruby-hmac
#
require 'rubygems'
require 'hmac-sha2'
secret = '7ad3773142a6692b25b8'
string_to_sign = "POST\n/apps/3/channels/test_channel/events\nauth_key=278d425bdf160c739803&auth_timestamp=1272044395&auth_version=1.0&body_md5=7b3d404f5cde4a0b9b8fb4789a0098cb&name=foo"
hmac = HMAC::SHA256.hexdigest(secret, string_to_sign)
puts hmac
# >> 309fc4be20f04e53e011b00744642d3fe66c2c7c5686f35ed6cd2af6f202e445
I checked the erlang crypto Library and i cannot even generate a SHA256 hex digest "directly"
How do i do this whole thing in Erlang ? help....
* UPDATE *
I have found solutions here: sha256 encryption in erlang and they have led me to erlsha2. But still, how do i generate the HMAC of a SHA256 hexdigest output from this module ?
With erlsha2, use the following to get the equivalent of your Ruby code:
1> hmac:hexlify(hmac:hmac256(<<"7ad3773142a6692b25b8">>, <<"POST\n/apps/3/channels/test_channel/events\nauth_key=278d425bdf160c739803&auth_timestamp=1272044395&auth_version=1.0&body_md5=7b3d404f5cde4a0b9b8fb4789a0098cb&name=foo">>)).
"309FC4BE20F04E53E011B00744642D3FE66C2C7C5686F35ED6CD2AF6F202E445"
I just stumbled through this myself and finally managed it just using crypto, so thought I would share. For your usage I think you would want:
:crypto.hmac(:sha256, secret, string_to_sign) |> Base.encode16
The hmac portion should take care of digest + hmac and then piping to encode 16 should provide the hex part. I imagine you probably moved on some time ago, but since I just had the same issue and wanted to try and figure it out in stdlib stuff I thought I would share.
The same project (erlsha2) has a module for this:
https://github.com/vinoski/erlsha2/blob/master/src/hmac.erl
If you're using Elixir, you can use
:crypto.hash(:sha256, [secret, string_to_sign])
|> Base.encode16
|> String.downcase
This is a one-liner (Erlang 24):
[begin if N < 10 -> 48 + N; true -> 87 + N end end ||
<<N:4>> <= crypto:mac(hmac, sha256, Secret1, StringToSign1)].
>>> "309fc4be20f04e53e011b00744642d3fe66c2c7c5686f35ed6cd2af6f202e445"
No need for external libs.
Related
I am trying to decrypt a string which has been encrypted in my rails project. This is how I am encrypting the data:
def encrypt_text(text_To_encrypt)
# 0. generate the key using command openssl rand -hex 16 on linux machines
# 1. Read the secret from config
# 2. Read the salt from config
# 3. Encrypt the data
# 4. return the encypted data
# Ref: http://www.monkeyandcrow.com/blog/reading_rails_how_does_message_encryptor_work/
secret = Rails.configuration.miscconfig['encryption_key']
salt = Rails.configuration.miscconfig['encryption_salt']
key = ActiveSupport::KeyGenerator.new(secret).generate_key(salt, 32)
crypt = ActiveSupport::MessageEncryptor.new(key)
encrypted_data = crypt.encrypt_and_sign(text_To_encrypt)
encrypted_data
end
Now the issue is I am not able to decrypt it using openssl. It just shows bad magic number. Once I do that in open ssl, my plan is to decrypt it in golang.
Here is how I tried to descrypt it using openssl:
openssl enc -d -aes-256-cbc -salt -in encrypted.txt -out decrypted.txt -d -pass pass:<the key given in rails> -a
This just shows bad magic number
Trying to decrypt data encrypted in a different system will not work unless you are aware and deal with the many intricate details of how both systems do the cryptography. Although both Rails and the openssl command line tool use the OpenSSL libraries under the hood for their crypto operations, they both use it in their own distinct ways that are not directly interoperable.
If you look close to the two systems, you'll see that for example:
Rails message encryptor not only encrypts the message but also signs it
Rails encryptor uses Marshal to serialize the input data
the openssl enc tool expects the encrypted data in a distinct file format with a Salted__<salt> header (this is why you get the bad magic number message from openssl)
the openssl tool must be properly configured to use the same ciphers as Rails encryptor and key generator, as openssl defaults are different from Rails defaults
the default ciphers configuration changed significantly since Rails 5.2.
With this general info, we can have a look at a a practical example. It is tested in Rails 4.2 but should work equally up to Rails 5.1.
Anatomy of a Rails-encrypted message
Let me start with a slightly amended code that you presented. The only changes there are to preset the password and salt to static values and print a lot of debug info:
def encrypt_text(text_to_encrypt)
password = "password" # the password to derive the key
salt = "saltsalt" # salt must be 8 bytes
key = ActiveSupport::KeyGenerator.new(password).generate_key(salt, 32)
puts "salt (hexa) = #{salt.unpack('H*').first}" # print the saltin HEX
puts "key (hexa) = #{key.unpack('H*').first}" # print the generated key in HEX
crypt = ActiveSupport::MessageEncryptor.new(key)
output = crypt.encrypt_and_sign(text_to_encrypt)
puts "output (base64) = #{output}"
output
end
encrypt_text("secret text")
When you run this, you'll get something like the following output:
salt (hexa) = 73616c7473616c74
key (hexa) = 196827b250431e911310f5dbc82d395782837b7ae56230dce24e497cf07b6518
output (base64) = SGRTUXYxRys1N1haVWNpVWxxWTdCMHlyMk15SnQ0dWFBOCt3Z0djWVdBZz0tLTkrd1hBNWJMVm9HcnptZ3loOG1mNHc9PQ==--80d091e8799776113b2c0efd1bf75b344bf39994
The last line (output of the encrypt_and_sign method) is a combination of two parts separated by -- (see source):
the encrypted message (Base64-encoded) and
the message signature (Base64-encoded).
The signature is not important for encryption so let's take a look in the first part - let's decode it in Rails console:
> Base64.strict_decode64("SGRTUXYxRys1N1haVWNpVWxxWTdCMHlyMk15SnQ0dWFBOCt3Z0djWVdBZz0tLTkrd1hBNWJMVm9HcnptZ3loOG1mNHc9PQ==")
=> "HdSQv1G+57XZUciUlqY7B0yr2MyJt4uaA8+wgGcYWAg=--9+wXA5bLVoGrzmgyh8mf4w=="
You can see that the decoded message again consists of two Base64-encoded parts separated by -- (see source):
the encrypted message itself
the initialization vector used in the encryption
Rails message encryptor uses the aes-256-cbc cipher by default (note that this has changed since Rails 5.2). This cipher needs an initialization vector, which is randomly generated by Rails and must be present in the encrypted output so that we can use it together with the key to decipher the message.
Moreover, Rails does not encrypt the input data as a simple plain text, but rather a serialized version of the data, using the Marshal serializer by default (source). If we decrypted such serialized value with openssl, we would still get a slightly garbled (serialized) version of the initial plain text data. That's why it will be more appropriate to disable serialization while encrypting the data in Rails. This can be done by passing a parameter to the encryption method:
# crypt = ActiveSupport::MessageEncryptor.new(key)
crypt = ActiveSupport::MessageEncryptor.new(key, serializer: ActiveSupport::MessageEncryptor::NullSerializer)
A re-run of the code yields output that is slightly shorter than the previous version, because the encrypted data has not been serialized now:
salt (hexa) = 73616c7473616c74
key (hexa) = 196827b250431e911310f5dbc82d395782837b7ae56230dce24e497cf07b6518
output (base64) = SUlIWFBjSXRUc0JodEMzLzhXckJzUT09LS1oZGtPV1ZRc2I5Wi8zOG01dFNOdVdBPT0=--58bbaf983fd20459062df8b6c59eb470311cbca9
Finally, we must find out some info about the encryption key derivation procedure. The source tells us that the KeyGenerator uses the pbkdf2_hmac_sha1 algorithm with 2**16 = 65536 iterations to derive the key from the password / secret.
Anatomy of an openssl encrypted message
Now, a similar investigation is needed on the openssl side to learn the details of its decryption process. First, if you encrypt anything using the openssl enc tool, you will find out that the output has a distinct format:
Salted__<salt><encrypted_message>
It begins with the Salted__ magic string, then followed by the salt (in hex form) and finally followed by the encrypted data. To be able to decrypt any data using this tool, we must get our encrypted data into the same format.
The openssl tool uses the EVP_BytesToKey (see source) to derive the key by default but can be configured to use the pbkdf2_hmac_sha1 algorithm using the -pbkdf2 and -md sha1 options. The number of iterations can be set using the -iter option.
How to decrypt Rails-encrypted message in openssl
So, finally we have enough information to actually try to decrypt a Rails-encrypted message in openssl.
First we must decode the first part of the Rails-encrypted output again to get the encrypted data and the initialization vector:
> Base64.strict_decode64("SUlIWFBjSXRUc0JodEMzLzhXckJzUT09LS1oZGtPV1ZRc2I5Wi8zOG01dFNOdVdBPT0=")
=> "IIHXPcItTsBhtC3/8WrBsQ==--hdkOWVQsb9Z/38m5tSNuWA=="
Now let's take the IV (the second part) and convert it to a hexa string form, as that is the form that openssl needs:
> Base64.strict_decode64("hdkOWVQsb9Z/38m5tSNuWA==").unpack("H*").first
=> "85d90e59542c6fd67fdfc9b9b5236e58" # the initialization vector in hex form
Now we need to take the Rails-encrypted data and convert it to the format that openssl will recognize, i.e. prepend it with the magic string and salt and Base64-encode it again:
> Base64.strict_encode64("Salted__" + "saltsalt" + Base64.strict_decode64("IIHXPcItTsBhtC3/8WrBsQ=="))
=> "U2FsdGVkX19zYWx0c2FsdCCB1z3CLU7AYbQt//FqwbE=" # encrypted data suitable for openssl
Finally, we can construct the openssl command to decrypt the data:
$ echo "U2FsdGVkX19zYWx0c2FsdCCB1z3CLU7AYbQt//FqwbE=" |
> openssl enc -aes-256-cbc -d -iv 85d90e59542c6fd67fdfc9b9b5236e58 \
> -pass pass:password -pbkdf2 -iter 65536 -md sha1 -a
secret text
And voilá, we successfully decrypted the initial message!
The openssl parameters are as follows:
-aes-256-cbc sets the same cipher as Rails uses for encryption
-d stands for decryption
-iv passes the initialization vector in the hex string form
-pass pass:password sets the password used to derive the encryption key to "password"
-pbkdf2 and -md sha1 set the same key derivation algorithm as is used by Rails (pbkdf2_hmac_sha1)
-iter 65536 sets the same number of iterations for key derivation as was done in Rails
-a allows to work with Base64-encoded encrypted data - no need to handle raw bytes in files
By default openssl reads from STDIN, so we simply pass the encrypted data (in proper format) to openssl using echo.
debugging
In case you hit any problems when decrypting with openssl, it is useful to add the -P parameter to the command line, which outputs debugging info about the cipher / key parameters:
$ echo ... | openssl ... -P
salt=73616C7473616C74
key=196827B250431E911310F5DBC82D395782837B7AE56230DCE24E497CF07B6518
iv =85D90E59542C6FD67FDFC9B9B5236E58
The salt, key, and iv values must correspond to the debugging values printed by the original code in the encrypt_text method printed above. If they are different, you know you are doing something wrong...
Now, I guess you can expect similar problems when trying to decrypt the message in go but I think you have some good pointers now to start.
I'm updating a Python script to use cryptography's AESGCM primitive so it can interact with a Rails server running OpenSSL's AES-256-GCM implementation.
To begin, I'm simulating an encryption using identical message/key/nonce to see if both implementations produce the same output.
Python 3
import os
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
data = b"a secret message"
aad = None
key = b'7\x98\xc1\xdf\x7f}\xea5?\\6\x17\tlT\xed\xa2a\x0fn\x87.(\x0c\xe4;*4\xda\x8fY\xc8'
aesgcm = AESGCM(key)
nonce = b'\x8ch\xbe\xfcR\xeee\xc1g\xd6\x80\xda'
ct = aesgcm.encrypt(nonce, data, aad)
ct: b'\xa8\xda\xdd\xdc\xca\xe8X\x84\xdb\x85\xef\xa6\xa6\x95\x00PN\x1e\xe7\xb0\x88\xae\xddc0\x19_\xae\x7f\xfd\x0c.'
Rails
cipher = OpenSSL::Cipher.new('aes-256-gcm').encrypt
data = "a secret message".encode('utf-8')
cipher.key = "7\x98\xc1\xdf\x7f}\xea5?\\6\x17\tlT\xed\xa2a\x0fn\x87.(\x0c\xe4;*4\xda\x8fY\xc8"
cipher.iv = "\x8ch\xbe\xfcR\xeee\xc1g\xd6\x80\xda"
encrypted = cipher.update(data) + cipher.final
encrypted: "\xA8\xDA\xDD\xDC\xCA\xE8X\x84\xDB\x85\xEF\xA6\xA6\x95\x00P"
tag = cipher.auth_tag
tag: "\xB7B\x84h\xDD\xB7y\xCE\x88\xFDI\x9F\xD3\x13\xC51"
From the above examples:
Rails' encrypted is the same as the first part of Python's ct.
Rails' tag is not the same as the second part of Python's ct.
How do I amend one or both of these processes so they produce the same output?
Just found it - the answer lies in how OpenSSL differentially defines data vs auth_data.
The OpenSSL docs are a little confusing in the linked example because they didn't make it clear to me that data refers to the message and auth_data is additional authenticated data.
In my testing, I had mistakenly set auth_data AND data to 'a secret message', which is why data was encrypted consistently but the authenticated data bit at the end was different.
TLDR: data is your message; auth_data is not encrypted and should be set to "" if blank.
Given a LDAP password stored in SHA-1/{SSHA} how would I validate it in erlang.
For example - given the following {SSHA}:
% slappasswd -s myPassword
{SSHA}GEH5kMEQZHYHS95dgr6KmFdg0a4BicBP
%
How would I (in erlang) validate that clear text 'myPassword' matches with the hashed value of '{SSHA}GEH5kMEQZHYHS95dgr6KmFdg0a4BicBP'.
Passwords stored in a directory server are validated using the BIND operation. A properly configured and secured directory server will not allow access to password data; therefore LDAP clients must not be coded expecting that the password data is available, whether encrypted or hashed. LDAP clients must use the BIND operation to validate passwords.
After some help from others I've come up with a routine to do this in Erlang. Following up here to share with others.
First - this link (found in another post) gives functions in other languages doing what I wanted:
http://www.openldap.org/faq/data/cache/347.html
The trick was that the 'ldap {SSHA}' encoding is a salted-SHA1 hash which is also base64 encoded. So - you must decode it, extract the salt and then use that in the re-encoding of the 'clear password' for comparison.
Here is a short Erlang routine which does this:
validatessha(ClearPassword, SshaHash) ->
D64 = base64:decode(lists:nthtail(6, SshaHash)),
{HashedData, Salt} = lists:split(20, binary_to_list(D64)),
NewHash = crypto:sha(list_to_binary(ClearPassword ++ Salt)),
string:equal(binary_to_list(NewHash), HashedData).
Given the data in my original post - here's the output:
67> run:validatessha("myPassword", "{SSHA}GEH5kMEQZHYHS95dgr6KmFdg0a4BicBP").
true
68>
Thanx all.
Mike
My erlang is very rusty, so this isn't very pretty, but maybe it gets my idea along anyway.
run() ->
Password = "myPassword",
HashRaw = os:cmd("slappasswd -s " ++ Password),
Hash1 = lists:nthtail(6, HashRaw),
Hash2 = lists:concat ([integer_to_list(X, 16) || X <- binary_to_list(crypto:sha(Password))]),
string:equal(string:to_lower(Hash1),
string:to_lower(Hash2)).
My idea is that you:
Run the command whose output you are interested in verifying (slappasswd), save the output and trim away the extra decoration preceding the hash.
Run crypto:sha() from the erlang libraries. Take the binary output from this, and convert it to a list of integers, each of which you then convert to a hexadecimal string, which you then concatenate, thereby create Hash2.
Compare the output of your command to the output of crypto:sha()
EDIT: I don't have this command you're using, so I couldn't really try this very thoroughly.. But it works for sha1sum. I hope they are the same!
One thing I miss about ipython is it has a ? operator which diggs up the docs for a particular function.
I know ruby has a similar command line tool but it is extremely inconvenient to call it while I am in irb.
Does ruby/irb have anything similar?
Pry is a Ruby version of IPython, it supports the ? command to look up documentation on methods, but uses a slightly different syntax:
pry(main)> ? File.dirname
From: file.c in Ruby Core (C Method):
Number of lines: 6
visibility: public
signature: dirname()
Returns all components of the filename given in file_name
except the last one. The filename must be formed using forward
slashes (/'') regardless of the separator used on the
local file system.
File.dirname("/home/gumby/work/ruby.rb") #=> "/home/gumby/work"
You can also look up sourcecode with the $ command:
pry(main)> $ File.link
From: file.c in Ruby Core (C Method):
Number of lines: 14
static VALUE
rb_file_s_link(VALUE klass, VALUE from, VALUE to)
{
rb_secure(2);
FilePathValue(from);
FilePathValue(to);
from = rb_str_encode_ospath(from);
to = rb_str_encode_ospath(to);
if (link(StringValueCStr(from), StringValueCStr(to)) < 0) {
sys_fail2(from, to);
}
return INT2FIX(0);
}
See http://pry.github.com for more information :)
You can start with
irb(main):001:0> `ri Object`
Although the output of this is less than readable. You'd need to filter out some metacharacters.
In fact, someone already made a gem for it
gem install ori
Then in irb
irb(main):001:0> require 'ori'
=> true
irb(main):002:0> Object.ri
Looking up topics [Object] o
= Object < BasicObject
------------------------------------------------------------------------------
= Includes:
Java (from gem activesupport-3.0.9)
(from gem activesupport-3.0.9) [...]
No, it doesn't. Python has docstrings:
def my_method(arg1,arg2):
""" What's inside this string will be made available as the __doc__ attribute """
# some code
So, when the ? is called from ipython, it probably calls the __doc__ attribute on the object. Ruby doesn't have this.
I have a Rails application that has to co-exist with a very old legacy application. The legacy application looks for a cookie that has a value containing a specific string of characters. Unfortunately, the characters in the legacy cookie often contain slashes. The problem I have is that when the Rails application writes the cookie it first does URL-encoding which causes the legacy app to break because the cookie values is incorrect.
I had this working in Rails 1.13.5 by editing the file cookie_performance_fix.rb (Path: ./actionpack-1.13.5/lib/action_controller/cgi_ext/cookie_performance_fix.rb)
In order to get this to work I changed the code as shown:
def to_s
buf = ""
buf << #name << '='
if #value.kind_of?(String)
rails code.
#buf << CGI::escape(#value)
buf << #value
else
#buf << #value.collect{|v| CGI::escape(v) }.join("&")
buf << #value.collect{|v| (v) }.join("&")
end
This actually worked fine until I decided to upgrade Rails to version 2.3.2
In Rails 2.3.2 the cookie_performance_fix.rb file no longer exists. I looked in the same directory and found a file called cookie.rb which I tried modifying in a similar fashion.
def to_s
buf = ''
buf << #name << '='
#buf << (#value.kind_of?(String) ? CGI::escape(#value) : #value.collect{|v| CGI::escape(v) }.join("&"))
buf << (#value.kind_of?(String) ? #value : #value.collect{|v| (v) }.join("&"))
buf << '; domain=' << #domain if #domain
buf << '; path=' << #path if #path
buf << '; expires=' << CGI::rfc1123_date(#expires) if #expires
buf << '; secure' if #secure
buf << '; HttpOnly' if #http_only
buf
end
This unfortunately does not seem to work. The cookie keeps getting URL-encoded in the new Rails 2.3.2. I know that turning off URL-encoding is not the best idea, but I don't have much choice until the legacy application is retired. I unfortunately do not have access to the legacy code to add support for URL-unencoding the cookie so I have to make sure the legacy cookie is written with the correct sequence including the slashes. If anyone can tell me how to turn off URL-encoding in Rails 2.3.2 it would be greatly appreciated.
Thanks.
After doing some digging I have found the answer to my question and I am documenting it here in case it is of use to anyone else.
In order to turn off URL-encoding in Rails 2.3.2 it is necessary to edit the following file: actionpack-2.3.2/lib/action_controller/vendor/rack-1.0/rack/response.rb
Around line 70 the ID and value of the cookie is set. I made the following change to turn of URL-encoding:
cookie = Utils.escape(key) + "=" +
#value.map { |v| Utils.escape v }.join("&") +
value.map { |v| v }.join("&") +
"#{domain}#{path}#{expires}#{secure}#{httponly}"
NOTE: This modification only affects standard cookies - not the cookies used as session data by Rails in version 2.3.2.
DISCLAIMER: I am in no way recommending this modification as a best practice. This modification was only done for the specific reason of handling legacy code requirement that required a cookie to be in a particular format. A better option would even be to modify the legacy code to handle URL-encoding. Unfortunately, that option was closed to me so I was forced to hack around on the underlying Rails code - which is not something I would generally recommend. Of course it should go without saying that making this type of modification runs the risk that the problem will have to be re-addressed every time you upgrade your Rails installation as the underlying code may change. That is actually what happened in my case. And of course there are also probably good reasons (security, standards compliance, etc.) for keeping the URL-encoding if at all possible.
A simple way to do this is using the rack method,
response["set-cookie"]="id_cookie=this cookie will be not escaped"