Related
I am trying to decrypt a string which has been encrypted in my rails project. This is how I am encrypting the data:
def encrypt_text(text_To_encrypt)
# 0. generate the key using command openssl rand -hex 16 on linux machines
# 1. Read the secret from config
# 2. Read the salt from config
# 3. Encrypt the data
# 4. return the encypted data
# Ref: http://www.monkeyandcrow.com/blog/reading_rails_how_does_message_encryptor_work/
secret = Rails.configuration.miscconfig['encryption_key']
salt = Rails.configuration.miscconfig['encryption_salt']
key = ActiveSupport::KeyGenerator.new(secret).generate_key(salt, 32)
crypt = ActiveSupport::MessageEncryptor.new(key)
encrypted_data = crypt.encrypt_and_sign(text_To_encrypt)
encrypted_data
end
Now the issue is I am not able to decrypt it using openssl. It just shows bad magic number. Once I do that in open ssl, my plan is to decrypt it in golang.
Here is how I tried to descrypt it using openssl:
openssl enc -d -aes-256-cbc -salt -in encrypted.txt -out decrypted.txt -d -pass pass:<the key given in rails> -a
This just shows bad magic number
Trying to decrypt data encrypted in a different system will not work unless you are aware and deal with the many intricate details of how both systems do the cryptography. Although both Rails and the openssl command line tool use the OpenSSL libraries under the hood for their crypto operations, they both use it in their own distinct ways that are not directly interoperable.
If you look close to the two systems, you'll see that for example:
Rails message encryptor not only encrypts the message but also signs it
Rails encryptor uses Marshal to serialize the input data
the openssl enc tool expects the encrypted data in a distinct file format with a Salted__<salt> header (this is why you get the bad magic number message from openssl)
the openssl tool must be properly configured to use the same ciphers as Rails encryptor and key generator, as openssl defaults are different from Rails defaults
the default ciphers configuration changed significantly since Rails 5.2.
With this general info, we can have a look at a a practical example. It is tested in Rails 4.2 but should work equally up to Rails 5.1.
Anatomy of a Rails-encrypted message
Let me start with a slightly amended code that you presented. The only changes there are to preset the password and salt to static values and print a lot of debug info:
def encrypt_text(text_to_encrypt)
password = "password" # the password to derive the key
salt = "saltsalt" # salt must be 8 bytes
key = ActiveSupport::KeyGenerator.new(password).generate_key(salt, 32)
puts "salt (hexa) = #{salt.unpack('H*').first}" # print the saltin HEX
puts "key (hexa) = #{key.unpack('H*').first}" # print the generated key in HEX
crypt = ActiveSupport::MessageEncryptor.new(key)
output = crypt.encrypt_and_sign(text_to_encrypt)
puts "output (base64) = #{output}"
output
end
encrypt_text("secret text")
When you run this, you'll get something like the following output:
salt (hexa) = 73616c7473616c74
key (hexa) = 196827b250431e911310f5dbc82d395782837b7ae56230dce24e497cf07b6518
output (base64) = SGRTUXYxRys1N1haVWNpVWxxWTdCMHlyMk15SnQ0dWFBOCt3Z0djWVdBZz0tLTkrd1hBNWJMVm9HcnptZ3loOG1mNHc9PQ==--80d091e8799776113b2c0efd1bf75b344bf39994
The last line (output of the encrypt_and_sign method) is a combination of two parts separated by -- (see source):
the encrypted message (Base64-encoded) and
the message signature (Base64-encoded).
The signature is not important for encryption so let's take a look in the first part - let's decode it in Rails console:
> Base64.strict_decode64("SGRTUXYxRys1N1haVWNpVWxxWTdCMHlyMk15SnQ0dWFBOCt3Z0djWVdBZz0tLTkrd1hBNWJMVm9HcnptZ3loOG1mNHc9PQ==")
=> "HdSQv1G+57XZUciUlqY7B0yr2MyJt4uaA8+wgGcYWAg=--9+wXA5bLVoGrzmgyh8mf4w=="
You can see that the decoded message again consists of two Base64-encoded parts separated by -- (see source):
the encrypted message itself
the initialization vector used in the encryption
Rails message encryptor uses the aes-256-cbc cipher by default (note that this has changed since Rails 5.2). This cipher needs an initialization vector, which is randomly generated by Rails and must be present in the encrypted output so that we can use it together with the key to decipher the message.
Moreover, Rails does not encrypt the input data as a simple plain text, but rather a serialized version of the data, using the Marshal serializer by default (source). If we decrypted such serialized value with openssl, we would still get a slightly garbled (serialized) version of the initial plain text data. That's why it will be more appropriate to disable serialization while encrypting the data in Rails. This can be done by passing a parameter to the encryption method:
# crypt = ActiveSupport::MessageEncryptor.new(key)
crypt = ActiveSupport::MessageEncryptor.new(key, serializer: ActiveSupport::MessageEncryptor::NullSerializer)
A re-run of the code yields output that is slightly shorter than the previous version, because the encrypted data has not been serialized now:
salt (hexa) = 73616c7473616c74
key (hexa) = 196827b250431e911310f5dbc82d395782837b7ae56230dce24e497cf07b6518
output (base64) = SUlIWFBjSXRUc0JodEMzLzhXckJzUT09LS1oZGtPV1ZRc2I5Wi8zOG01dFNOdVdBPT0=--58bbaf983fd20459062df8b6c59eb470311cbca9
Finally, we must find out some info about the encryption key derivation procedure. The source tells us that the KeyGenerator uses the pbkdf2_hmac_sha1 algorithm with 2**16 = 65536 iterations to derive the key from the password / secret.
Anatomy of an openssl encrypted message
Now, a similar investigation is needed on the openssl side to learn the details of its decryption process. First, if you encrypt anything using the openssl enc tool, you will find out that the output has a distinct format:
Salted__<salt><encrypted_message>
It begins with the Salted__ magic string, then followed by the salt (in hex form) and finally followed by the encrypted data. To be able to decrypt any data using this tool, we must get our encrypted data into the same format.
The openssl tool uses the EVP_BytesToKey (see source) to derive the key by default but can be configured to use the pbkdf2_hmac_sha1 algorithm using the -pbkdf2 and -md sha1 options. The number of iterations can be set using the -iter option.
How to decrypt Rails-encrypted message in openssl
So, finally we have enough information to actually try to decrypt a Rails-encrypted message in openssl.
First we must decode the first part of the Rails-encrypted output again to get the encrypted data and the initialization vector:
> Base64.strict_decode64("SUlIWFBjSXRUc0JodEMzLzhXckJzUT09LS1oZGtPV1ZRc2I5Wi8zOG01dFNOdVdBPT0=")
=> "IIHXPcItTsBhtC3/8WrBsQ==--hdkOWVQsb9Z/38m5tSNuWA=="
Now let's take the IV (the second part) and convert it to a hexa string form, as that is the form that openssl needs:
> Base64.strict_decode64("hdkOWVQsb9Z/38m5tSNuWA==").unpack("H*").first
=> "85d90e59542c6fd67fdfc9b9b5236e58" # the initialization vector in hex form
Now we need to take the Rails-encrypted data and convert it to the format that openssl will recognize, i.e. prepend it with the magic string and salt and Base64-encode it again:
> Base64.strict_encode64("Salted__" + "saltsalt" + Base64.strict_decode64("IIHXPcItTsBhtC3/8WrBsQ=="))
=> "U2FsdGVkX19zYWx0c2FsdCCB1z3CLU7AYbQt//FqwbE=" # encrypted data suitable for openssl
Finally, we can construct the openssl command to decrypt the data:
$ echo "U2FsdGVkX19zYWx0c2FsdCCB1z3CLU7AYbQt//FqwbE=" |
> openssl enc -aes-256-cbc -d -iv 85d90e59542c6fd67fdfc9b9b5236e58 \
> -pass pass:password -pbkdf2 -iter 65536 -md sha1 -a
secret text
And voilá, we successfully decrypted the initial message!
The openssl parameters are as follows:
-aes-256-cbc sets the same cipher as Rails uses for encryption
-d stands for decryption
-iv passes the initialization vector in the hex string form
-pass pass:password sets the password used to derive the encryption key to "password"
-pbkdf2 and -md sha1 set the same key derivation algorithm as is used by Rails (pbkdf2_hmac_sha1)
-iter 65536 sets the same number of iterations for key derivation as was done in Rails
-a allows to work with Base64-encoded encrypted data - no need to handle raw bytes in files
By default openssl reads from STDIN, so we simply pass the encrypted data (in proper format) to openssl using echo.
debugging
In case you hit any problems when decrypting with openssl, it is useful to add the -P parameter to the command line, which outputs debugging info about the cipher / key parameters:
$ echo ... | openssl ... -P
salt=73616C7473616C74
key=196827B250431E911310F5DBC82D395782837B7AE56230DCE24E497CF07B6518
iv =85D90E59542C6FD67FDFC9B9B5236E58
The salt, key, and iv values must correspond to the debugging values printed by the original code in the encrypt_text method printed above. If they are different, you know you are doing something wrong...
Now, I guess you can expect similar problems when trying to decrypt the message in go but I think you have some good pointers now to start.
Using the most basic setup:
class User < ActiveRecord::Base
attr_encrypted :name,
key: 'This is a key that is 256 bits!!',
encode: true,
encode_iv: true,
encode_salt: true
end
The results look like this in the database when supplying an identical name:
╔════╦══════════════════════════════╦═══════════════════╗
║ id ║ encrypted_name ║ encrypted_name_iv ║
╠════╬══════════════════════════════╬═══════════════════╣
║ 1 ║ aVXZb1b317nroumXVBdV9pGxA2o= ║ JyE7wHups+3upY5e ║
║ 2 ║ aVXZb1b317nroumXVBdV9pGxA2o= ║ uz/ktrtbUAksg5Vp ║
╚════╩══════════════════════════════╩═══════════════════╝
Why is the ciphertext identical? Isn't that the part of the point of iv, which the gem is using by default?
Update: the following is the original post explaining the whole problem, the issue is fixed now, see the bottom of this answer for a solution.
I am quite sure you noticed a rather nasty security issue in the encryptor gem (the gem that is used by attr_encrypted to do the actual encryptions).
The problem is that when using the aes-256-gcm algorithm (or any of the AES GCM algorithms), the initialization vector (IV) is currently indeed not taken into account when encrypting. The issue does not affect other algorithms but unfortunately the aes-256-gcm is the default algorithm in attr_encrypted.
As it turns out, it is the order of setting the IV vs. the encryption key what causes the issue. When IV is set before the key (as is in the gem), the IV is not taken into account but it is if set after the key.
Some tests to prove the problem:
While taking parts of the encryptor gem code, I created the simplest test case to prove the problem (tested under ruby 2.3.0 compiled against OpenSSL version "1.0.1f 6 Jan 2014"):
def base64_enc(bytes)
[bytes].pack("m")
end
def test_aes_encr(n, cipher, data, key, iv, iv_before_key = true)
cipher = OpenSSL::Cipher.new(cipher)
cipher.encrypt
# THIS IS THE KEY PART OF THE ISSUE
if iv_before_key
# this is how it's currently present in the encryptor gem code
cipher.iv = iv
cipher.key = key
else
# this is the version that actually works
cipher.key = key
cipher.iv = iv
end
if cipher.name.downcase.end_with?("gcm")
cipher.auth_data = ""
end
result = cipher.update(data)
result << cipher.final
puts "#{n} #{cipher.name}, iv #{iv_before_key ? "BEFORE" : "AFTER "} key: " +
"iv=#{iv}, result=#{base64_enc(result)}"
end
def test_encryption
data = "something private"
key = "This is a key that is 256 bits!!"
# control tests using AES-256-CBC
test_aes_encr(1, "aes-256-cbc", data, key, "aaaabbbbccccdddd", true)
test_aes_encr(2, "aes-256-cbc", data, key, "eeeeffffgggghhhh", true)
test_aes_encr(3, "aes-256-cbc", data, key, "aaaabbbbccccdddd", false)
test_aes_encr(4, "aes-256-cbc", data, key, "eeeeffffgggghhhh", false)
# failing tests using AES-256-GCM
test_aes_encr(5, "aes-256-gcm", data, key, "aaaabbbbcccc", true)
test_aes_encr(6, "aes-256-gcm", data, key, "eeeeffffgggg", true)
test_aes_encr(7, "aes-256-gcm", data, key, "aaaabbbbcccc", false)
test_aes_encr(8, "aes-256-gcm", data, key, "eeeeffffgggg", false)
end
Running test_encryption which encrypts a text using AES-256-CBC and then using AES-256-GCM, each time with two different IVs in two regimes (IV set before/after key), gets us the following results:
# control tests with CBC:
1 AES-256-CBC, iv BEFORE key: iv=aaaabbbbccccdddd, result=4IAGcazRmEUIRDE3ZpEgoS0Nmm1/+nrd5VT2/Xab0WM=
2 AES-256-CBC, iv BEFORE key: iv=eeeeffffgggghhhh, result=T7um2Wgb2vw1r4uryF3xnBeq+KozuetjKGItfNKurGE=
3 AES-256-CBC, iv AFTER key: iv=aaaabbbbccccdddd, result=4IAGcazRmEUIRDE3ZpEgoS0Nmm1/+nrd5VT2/Xab0WM=
4 AES-256-CBC, iv AFTER key: iv=eeeeffffgggghhhh, result=T7um2Wgb2vw1r4uryF3xnBeq+KozuetjKGItfNKurGE=
# the problematic tests with GCM:
5 id-aes256-GCM, iv BEFORE key: iv=aaaabbbbcccc, result=Tl/HfkWpwoByeYRz6Mz4yIo=
6 id-aes256-GCM, iv BEFORE key: iv=eeeeffffgggg, result=Tl/HfkWpwoByeYRz6Mz4yIo=
7 id-aes256-GCM, iv AFTER key: iv=aaaabbbbcccc, result=+4Iyn7RSDKimTQi0S3gn58E=
8 id-aes256-GCM, iv AFTER key: iv=eeeeffffgggg, result=3m9uEDyb9eh1RD3CuOCmc50=
These tests show that while the order of setting IV vs. key is not relevant for CBC, it is for GCM. More importantly, the encrypted result in CBC is different for two different IVs, whereas it is not for GCM if IV set before the key.
I just created a pull request to fix this issue in the encryptor gem. Practically, you have a few options now:
Wait till a new version of the encryptor gem is released.
Use also salt with attr_encrypted. You should use salt anyway to further secure the encrypted data.
The very unfortunate thing is that all already encrypted data will become undecipherable after the fix as suddenly the IVs will be taken into account.
Update: encryptor 3.0.0 available
You can now upgrade the encryptor gem to version 3.0 in which the bug is fixed. Now, if this is the first time you use the encryptor or attr_encrypted gems you are all set and everything should work correctly.
If you have data that is already encrypted using encryptor 2.0.0, then you must manually re-encrypt the data after the gem upgrade, otherwise it will fail to decrypt correctly! You will be warned about this during the gem upgrade. The schematic procedure is as follows:
You have to decrypt all your encrypted data using the Encryptor class (see the README for examples), using the :v2_gcm_iv => true option. This should correctly decrypt your data.
Then you must encrypt the same data back again, now without this option (i.e. :v2_gcm_iv => false) but including the proper IV from your database.
If you have production data, you will need to do this upgrade offline and immediately after the gem update to ensure no data corruption.
Update 2: issue in the openssl gem confirmed and fixed
FYI, it was recently confirmed that this had actually been an issue in the underlying ruby-openssl library and the bug has been fixed now. So, in the future, it is possible that even attr_encrypted gem version 2.x will actually work correctly when used with the new openssl-2.0.0 gem version (which is now in beta as of Sep 2016).
If the plain text, key and IV are the same the encrypted text will be the same.
It looks like you need to use:
attr_encrypted :email, key: 'some secret key', encode: true, encode_iv: true, encode_salt: true
Note: encode_iv: true
Or perhaps set the default option encode_iv: true
Documentation: attr_encrypted
Given a LDAP password stored in SHA-1/{SSHA} how would I validate it in erlang.
For example - given the following {SSHA}:
% slappasswd -s myPassword
{SSHA}GEH5kMEQZHYHS95dgr6KmFdg0a4BicBP
%
How would I (in erlang) validate that clear text 'myPassword' matches with the hashed value of '{SSHA}GEH5kMEQZHYHS95dgr6KmFdg0a4BicBP'.
Passwords stored in a directory server are validated using the BIND operation. A properly configured and secured directory server will not allow access to password data; therefore LDAP clients must not be coded expecting that the password data is available, whether encrypted or hashed. LDAP clients must use the BIND operation to validate passwords.
After some help from others I've come up with a routine to do this in Erlang. Following up here to share with others.
First - this link (found in another post) gives functions in other languages doing what I wanted:
http://www.openldap.org/faq/data/cache/347.html
The trick was that the 'ldap {SSHA}' encoding is a salted-SHA1 hash which is also base64 encoded. So - you must decode it, extract the salt and then use that in the re-encoding of the 'clear password' for comparison.
Here is a short Erlang routine which does this:
validatessha(ClearPassword, SshaHash) ->
D64 = base64:decode(lists:nthtail(6, SshaHash)),
{HashedData, Salt} = lists:split(20, binary_to_list(D64)),
NewHash = crypto:sha(list_to_binary(ClearPassword ++ Salt)),
string:equal(binary_to_list(NewHash), HashedData).
Given the data in my original post - here's the output:
67> run:validatessha("myPassword", "{SSHA}GEH5kMEQZHYHS95dgr6KmFdg0a4BicBP").
true
68>
Thanx all.
Mike
My erlang is very rusty, so this isn't very pretty, but maybe it gets my idea along anyway.
run() ->
Password = "myPassword",
HashRaw = os:cmd("slappasswd -s " ++ Password),
Hash1 = lists:nthtail(6, HashRaw),
Hash2 = lists:concat ([integer_to_list(X, 16) || X <- binary_to_list(crypto:sha(Password))]),
string:equal(string:to_lower(Hash1),
string:to_lower(Hash2)).
My idea is that you:
Run the command whose output you are interested in verifying (slappasswd), save the output and trim away the extra decoration preceding the hash.
Run crypto:sha() from the erlang libraries. Take the binary output from this, and convert it to a list of integers, each of which you then convert to a hexadecimal string, which you then concatenate, thereby create Hash2.
Compare the output of your command to the output of crypto:sha()
EDIT: I don't have this command you're using, so I couldn't really try this very thoroughly.. But it works for sha1sum. I hope they are the same!
I am trying to interact with third party real time Web messaging System created and maintained by Pusher.com. Now, i cannot send anything through the API unless i produce an HMAC SHA256 hex digest of my data. A sample source code written in ruby could try to illustrate this:
# Dependencies
# gem install ruby-hmac
#
require 'rubygems'
require 'hmac-sha2'
secret = '7ad3773142a6692b25b8'
string_to_sign = "POST\n/apps/3/channels/test_channel/events\nauth_key=278d425bdf160c739803&auth_timestamp=1272044395&auth_version=1.0&body_md5=7b3d404f5cde4a0b9b8fb4789a0098cb&name=foo"
hmac = HMAC::SHA256.hexdigest(secret, string_to_sign)
puts hmac
# >> 309fc4be20f04e53e011b00744642d3fe66c2c7c5686f35ed6cd2af6f202e445
I checked the erlang crypto Library and i cannot even generate a SHA256 hex digest "directly"
How do i do this whole thing in Erlang ? help....
* UPDATE *
I have found solutions here: sha256 encryption in erlang and they have led me to erlsha2. But still, how do i generate the HMAC of a SHA256 hexdigest output from this module ?
With erlsha2, use the following to get the equivalent of your Ruby code:
1> hmac:hexlify(hmac:hmac256(<<"7ad3773142a6692b25b8">>, <<"POST\n/apps/3/channels/test_channel/events\nauth_key=278d425bdf160c739803&auth_timestamp=1272044395&auth_version=1.0&body_md5=7b3d404f5cde4a0b9b8fb4789a0098cb&name=foo">>)).
"309FC4BE20F04E53E011B00744642D3FE66C2C7C5686F35ED6CD2AF6F202E445"
I just stumbled through this myself and finally managed it just using crypto, so thought I would share. For your usage I think you would want:
:crypto.hmac(:sha256, secret, string_to_sign) |> Base.encode16
The hmac portion should take care of digest + hmac and then piping to encode 16 should provide the hex part. I imagine you probably moved on some time ago, but since I just had the same issue and wanted to try and figure it out in stdlib stuff I thought I would share.
The same project (erlsha2) has a module for this:
https://github.com/vinoski/erlsha2/blob/master/src/hmac.erl
If you're using Elixir, you can use
:crypto.hash(:sha256, [secret, string_to_sign])
|> Base.encode16
|> String.downcase
This is a one-liner (Erlang 24):
[begin if N < 10 -> 48 + N; true -> 87 + N end end ||
<<N:4>> <= crypto:mac(hmac, sha256, Secret1, StringToSign1)].
>>> "309fc4be20f04e53e011b00744642d3fe66c2c7c5686f35ed6cd2af6f202e445"
No need for external libs.
I'm trying to send a push notification to APNs using Erlang.
This is the code I came up with so far:
-module(apnstest2).
-export([connect/0]).
connect() ->
application:start(ssl),
ssl:seed("someseedstring"),
Address = "gateway.sandbox.push.apple.com",
Port = 2195,
Cert = "/path/to/Certificate.pem",
Key = "/path/to/Key.unenc.pem",
Options = [{certfile, Cert}, {keyfile, Key}, {mode, binary}],
Timeout = 1000,
{ok, Socket} = ssl:connect(Address, Port, Options, Timeout),
Token = "195ec05a962b24954693c0b638b6216579a0d1d74b3e1c6f534c6f8fd0d50d03",
Payload = "{\"aps\":{\"alert\":\"Just testing.\",\"sound\":\"chime\", \"badge\":10}}",
TokenLength = length(Token),
PayloadLength = length(Payload),
Packet = [<<0:8, TokenLength, Token, PayloadLength, Payload>>],
ssl:send(Socket, list_to_binary(Packet)),
ssl:close(Socket).
The code doesn't take advantage of Erlang's concurrency but is just a prototype. I only want to test if I can send the push in the most simple way.
I think the problem is in the packet being sent to the APNs.
This is the binary format of a push notification:
alt text http://developer.apple.com/IPhone/library/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/Art/aps_provider_binary.jpg
How should I create such a packet in Erlang?
Could someone please take a look at my code and tell me where the problem is?
Also I used Erlang's SSL application to create the connection and send the data and I don't know if this is the problem or the packet.
Thanks!
To start with, there is no need for creating a list of a single binary and then calling list_to_binary/1 on it. You can just send the binary itself.
Also, make sure the field lengths and values are appropriate according to the protocol:
TokenLength = 32 = length(Token),
Packet = <<0:8, TokenLength:16/big, Token, PayloadLength:16/big, Payload>>,
ssl:send(Socket, Packet),
Now that we have gotten this far, we will see that length(Token) is in fact 64, not 32:
You forgot to convert the hex string for Token to a binary, so you are sending a 64 byte hex character string instead of 32 binary bytes.
So... making Payload a binary from the start, and making Token a numeric constant, you can do something like the following:
Payload = <<"{\"aps\":{\"alert\":\"Just testing.\",\"sound\":\"chime\", \"badge\":10}}">>,
PayloadLength = size(Payload),
Packet = <<0:8, 32:16/big,
16#195ec05a962b24954693c0b638b6216579a0d1d74b3e1c6f534c6f8fd0d50d03:256/big,
PayloadLength:16/big, Payload/binary>>,
ssl:send(Socket, Packet),
Thanks to Christian for pointing out a number of mistakes in the former revisions of this answer.
I see two mistakes:
Token should be passed in binary and not in hex ascii.
You can't use the binary syntax to turn lists into binaries.
For parsing hex to an integer and then down to binary use something like this:
Token = "dead",
TokenNum = erlang:list_to_integer(Token, 16),
TokenBin = <<TokenNum:32/integer-unit:8>>,
Build the protocol packet with something like this:
TokenBin = ...,
Payload = <<"Payload">>,
PayloadSize = byte_size(Payload),
Packet = <<0:8, 32:16, TokenBin/binary, PayloadSize:16, Payload/binary>>,
Try use a simple library epns(Erlang Push Notifications)
This library can send push notification as APNS and FCM by Erlang side. How use epns(Erlang Push Notifications) library - you can read in README.md. This liblary can used as 3-rd party in project, or you can run it locally for see how this library works:
$ git clone https://github.com/vkatsuba/epns.git
$ cd epns
$ make
Retested on Erlang 20~21