I am searching for an algorithm for file encryption/decryption which satisfies the following requirements:
Algorithm must be reliable
Algorithm should be fast for rather big files
Private key can be generated by some parameter (for example, password)
Generated private key must be compatible with public key (public key is generated only once and stored in database)
Is there any Ruby implementation of suggested algorithms?
Note Well: As emboss mentions in the comments, this answer is a poor fit for an actual system. Firstly, file encryption should not be carried out using this method (The lib provides AES, for example.). Secondly, this answer does not address any of the wider issues that will also affect how you engineer your solution.
The original source also goes into more details.
Ruby can use openssl to do this:
#!/usr/bin/env ruby
# ENCRYPT
require 'openssl'
require 'base64'
public_key_file = 'public.pem';
string = 'Hello World!';
public_key = OpenSSL::PKey::RSA.new(File.read(public_key_file))
encrypted_string = Base64.encode64(public_key.public_encrypt(string))
And decrypt:
#!/usr/bin/env ruby
# DECRYPT
require 'openssl'
require 'base64'
private_key_file = 'private.pem';
password = 'boost facile'
encrypted_string = %Q{
...
}
private_key = OpenSSL::PKey::RSA.new(File.read(private_key_file),password)
string = private_key.private_decrypt(Base64.decode64(encrypted_string))
from here
I'm afraid you are mixing two concepts here, authentication/authorization and confidentiality, trying to cover both aspects in one single step, and that won't work. You should never encrypt "real data" with asymmetric algorithms. a) they are way too slow for that, b) there are subtle issues that, if not done right, will severely weaken the security of your solution.
A good rule of thumb is that the only thing you should end up encrypting with private asymmetric keys is symmetric keys used by a much faster symmetric algorithm. But in almost all cases you shouldn't even be doing that, because in 90% of the cases what you actually want is TLS (SSL) in those cases - I tried to explain why here a while ago.
In your case, I assume the requirements are:
confidentiality of the data that is to be stored in the database: the general public shouldn't be able to read it (or even access it)
a selected few (probably just one person) should be able to access and read that data
The first goal is generally achieved by using symmetric encryption. The second goal is, albeit related, realized by quite different means. You want the user accessing the file to be authenticated (i.e. establish the identity) and on top of that you also want them to be authorized (i.e. check whether the established identity has the right to do what they intend to). This is where asymmetric cryptography may enter the stage, but not necessarily. Since your question is tagged with Rails I assume we are talking about a Rails application. You typically already have some means to authenticate and authorize users there (most likely involving the afore-mentioned TLS), you may simply reuse them in order to establish a symmetric key for actual file encryption/decryption. Password-based encryption would fit for this purpose, if you want to avoid asymmetric crypto at all. Things get even more complicated if you also want to ensure integrity of the already confidential data, that is, you want to give a kind of guarantee to the authenticated and authorized user in the sense that what they finally access has not been altered in any way in the meantime.
Developing a solution for this will be no trivial task and depend to a large extent on your given requirements, so I'm afraid there's no "golden way" that suits everyone. I would suggest to do some research, get a clearer picture of what you are trying to achieve and how, then try to get additional advice on subjects that you still feel uncertain/uncomfortable with.
Symmetric Encryption is definitely fast and has excellent support for streaming of very large files.
SymmetricEncryption::Writer.open('my_file.enc') do |file|
file.write "Hello World\n"
file.write "Keep this secret"
end
Symmetric Encryption is designed for encrypting data and large files within an organization.
When it comes to sharing files with other organizations then the best option is PGP. For streaming of very large files with PGP consider: IOStreams
IOStreams.writer('hello.pgp', recipient: 'receiver#example.org') do |writer|
writer.write('Hello World')
writer.write('and some more')
end
Look at the file iostreams/lib/io_streams/pgp.rb for more PGP examples. It also supports PGP key management directly from Ruby.
I made a gem to help with this. It's called cryptosystem. Simply configure the path and password to your private key as well as the path to your public key, and it does the rest.
Encrypting is as simple as:
rsa = Cryptosystem::RSA.new
rsa.encrypt('secret') # => "JxpuhTpEqRtMLmaSfaq/X6XONkBnMe..."
And decrypting:
encrypted_value = rsa.encrypt('secret') # => "Y8DWJc2/+7TIxdLEolV99XI2sclHuK..."
rsa.decrypt(encrypted_value) # => "secret"
You can check it out on GitHub or RubyGems.
Related
To encrypt and store user information in database, my Rails application uses salt (per User) to generate key using ActiveSupport::KeyGenerator. However, the default iterations performed by generate_key method is 2**16 [1]. Performing key generation using the user's salt on every read (for decryption) and write (for encryption) is slowing down my application.
I found that ActiveSupport::CachingKeyGenerator can be used to cache the key if salt and length used for key generation are remains same [2]. Internally, it uses Concurrent::Map [3] for caching the keys. Using ActiveSupport::CachingKeyGenerator has increased performance of my application because it doesn't generate the key always.
Will this increase memory usage of my application to a level where it can bring the app down?
References:
https://github.com/rails/rails/blob/b9ca94caea2ca6a6cc09abaffaad67b447134079/activesupport/lib/active_support/key_generator.rb#L16
https://api.rubyonrails.org/classes/ActiveSupport/CachingKeyGenerator.html
https://github.com/rails/rails/blob/b9ca94caea2ca6a6cc09abaffaad67b447134079/activesupport/lib/active_support/key_generator.rb#L33
Sharing a "working" solution
Since ActiveSupport::CachingKeyGenerator uses Concurrent::Map, I found that caching keys will result in increased memory usage proportional to the number of users (provided all the requests are somehow routed to same node).
For solving this, I wrote a similar CachingKeyGenerator that wraps ActiveSupport::KeyGenerator and ActiveSupport::Cache::MemoryStore.
class CachingKeyGenerator
BASE = Rails.application.secrets.secret_key_base
LENGTH = ActiveSupport::MessageEncryptor.key_len
KEY_EXPIRY = 1.day
def initialize
#key_generator = ActiveSupport::KeyGenerator.new(Rails.application.secrets.secret_key_base)
#keys_cache = ActiveSupport::Cache::MemoryStore.new(expires_in: KEY_EXPIRY)
end
def generate_key(salt)
key = #keys_cache.fetch(salt)
if key.nil?
key = #key_generator.generate_key(salt, LENGTH)
#keys_cache.write(salt, key)
end
key
end
end
As per the Rails documentation, ActiveSupport::Cache::MemoryStore is thread-safe and implements an LRU based cleanup mechanism [1]. This should make the cache's memory usage deterministic - to a size bound set for the memory store (default - 32Mb, can be defined during initialization).
P.S: Yet to deploy this on production, will update here if I face any unexpected issues.
[1] http://api.rubyonrails.org/classes/ActiveSupport/Cache/MemoryStore.html
So far I see these options (pseudo code):
A. Quite simple MD5 hash:
$identifier = MD5(object.id + created_at + app_secret)
=> 4c0dc8d3fdffacb65d04911291aac4cf
B. UUID:
$identifier = uuid()
=> fbcf6520-ab93-11e8-86b4-080027b55b5e
But which UUID version makes most sense? I tend to v4.
C. I'd like to have a prefix for those IDs, so I immediately know what kind of object is meant e.g. in the logs or support request.
$identifier = 'trx_' + uuid()
=> trx_fbcf6520-ab93-11e8-86b4-080027b55b5e
But is this a nice style? I could store without prefix but expose with prefix and allow requests with or without it.
What's your best praktise?
It shouldn't really matter. If I used UUID-like identifiers, I do think I would slightly prefer the UUID format because it signals to a user of an API 'This is a UUID'.
It's possible that there's some small benefits for a user, because if I see a UUID, I know I can store it in a database as a 128 bit integer instead of a string.
One thing to look out for though is security. Your first example uses the word secret which might tell me that these id's should not be guessable. UUID's are guessable and not cryptographically secure.
That being said, MD5 is insecure too so in that case both your examples are bad.
Does ActiveSupport::MessageEncryptor support deterministic encryption so that we get the same encrypted value every time we encrypt a particular string? If not, are there any other Ruby libs that support deterministic encryption?
My goal is to get same encrypted value every time I encrypt a string and I should be able to decrypt it to original value as well.
Thanks.
You get different crypts because ActiveSupport::MessageEncryptor uses OpenSSL for encryption which requires an iv by default to prevent attackers from inferring relationships between segments of the encrypted message. I would highly recommend you to not mess around with that because you open ways for attackers to infer the encryption key.
However if you still want to do that take a look into the OpenSSL documentation of ruby. There should be a way to encrypt without vector.
Because it's a high security risk I don't add code to the answer to protect others from unnecessary loop holes.
I did not get why ActiveSupport:MessageEncryptor didn't work.
Here is another way to do it.
require 'bcrypt'
encrypted_password = BCrypt::Engine.hash_secret('password#!2#4', 'ADD SALT HERE')
you can also use it like this:
class User
SALT = 'GENERATE A STATIC SALT HERE AND KEEP IT SECURE'.freeze
include BCrypt
def password=(given_password)
#encrypted_password = Engine.hash_secret(given_password, SALT)
end
end
For the full documentation please check their repo
PS: using a static salt for all users for authentication is a bad idea.
Of course: one just need to use the same key to get the same encryption
x = ActiveSupport::MessageEncryptor.new('12345678901234567890123456789012').encrypt_and_sign('foo')
=> "bXJmRUczdjVXRFdLTitUcmkvRnk1UT09LS0vb2ZYdDRybGdWbmNXMUI1VDNnQzVBPT0=--13232bbe31d966f7d1df3aaa6fcc1cdc9eea60a1"
ActiveSupport::MessageEncryptor.new('12345678901234567890123456789012').decrypt_and_verify(x)
=> "foo"
It's hard to tell why you get different results since you didn't post any code...
just looking for some advise.
I have a website with around 2500 users - small but growing.
I built it with using SHA1 encryption on the passwords.
I've since read the SHA1 is insecure and would like to change to say SHA256 with a Salt.
Does anyone have any advice on how to make a transition like this?
Would be great if I could decrypt the passwords and just re-hash them but it doesn't appear doing able.
thx
Adam
The usual way of going about this is this:
Make the hashed-password column larger to accommodate a sha256 hash, and add a 'salt' column
Set the salt field to NULL initially, and adjust your password-check code so that a NULL salt means sha1, and non-NULL means sha256
Once a sha1-use has logged in successfully, re-hash the password to sha256 with salt, and update the database.
Over time, users will migrate to sha256 by themselves; the only problem are users who log in only very sporadically or not at all. For these, you may want to send a reminder e-mail, or even threaten to shut their account down if they don't log in before day X (don't give the actual reason though...)
Just to clarify, SHA is a hashing algorithm, which is (generally) a one way street. You can't decrypt hashes, which is kind of the strength of using them to authenticate passwords. You're on the right track with moving to a salted hash, and here's how I would do it.
The only way you're getting passwords is to let the user type it in themselves. As users visit your site and log in, update the passwords one by one. In your authentication method, I would perform the hash you're doing now, and compare it against what's in the existing field (nothing new here). Assuming it matches, go ahead and salt / re-hash using SHA256, and update the password field in the database. If you want, keep a bit in your user table tracking which users have been updated.
I'm making a lot of assumptions, but this is how I've solved the hash algorithm dance in the past. Good luck!
I have another suggestion to migrate your password hash from SHA1 to SHA256 immediately without waiting for user to visit the site again to rehash the password. The change will be one time password hash migration and change to your logon validation function.
Suppose your password hash are generated using the function: password + salt [Sha1]-> Hash-sha1
To migrate to Sha256, you may convert your password hash using the following algorithm:
Hash-sha1 + salt [Sha256]-> Hash-sha256 (The salt is used to increase the complexity of input.)
Depending on the acceptable value of your sha256 function, you can consider to encode the Hash-sha1 to base64 for printable ascii.
For your logon validation function, the password should be hashed using the following algorithm:
Password + salt [sha1] -> hash-sha1 + salt [sha 256] -> hash-sha256
The disadvantage is hashed twice (use some CPU time) but simplify the migration and better security.
Switching to SHA256 will hardly make your website more secure.
SHA1 and SH512 are message digests, they were never meant to be password-hashing (or key-derivation) functions. (Although a message digest could be used a building block for a KDF, such as in PBKDF2 with HMAC-SHA1.)
A password-hashing function should defend against dictionary attacks and rainbow tables.
Currently, the only standard (as in sanctioned by NIST) password-hashing or key-derivation function is PBKDF2. Better choices, if using a standard is not required, are bcrypt and the newer scrypt. Wikipedia has pages for all three functions:
https://en.wikipedia.org/wiki/PBKDF2
https://en.wikipedia.org/wiki/Bcrypt
https://en.wikipedia.org/wiki/Scrypt
The page at https://crackstation.net/hashing-security.htm contains an extensive discussion of password security.
This being said, tdhammers offers good advice regarding how to handle the migration.
I have a rails 2.3.4 app and a line that looks like:
temp = Rails.cache.fetch(:temp_id) { User.find_by_name('Temp').id }
and everything worked fine, until I decided to switch the caching layer to memcached by adding the following to my environment.rb:
config.cache_store = :mem_cache_store
Now the line which used to work fine gives me the following error:
undefined method 'length' for :temp_id:Symbol
/usr/local/lib/ruby/gems/1.8/gems/activesupport-2.3.4/lib/active_support/vendor/memcache-client-1.7.4/memcache.rb:645:in 'get_server_for_key'
I understand the error, but I would imagine this common case would have been quickly discovered by a rails test case, so I am wondering if I am doing something wrong. Otherwise, I'm sure I can monkeypatch this issue to convert the symbol to a string.
Thanks
Just use string keys if you can. All the documentation examples use string keys. Although it's not explicitly mentioned as far as I can see, other keys are not supported.
The key arguments are passed directly to the cache implementation, so the different caching flavours may disagree on whether or not they accept anything other than strings.
Because the caches are external with the exception of in-memory cache, I'm not sure that supporting symbols would be useful apart from preventing cases like yours. The key will actually be written to some output somewhere (it's not just internal to your Ruby app), so conceptually the key should be a string.
Edit in reaction to comment: yes, it is of course possible and perfectly reasonable in this case to create a monkey patch to circumvent having to change all calls. What you're suggesting is this (copied into the answer for readability):
class MemCache
def get_server_for_key_with_symbols(key, options = {})
key = key.to_s if key.is_a? Symbol
get_server_for_key_without_symbols(key, options)
end
alias_method_chain :get_server_for_key, :symbols
end
I would also consider just doing a project wide search-and-replace for \.fetch(:\w+) and replace it with \.fetch("$1") (repeat for read and write if necessary). This should probably cover 95% of all cases and a subsequent run of your test suite should catch the rest of the errors.
In general: While the documentation of Rails is pretty good these days, a lot of assumptions are unfortunately still implicit. It's generally a good idea to take a good look at the examples that are given in the documentation, and use the same style. The documented examples are always how the framework was intended to be used.
FWIW, it's canonically Rails.cache.read and Rails.cache.write.