I wish to create a simple bash alias due to the amount of options & switches this particular set of functions requires. This is the background info and NOT the problem.
To perform the command in question requires a passphrase (often in multiple locations) which I would like to minimize and provide some privacy from other users at the same time.
Here is the alias example...
alias test="read -sp 'Enter passphrase: ' pass; gpg --batch --passphrase $pass --symmetric --cipher-algo aes256 -o file.ext.gpg file.ext"
The alias works fine, prompts the user to enter a passphrase and applies it to the decryption process.
THIS IS THE PROBLEM:
If I encrypt the file and enter a passphrase (without using the read -sp utlity as shown in the above example) the encrypted files password is different than if I use the 'read' binary to mask the input.
If I display the contents of $pass that was captured with read -sp it displays just as I typed it without any additional line endings etc.
Anyone experience this?
Shouldn't you quote $pass? In case it contains spaces etc. Also, you need to escape the $ to make it not expand while setting the alias.
So:
alias test="read -sp 'Enter pp: ' pass; gpg --passphrase \"\$pass\" --batch --symmetric --cipher-algo aes256 -o file.ext.gpg file.ext"
Besides this, you may want to use --passphrase-fd so the passphrase doesn't end up in ps output:
alias test="read -sp 'pp: ' pass; gpg --passphrase-fd 3 --etc-etc 3<<< \$pass"
A shell function will be simpler, as you eliminate a level of quoting, and allows parameters if the function needs to be generalized to other files.
# There is already a standard command called 'test'; use a different name
pass_encrypt () {
in=$1
out=$in.gpg
read -rsp 'Enter passphrase: ' pass
gpg --batch --passphrase-fd 3 --symmetric \
--cipher-algo aes256 -o "$out" "$in" 3<<<$pass
}
(with mvds' suggestion of using --passphrase-fd)
Related
I am trying to decrypt a string which has been encrypted in my rails project. This is how I am encrypting the data:
def encrypt_text(text_To_encrypt)
# 0. generate the key using command openssl rand -hex 16 on linux machines
# 1. Read the secret from config
# 2. Read the salt from config
# 3. Encrypt the data
# 4. return the encypted data
# Ref: http://www.monkeyandcrow.com/blog/reading_rails_how_does_message_encryptor_work/
secret = Rails.configuration.miscconfig['encryption_key']
salt = Rails.configuration.miscconfig['encryption_salt']
key = ActiveSupport::KeyGenerator.new(secret).generate_key(salt, 32)
crypt = ActiveSupport::MessageEncryptor.new(key)
encrypted_data = crypt.encrypt_and_sign(text_To_encrypt)
encrypted_data
end
Now the issue is I am not able to decrypt it using openssl. It just shows bad magic number. Once I do that in open ssl, my plan is to decrypt it in golang.
Here is how I tried to descrypt it using openssl:
openssl enc -d -aes-256-cbc -salt -in encrypted.txt -out decrypted.txt -d -pass pass:<the key given in rails> -a
This just shows bad magic number
Trying to decrypt data encrypted in a different system will not work unless you are aware and deal with the many intricate details of how both systems do the cryptography. Although both Rails and the openssl command line tool use the OpenSSL libraries under the hood for their crypto operations, they both use it in their own distinct ways that are not directly interoperable.
If you look close to the two systems, you'll see that for example:
Rails message encryptor not only encrypts the message but also signs it
Rails encryptor uses Marshal to serialize the input data
the openssl enc tool expects the encrypted data in a distinct file format with a Salted__<salt> header (this is why you get the bad magic number message from openssl)
the openssl tool must be properly configured to use the same ciphers as Rails encryptor and key generator, as openssl defaults are different from Rails defaults
the default ciphers configuration changed significantly since Rails 5.2.
With this general info, we can have a look at a a practical example. It is tested in Rails 4.2 but should work equally up to Rails 5.1.
Anatomy of a Rails-encrypted message
Let me start with a slightly amended code that you presented. The only changes there are to preset the password and salt to static values and print a lot of debug info:
def encrypt_text(text_to_encrypt)
password = "password" # the password to derive the key
salt = "saltsalt" # salt must be 8 bytes
key = ActiveSupport::KeyGenerator.new(password).generate_key(salt, 32)
puts "salt (hexa) = #{salt.unpack('H*').first}" # print the saltin HEX
puts "key (hexa) = #{key.unpack('H*').first}" # print the generated key in HEX
crypt = ActiveSupport::MessageEncryptor.new(key)
output = crypt.encrypt_and_sign(text_to_encrypt)
puts "output (base64) = #{output}"
output
end
encrypt_text("secret text")
When you run this, you'll get something like the following output:
salt (hexa) = 73616c7473616c74
key (hexa) = 196827b250431e911310f5dbc82d395782837b7ae56230dce24e497cf07b6518
output (base64) = SGRTUXYxRys1N1haVWNpVWxxWTdCMHlyMk15SnQ0dWFBOCt3Z0djWVdBZz0tLTkrd1hBNWJMVm9HcnptZ3loOG1mNHc9PQ==--80d091e8799776113b2c0efd1bf75b344bf39994
The last line (output of the encrypt_and_sign method) is a combination of two parts separated by -- (see source):
the encrypted message (Base64-encoded) and
the message signature (Base64-encoded).
The signature is not important for encryption so let's take a look in the first part - let's decode it in Rails console:
> Base64.strict_decode64("SGRTUXYxRys1N1haVWNpVWxxWTdCMHlyMk15SnQ0dWFBOCt3Z0djWVdBZz0tLTkrd1hBNWJMVm9HcnptZ3loOG1mNHc9PQ==")
=> "HdSQv1G+57XZUciUlqY7B0yr2MyJt4uaA8+wgGcYWAg=--9+wXA5bLVoGrzmgyh8mf4w=="
You can see that the decoded message again consists of two Base64-encoded parts separated by -- (see source):
the encrypted message itself
the initialization vector used in the encryption
Rails message encryptor uses the aes-256-cbc cipher by default (note that this has changed since Rails 5.2). This cipher needs an initialization vector, which is randomly generated by Rails and must be present in the encrypted output so that we can use it together with the key to decipher the message.
Moreover, Rails does not encrypt the input data as a simple plain text, but rather a serialized version of the data, using the Marshal serializer by default (source). If we decrypted such serialized value with openssl, we would still get a slightly garbled (serialized) version of the initial plain text data. That's why it will be more appropriate to disable serialization while encrypting the data in Rails. This can be done by passing a parameter to the encryption method:
# crypt = ActiveSupport::MessageEncryptor.new(key)
crypt = ActiveSupport::MessageEncryptor.new(key, serializer: ActiveSupport::MessageEncryptor::NullSerializer)
A re-run of the code yields output that is slightly shorter than the previous version, because the encrypted data has not been serialized now:
salt (hexa) = 73616c7473616c74
key (hexa) = 196827b250431e911310f5dbc82d395782837b7ae56230dce24e497cf07b6518
output (base64) = SUlIWFBjSXRUc0JodEMzLzhXckJzUT09LS1oZGtPV1ZRc2I5Wi8zOG01dFNOdVdBPT0=--58bbaf983fd20459062df8b6c59eb470311cbca9
Finally, we must find out some info about the encryption key derivation procedure. The source tells us that the KeyGenerator uses the pbkdf2_hmac_sha1 algorithm with 2**16 = 65536 iterations to derive the key from the password / secret.
Anatomy of an openssl encrypted message
Now, a similar investigation is needed on the openssl side to learn the details of its decryption process. First, if you encrypt anything using the openssl enc tool, you will find out that the output has a distinct format:
Salted__<salt><encrypted_message>
It begins with the Salted__ magic string, then followed by the salt (in hex form) and finally followed by the encrypted data. To be able to decrypt any data using this tool, we must get our encrypted data into the same format.
The openssl tool uses the EVP_BytesToKey (see source) to derive the key by default but can be configured to use the pbkdf2_hmac_sha1 algorithm using the -pbkdf2 and -md sha1 options. The number of iterations can be set using the -iter option.
How to decrypt Rails-encrypted message in openssl
So, finally we have enough information to actually try to decrypt a Rails-encrypted message in openssl.
First we must decode the first part of the Rails-encrypted output again to get the encrypted data and the initialization vector:
> Base64.strict_decode64("SUlIWFBjSXRUc0JodEMzLzhXckJzUT09LS1oZGtPV1ZRc2I5Wi8zOG01dFNOdVdBPT0=")
=> "IIHXPcItTsBhtC3/8WrBsQ==--hdkOWVQsb9Z/38m5tSNuWA=="
Now let's take the IV (the second part) and convert it to a hexa string form, as that is the form that openssl needs:
> Base64.strict_decode64("hdkOWVQsb9Z/38m5tSNuWA==").unpack("H*").first
=> "85d90e59542c6fd67fdfc9b9b5236e58" # the initialization vector in hex form
Now we need to take the Rails-encrypted data and convert it to the format that openssl will recognize, i.e. prepend it with the magic string and salt and Base64-encode it again:
> Base64.strict_encode64("Salted__" + "saltsalt" + Base64.strict_decode64("IIHXPcItTsBhtC3/8WrBsQ=="))
=> "U2FsdGVkX19zYWx0c2FsdCCB1z3CLU7AYbQt//FqwbE=" # encrypted data suitable for openssl
Finally, we can construct the openssl command to decrypt the data:
$ echo "U2FsdGVkX19zYWx0c2FsdCCB1z3CLU7AYbQt//FqwbE=" |
> openssl enc -aes-256-cbc -d -iv 85d90e59542c6fd67fdfc9b9b5236e58 \
> -pass pass:password -pbkdf2 -iter 65536 -md sha1 -a
secret text
And voilá, we successfully decrypted the initial message!
The openssl parameters are as follows:
-aes-256-cbc sets the same cipher as Rails uses for encryption
-d stands for decryption
-iv passes the initialization vector in the hex string form
-pass pass:password sets the password used to derive the encryption key to "password"
-pbkdf2 and -md sha1 set the same key derivation algorithm as is used by Rails (pbkdf2_hmac_sha1)
-iter 65536 sets the same number of iterations for key derivation as was done in Rails
-a allows to work with Base64-encoded encrypted data - no need to handle raw bytes in files
By default openssl reads from STDIN, so we simply pass the encrypted data (in proper format) to openssl using echo.
debugging
In case you hit any problems when decrypting with openssl, it is useful to add the -P parameter to the command line, which outputs debugging info about the cipher / key parameters:
$ echo ... | openssl ... -P
salt=73616C7473616C74
key=196827B250431E911310F5DBC82D395782837B7AE56230DCE24E497CF07B6518
iv =85D90E59542C6FD67FDFC9B9B5236E58
The salt, key, and iv values must correspond to the debugging values printed by the original code in the encrypt_text method printed above. If they are different, you know you are doing something wrong...
Now, I guess you can expect similar problems when trying to decrypt the message in go but I think you have some good pointers now to start.
I'd like to strip parameter values from a URL, but leave the parameter names in place: I.e.,
Change
http://abc.def.edu/pager/page.cfm?pai=97878&pager=123
into
http://abc.def.edu/pager/page.cfm?pai=&pager=
I've tried:
sed "s/=.*\&/=\&/g"
With no success. Am I getting close? I've seen lots of posts about stripping parameters entirely, but nothing about just stripping the values. Please redirect me and accept my apologies if this has already been addressed.
Thanks,
Al
$ sed -r 's/=[^\&]+/=/g' <<< 'http://abc.def.edu/pager/page.cfm?pai=97878&pager=123'
OUTPUT:
http://abc.def.edu/pager/page.cfm?pai=&pager=
A bit more heavy-weight than sed:
$ perl -pe 's/(?<==).+?(?=&|$)//g' <<< "$url"
http://abc.def.edu/pager/page.cfm?pai=&pager=
Given a LDAP password stored in SHA-1/{SSHA} how would I validate it in erlang.
For example - given the following {SSHA}:
% slappasswd -s myPassword
{SSHA}GEH5kMEQZHYHS95dgr6KmFdg0a4BicBP
%
How would I (in erlang) validate that clear text 'myPassword' matches with the hashed value of '{SSHA}GEH5kMEQZHYHS95dgr6KmFdg0a4BicBP'.
Passwords stored in a directory server are validated using the BIND operation. A properly configured and secured directory server will not allow access to password data; therefore LDAP clients must not be coded expecting that the password data is available, whether encrypted or hashed. LDAP clients must use the BIND operation to validate passwords.
After some help from others I've come up with a routine to do this in Erlang. Following up here to share with others.
First - this link (found in another post) gives functions in other languages doing what I wanted:
http://www.openldap.org/faq/data/cache/347.html
The trick was that the 'ldap {SSHA}' encoding is a salted-SHA1 hash which is also base64 encoded. So - you must decode it, extract the salt and then use that in the re-encoding of the 'clear password' for comparison.
Here is a short Erlang routine which does this:
validatessha(ClearPassword, SshaHash) ->
D64 = base64:decode(lists:nthtail(6, SshaHash)),
{HashedData, Salt} = lists:split(20, binary_to_list(D64)),
NewHash = crypto:sha(list_to_binary(ClearPassword ++ Salt)),
string:equal(binary_to_list(NewHash), HashedData).
Given the data in my original post - here's the output:
67> run:validatessha("myPassword", "{SSHA}GEH5kMEQZHYHS95dgr6KmFdg0a4BicBP").
true
68>
Thanx all.
Mike
My erlang is very rusty, so this isn't very pretty, but maybe it gets my idea along anyway.
run() ->
Password = "myPassword",
HashRaw = os:cmd("slappasswd -s " ++ Password),
Hash1 = lists:nthtail(6, HashRaw),
Hash2 = lists:concat ([integer_to_list(X, 16) || X <- binary_to_list(crypto:sha(Password))]),
string:equal(string:to_lower(Hash1),
string:to_lower(Hash2)).
My idea is that you:
Run the command whose output you are interested in verifying (slappasswd), save the output and trim away the extra decoration preceding the hash.
Run crypto:sha() from the erlang libraries. Take the binary output from this, and convert it to a list of integers, each of which you then convert to a hexadecimal string, which you then concatenate, thereby create Hash2.
Compare the output of your command to the output of crypto:sha()
EDIT: I don't have this command you're using, so I couldn't really try this very thoroughly.. But it works for sha1sum. I hope they are the same!
The log file contains many email addresses and i need to write a shell script to parse the log file and get all the email addresses. The log file's size is 1 GB, and my vps server's RAM is just 512m, so I want to take the performance into account. how can i do that?
if every line starts with email, you can use these coommands. First one select first 'word' of a file, and second gives unique values:
cut -f 1 -d ' ' LOGFILE.txt | sort -u
(Really surprised this isn't answered anywhere online; couple posts over the past few years with a similar question, but never answered. Let's hope the Stackoverflow crew can come to the rescue)
Situation:
When using gettext to support application localization, one sometimes wishes to specify a 'domain' with dgettext('domain', 'some text string'). However, when running xgettext, all strings wrapped with dgettext(...) are spit out into one file (default: messages.po).
Given the following example:
dgettext('menus', 'login link');
dgettext('menus', 'account link');
dgettext('footer', 'copyright notice');
dgettext('footer', 'contact form');
is there any way to end up with
menus.po
footer.po
using an extractor such as xgettext?
PHP response desired, although I believe this should be applicable across all languages
The only way I've found to do this is to redefine gettext functions...
example:
function _menus ($str) {
return dgettext('menus', $str);
}
function _footer ($_str) {
return dgettext('footer', $str);
}
_menus('login link');
_menus('account link');
_footer('copyright notice');
_footer('contact form');
else, you only have to run following commands:
xgettext [usual options] -k --keyword=_menus:1 -d menus
xgettext [usual options] -k --keyword=_footer:1 -d footer
Bye!
I do not know how to put different contexts in different files, but I did find that xgettext can extract the domain names into msgctxt fields in the po file. For PHP this is not done by default. To enable this, use for example --keyword=dgettext:1c,2 (in poedit, add "dgettext:1c,2") to the keyword list.
See also:
http://developer.gnome.org/glib/2.28/glib-I18N.html
https://www.gnu.org/savannah-checkouts/gnu/gettext/manual/html_node/xgettext-Invocation.html
Achieving this is best done through either code separation or the use of context disambiguation.
If you can separate your menu code from your footer code, then you can truly consider them different domains and extract them accordingly from known locations.
If modular separation is impossible and all the code lives together, then really you should be using context instead of domains. e.g.
translate( 'A string', 'myproject', 'some module' )
Where "myproject" is your domain and "some module" disambiguates the string.
However, reality doesn't always align with best practice, so if you can't refactor your code as Asevere suggests (and that is probably the best answer) then I have a massive hack to offer.
You could exploit the context flag mentioned in Boris's answer - We can repurpose this but only if we're not otherwise going to be using contexts.
I'll repeat that. This hack will only work if your code is not using contexts.
Some PHP holding two domains (including one string used in both) -
<?php // test.php
dgettext( 'abc', 'foo' );
dgettext( 'abc', 'bar' );
dgettext( 'xyz', 'bar' );
We can cheat, and take the domain argument as if we intended it to be the message context (msgctxt field). Extracting from the command line:
xgettext -LPHP --keyword=dgettext:1,2c -o - test.php \
| sed 's/CHARSET/utf-8/' \
> combined.pot
This generates a combined.pot file containing all the strings with our context hack. (note we also fixed the placeholder character set field which would break the next bit)
We can now filter out all messages of a given context into separate files using msggrep. Note we also trash the context field as we're not using it.
msggrep -J -e foo -o - combined.pot | sed '/^msgctxt/d' > foo.pot
msggrep -J -e bar -o - combined.pot | sed '/^msgctxt/d' > bar.pot
Improper, but it works.