I'm using MongoDB as a backend for a Rails app I'm building. Mongo, by default, generates 24-character hexadecimal ids for its records to make sharding easier, so my URLs wind up looking like:
example.com/companies/4b3fc1400de0690bf2000001/employees/4b3ea6e30de0691552000001
Which is not very pretty. I'd like to stick to the Rails url conventions, but also leave these ids as they are in the database. I think a happy compromise would be to compress these hex ids to shorter collections using more characters, so they'd look something like:
example.com/companies/3ewqkvr5nj/employees/9srbsjlb2r
Then in my controller I'd reverse the compression, get the original hex id and use that to look up the record.
My question is, what's the best way to convert these ids back and forth? I'd of course want them to be as short as possible, but also url-safe and simple to convert.
Thanks!
You could represent a hexadecimal id in a base higher than 16 to make its string representation shorter. Ruby has built-in support for working with bases from 2 up to 36.
b36 = '4b3fc1400de0690bf2000001'.hex.to_s(36)
# => "29a6dblglcujcoeboqp"
To convert it back to a 24-character string you could do something like this:
'%024x' % b36.to_i(36)
# => "4b3fc1400de0690bf2000001"
To achieve better "compression" you could represent the id in base higher than 36. There are Ruby libraries that will help you with that. all-your-base gem is one such library.
I recommend base 62 representation as it only uses 0-9, a-z and A-Z characters which means it is URL safe by default.
Even with base 62 representation you end up with still unwieldy 16-character ids:
'4b3fc1400de0690bf2000001'.hex.to_base_62
# => "UHpdfMzq7jKLcvyr"
Sidestepping Rails convention a bit, another compromise is to use as the "URL id" the base 32 representation of the created_at date of the object.
aCompany.created_at
# => Sat Aug 13 20:05:35 -0500 2011
aCompany.created_at.to_i.to_s(32)
# => "174e7qv"
This way you get super short ids (7 characters) without having to keep track of a special purpose attribute (in MongoMapper, it's a simple matter of adding timestamps! in the model to get automatic created_at and updated_at attributes).
You can use base64 to make it shorter. Make sure that you are using '-' and '_' instead of '+' and '/'. You can also chop of the padding =.
Code to convert from a hex value to base 64
def MD5hex2base64(str)
h1=[].clear
# split the 32 byte hex into a 16 byte array
16.times{ h1.push(str.slice!(0,2).hex) }
# pack (C* = unsigned char), (m = base64 encoded output)
[h1.pack("C*")].pack("m")
end
Related
Is there any in built method in ruby which will generate unique alphabetic string every time(it should not have numbers only alphabets)?
i have tried SecureRandom but it doesn't provide method which will return string containing only alphabets.
SecureRandom has a method choose which:
[...] generates a string that randomly draws from a source array of characters.
Unfortunately it's private, but you can call it via send:
SecureRandom.send(:choose, [*'a'..'z'], 8)
#=> "nupvjhjw"
You could also monkey-patch Random::Formatter:
module Random::Formatter
def alphabetic(n = 16)
choose([*'a'..'z'], n)
end
end
SecureRandom.alphabetic
#=> "qfwkgsnzfmyogyya"
Note that the result is totally random and therefore not necessarily unique.
UUID are designed to have extremely low chance of collision. Since UUID only uses 17 characters, it's easy to change the non-alphabetic characters into unused alphabetic slots.
SecureRandom.uuid.gsub(/[\d-]/) { |x| (x.ord + 58).chr }
Is there any in built method in ruby which will generate unique alphabetic string every time(it should not have numbers only alphabets)?
This is not possible. The only way that a string can be unique if you are generating an unlimited number of strings is if the string is infinitely long.
So, it is impossible to generate a string that will be unique every time.
def get_random_string(length=2)
source=("a".."z").to_a + ("A".."Z").to_a
key=""
length.times{ key += source[rand(source.size)].to_s }
key
end
How about something like this if you like some monkey-patching, i have set length 2 here , please feel free to change it as per your needs
get_random_string(7)
I used Time in miliseconds, than converted it into base 36 which gives me unique aplhanumeric value and since it depends on time so, it will be very unique.
Example: Time.now.to_f.to_s.gsub('.', '').ljust(17, '0').to_i.to_s(36) # => "4j26m4zm2ss"
Take a look at this for full answer: https://stackoverflow.com/a/72738840/7365329
Try this one
length = 50
Array.new(length) { [*"A".."Z", *"a".."z"].sample }.join
# => bDKvNSySuKomcaDiAlTeOzwLyqagvtjeUkDBKPnUpYEpZUnMGF
In a Ruby on Rails app I am working on I allow users to upload files and want to give these files a short, random alphanumeric name. (Eg 'g7jf8' or '3bp76'). What is the best way to do this?
I sas thinking of generating a hash / encrypted string from the original filename and timestamp. Then query the database to double check it doesnt exist. If it does, generate another and repeat.
The issue i see with this approach is if there is high propability of duplicate strings, it could add quite a lote of datbase load.
I use this :)
def generate_token(column, length = 64)
begin
self[column] = SecureRandom.urlsafe_base64 length
end while Model.exists?(column => self[column])
end
Replace Model by your model name
SecureRandom.uuid
Will give you a globally unique String. http://en.m.wikipedia.org/wiki/Universally_unique_identifier
SecureRandom.hex 32
Will give a random String, but it's algorithm is not optimised for uniqueness. Of course the chance of collision with 32 digits, assuming true randomness, is basically theoretical. You could make 1 billion per second for 100 years and have only a 50% chance of a collision.
Use Ruby's SecureRandom.hex function with optional number of character you wanted to generate.
This will always produce new uniq 40 size alpha-numeric string, because it has Time stamp also.
loop do
random_token = Digest::SHA1.hexdigest([Time.now, rand(111..999)].join)
break random_token unless Model.exists?(column_name: random_token)
end
Note: Replace Model by your model_name and column_name by any existing column of your model.
You can assign a unique id by incrementing it each time a new file is added, and convert that id into an encrypted string using OpenSSL::Cipher with a constant key that you save somewhere.
If you end up generating a hex or numeric digest, you can keep the code shorter by representing the number as e.g. Base 62:
# This is a lightweight base62 encoding for Ruby integers.
B62CHARS = ('0'..'9').to_a + ('a'..'z').to_a + ('A'..'Z').to_a
def base62_string nbr
b62 = ''
while nbr > 0
b62 << B62CHARS[nbr % 62]
nbr /= 62
end
b62.reverse
end
If it is important for you to restrict the character set used (for instance not have uppercase chars in file names), then this code can easily be adapted, provided you can find a way of feeding in a suitable random number.
If your file names are supposed to be semi-secure, you need to arrange that there are many more possible names than actual names in storage.
It looks like you actually need a unique filenames, right? Why not forget about complex solutions and simply use Time#nsec?
t = Time.now #=> 2007-11-17 15:18:03 +0900
"%10.9f" % t.to_f #=> "1195280283.536151409"
You can use Time in miliseconds and than convert it into base 36 to reduce it length.
and since it depends on time to it will be very unique.
Example:
Time.now.to_f.to_s.gsub('.', '').ljust(17, '0').to_i.to_s(36) # => "4j26lna7g62"
Take a look at this answer:
https://stackoverflow.com/a/72738840/7365329
I would like to generate a completely random "unique" (I will ensure that using my model) identifier of a given (the length may varies) length containing numbers, letter and special characters
For example:
161551960578281|2.AQAIPhEcKsDLOVJZ.3600.1310065200.0-514191032|
Can someone please suggest the most efficient way to do that in Ruby on Rails?
EDIT: IMPORTANT:
If it is possible please comment on how efficient your proposed solution is because this will be used every time a user enters a website!
Thanks
Using this for an access token is a different story than UUIDs. You need not only pseudo-randomness but additionally this needs to be a cryptographically secure PRNG. If you don't really care what characters you use (they don't add anything to the security) you could use something as the following, producing a URL-safe Base64-encoded access token. URL-safeness becomes important in case you append the token to URLs, similar to what some Java web apps do: "http://www.bla.com/jsessionid=". If you would use raw Base64 strings for that purpose you would produce potentially invalid URLs.
require 'securerandom'
def produce_token(length=32)
token = SecureRandom.urlsafe_base64(length)
end
The probability of getting a duplicate is equal to 2^(-length). Since the output will be Base64-encoded, the actual output will be 4/3 * length long. If installed, this is based on the native OpenSSL PRNG implementation, so it should be pretty efficient in terms of performance. Should the OpenSSL extension not be installed, /dev/urandom will be used if available and finally, if you are on a Windows machine, CryptGenRandom would be used as fallback. Each of these options should be sufficiently performant. E.g., on my laptop running produce_tokena million times finishes in ~6s.
The best solution is:
require 'active_support/secure_random'
ActiveSupport::SecureRandom.hex(16) # => "00c62d9820d16b52740ca6e15d142854"
This will generate a cryptographically secure random string (i.e. completely unpredictable)
Similarly, you could use a library to generate UUIDs as suggested by others. In that case, be sure to use the random version (version 4) and make sure the implementation uses a cryptosecure random generator.
As anything related to security, rolling your own is not the best idea (even though I succumbed to it too, see first versions! :-). If you really want an homemade random string, here's a rewrite of tybro0103's approach:
require 'digest/sha1'
ALPHABET = "|,.!-0123456789".split(//) + ('a'..'z').to_a + ('A'..'Z').to_a
def random_string
not_quite_secure = Array.new(32){ ALPHABET.sample }.join
secure = Digest::SHA1.hexdigest(not_quite_secure)
end
random_string # => "2555265b2ff3ecb0a13d65a3d177b326733bc143"
Note that it hashes the random string, otherwise it could be subject to attack.
Performance should be similar.
Universally Unique Identifieres - UUIDs are tricky to generate yourself ;-) If you want something really reliable, use the uuid4r gem and call it with UUID4R::uuid(1). This will spit out a uuid based on time and a hardware id (the computers mac address). So it's even unique across multiple machines if generated at the exact same time.
A requirement for uuid4r is the ossp-uuid c library which you can install with the packetmanager of your choice (apt-get install libossp-uuid libossp-uuid-dev on debian or brew install ossp-uuid on a mac with homebrew for example) or by manually downloading and compiling it of course.
The advantage of using uuid4r over a manual (simpler?) implementation is that it is a) truly unique and not just "some sort of pseudo random number generator kind of sometimes reliable" and b) it's fast (even with higher uuid versions) by using a native extension to the c library
require 'rubygems'
require 'uuid4r'
UUID4R::uuid(1) #=> "67074ea4-a8c3-11e0-8a8c-2b12e1ad57c3"
UUID4R::uuid(1) #=> "68ad5668-a8c3-11e0-b5b7-370d85fa740d"
update:
regarding speed, see my (totally not scientific!) little benchmark over 50k iterations
user system total real
version 1 0.600000 1.370000 1.970000 ( 1.980516)
version 4 0.500000 1.360000 1.860000 ( 1.855086)
so on my machine, generating a uuid takes ~0.4 milliseconds (keep in mind I used 50000 iterations for the whole benchmark). hope that's fast enough for you
(following the "benchmark")
require 'rubygems'
require 'uuid4r'
require 'benchmark'
n = 50000
Benchmark.bm do |bm|
bm.report("version 1") { n.times { UUID4R::uuid(1) } }
bm.report("version 4") { n.times { UUID4R::uuid(4) } }
end
Update on heroku: the gem is available on heroku as well
def random_string(length=32)
chars = (0..9).to_a.concat(('a'..'z').to_a).concat(('A'..'Z').to_a).concat(['|',',','.','!','-'])
str = ""; length.times {str += chars.sample.to_s}
str
end
The Result:
>> random_string(42)
=> "a!,FEv,g3HptLCImw0oHnHNNj1drzMFM,1tptMS|rO"
It is a bit trickier to generate random letters in Ruby 1.9 vs 1.8 due to the change in behavior of characters. The easiest way to do this in 1.9 is to generate an array of the characters you want to use, then randomly grab characters out of that array.
See http://snippets.dzone.com/posts/show/491
You can check implementations here I used this one
I used current time in miliseconds to generate random but uniqure itentifier.
Time.now.to_f # => 1656041985.488494
Time.now.to_f.to_s.gsub('.', '') # => "16560419854884948"
this will give 17 digits number
sometime it can give 16 digits number because if last digit after point (.) is 0 than it is ignore by to_f.
so, I used rleft(17, '0')
example:
Time.now.to_f.to_s.gsub('.', '').ljust(17, '0') # => "1656041985488490"
Than I used to_s(36) to convert it into short length alphanumeric string.
Time.now.to_f.to_s.gsub('.', '').ljust(17, '0').to_i.to_s(36) # => "4j26hz9640k"
to_s(36) is radix base (36)
https://apidock.com/ruby/v2_5_5/Integer/to_s
if you want to limit the length than you can select first few digits of time in miliseconds:
Time.now.to_f.to_s.gsub('.', '').ljust(17, '0').first(12).to_i.to_s(36) # => "242sii2l"
but if you want the uniqueness accuracy in miliseconds than I would suggest to have atleast first(15) digits of time
I am doing some client side validation in ASP.NET MVC and I found myself trying to do conditional validation on a set of items (ie, if the checkbox is checked then validate and visa versa). This was problematic, to say the least.
To get around this, I figured that I could "cheat" by having a hidden element that would contain all of the information for each set, thus the idea of a CSV string containing this information.
I already use a custom [HiddenRequired] attribute to validate if the hidden input contains a value, with success, but I thought as I will need to validate each piece of data in the csv, that a regular expression would solve this.
My regular expression work is extremely weak and after a good 2 hours I've almost given up.
This is an example of the csv string:
true,3,24,over,0.5
to explain:
true denotes if I should validate the rest. I need to conditionally switch in the regex using this
3 and 24 are integers and will only ever fall in the range 0-24.
over is a string and will either be over or under
0.5 is a decimal value, of unknown precision.
In the validation, all values should be present and at least of the correct type
Is there someone who can either provide such a regex or at least provide some hints, i'm really stuck!
Try this regex:
#"^(true,([01]?\d|2[0-4]),([01]?\d|2[0-4]),(over|under),\d+\.?\d+|false.*)$"
I'll try to explain it using comments. Feel free to ask if anything is unclear. =)
#"
^ # start of line
(
true, # literal true
([01]?\d # Either 0, 1, or nothing followed by a digit
| # or
2[0-4]), # 20 - 24
([01]?\d|2[0-4]), # again
(over|under), # over or under
\d+\.?\d+ # any number of digits, optional dot, any number of digits
| #... OR ...
false.* # false followed by anything
)
$ # end of line
");
I would probably use a Split(',') and validate elements of the resulting array instead of using a regex. Also you should watch out for the \, case (the comma is part of the value).
I have user entries as filenames. Of course this is not a good idea, so I want to drop everything except [a-z], [A-Z], [0-9], _ and -.
For instance:
my§document$is°° very&interesting___thisIs%nice445.doc.pdf
should become
my_document_is_____very_interesting___thisIs_nice445_doc.pdf
and then ideally
my_document_is_very_interesting_thisIs_nice445_doc.pdf
Is there a nice and elegant way for doing this?
I'd like to suggest a solution that differs from the old one. Note that the old one uses the deprecated returning. By the way, it's anyway specific to Rails, and you didn't explicitly mention Rails in your question (only as a tag). Also, the existing solution fails to encode .doc.pdf into _doc.pdf, as you requested. And, of course, it doesn't collapse the underscores into one.
Here's my solution:
def sanitize_filename(filename)
# Split the name when finding a period which is preceded by some
# character, and is followed by some character other than a period,
# if there is no following period that is followed by something
# other than a period (yeah, confusing, I know)
fn = filename.split /(?<=.)\.(?=[^.])(?!.*\.[^.])/m
# We now have one or two parts (depending on whether we could find
# a suitable period). For each of these parts, replace any unwanted
# sequence of characters with an underscore
fn.map! { |s| s.gsub /[^a-z0-9\-]+/i, '_' }
# Finally, join the parts with a period and return the result
return fn.join '.'
end
You haven't specified all the details about the conversion. Thus, I'm making the following assumptions:
There should be at most one filename extension, which means that there should be at most one period in the filename
Trailing periods do not mark the start of an extension
Leading periods do not mark the start of an extension
Any sequence of characters beyond A–Z, a–z, 0–9 and - should be collapsed into a single _ (i.e. underscore is itself regarded as a disallowed character, and the string '$%__°#' would become '_' – rather than '___' from the parts '$%', '__' and '°#')
The complicated part of this is where I split the filename into the main part and extension. With the help of a regular expression, I'm searching for the last period, which is followed by something else than a period, so that there are no following periods matching the same criteria in the string. It must, however, be preceded by some character to make sure it's not the first character in the string.
My results from testing the function:
1.9.3p125 :006 > sanitize_filename 'my§document$is°° very&interesting___thisIs%nice445.doc.pdf'
=> "my_document_is_very_interesting_thisIs_nice445_doc.pdf"
which I think is what you requested. I hope this is nice and elegant enough.
From http://web.archive.org/web/20110529023841/http://devblog.muziboo.com/2008/06/17/attachment-fu-sanitize-filename-regex-and-unicode-gotcha/:
def sanitize_filename(filename)
returning filename.strip do |name|
# NOTE: File.basename doesn't work right with Windows paths on Unix
# get only the filename, not the whole path
name.gsub!(/^.*(\\|\/)/, '')
# Strip out the non-ascii character
name.gsub!(/[^0-9A-Za-z.\-]/, '_')
end
end
In Rails you might also be able to use ActiveStorage::Filename#sanitized:
ActiveStorage::Filename.new("foo:bar.jpg").sanitized # => "foo-bar.jpg"
ActiveStorage::Filename.new("foo/bar.jpg").sanitized # => "foo-bar.jpg"
If you use Rails you can also use String#parameterize. This is not particularly intended for that, but you will obtain a satisfying result.
"my§document$is°° very&interesting___thisIs%nice445.doc.pdf".parameterize
For Rails I found myself wanting to keep any file extensions but using parameterize for the remainder of the characters:
filename = "my§doc$is°° very&itng___thsIs%nie445.doc.pdf"
cleaned = filename.split(".").map(&:parameterize).join(".")
Implementation details and ideas see source: https://github.com/rails/rails/blob/master/activesupport/lib/active_support/inflector/transliterate.rb
def parameterize(string, separator: "-", preserve_case: false)
# Turn unwanted chars into the separator.
parameterized_string.gsub!(/[^a-z0-9\-_]+/i, separator)
#... some more stuff
end
If your goal is just to generate a filename that is "safe" to use on all operating systems (and not to remove any and all non-ASCII characters), then I would recommend the zaru gem. It doesn't do everything the original question specifies, but the filename produced should be safe to use (and still keep any filename-safe unicode characters untouched):
Zaru.sanitize! " what\ēver//wëird:user:înput:"
# => "whatēverwëirduserînput"
Zaru.sanitize! "my§docu*ment$is°° very&interes:ting___thisIs%nice445.doc.pdf"
# => "my§document$is°° very&interesting___thisIs%nice445.doc.pdf"
There is a library that may be helpful, especially if you're interested in replacing weird Unicode characters with ASCII: unidecode.
irb(main):001:0> require 'unidecoder'
=> true
irb(main):004:0> "Grzegżółka".to_ascii
=> "Grzegzolka"