Let's assume I have a config/secrets.yml file that contains:
development:
app_name: MyApp
secret: <%= ENV['SECRET_VALUE'] %>
If I set SECRET_VALUE with a newline, it will break. E.g:
export SECRET_VALUE=$(printf "hello\nworld")
Then when I start my rail application I get this error:
/usr/local/lib/ruby/3.0.0/psych.rb:457:in 'parse': (<unknown>): could not find expected ':' while scanning a simple key at line 4 column 1 (Psych::SyntaxError)
While debugging the issue, I realized that the literal value of ENV['SECRET_VALUE'] is added to the yaml file before parsing it. That means if I wanted to achieve what I'm trying to do, I would have to do something like:
export SECRET_VALUE=$(printf "|\n hello\n world")
This works but this is very ugly and I can't be the only one who think this behaviour is absurd?? Is there a better way to do this?
EDIT:
I tried adding quotes around the value:
development:
app_name: MyApp
secret: "<%= ENV['SECRET_VALUE'] %>"
It "works" but the newline gets removed from the string...
root#4e4431bae32e:/app# rails console
Loading development environment (Rails 7.0.3)
irb(main):001:0> puts ENV['SECRET_VALUE']
hello
world
=> nil
irb(main):002:0> puts Rails.application.secrets[:secret]
hello world
=> nil
It's important that the newline remains for my use case.
You can use an escaped quoted string. YAML's string format is similar enough to a superset of Ruby's for String#dump to work:
development:
app_name: MyApp
secret: <%= ENV['SECRET_VALUE'].dump %>
When copy pasting in some code like:
params = { :user => { :email =>“yencn02#yahoo.com"} }
The console still is awaiting input, and I have to CTRL+C to escape from it.
The console then says:
>> params
=> nil
But it works at my colleague's. What's wrong?
You have a UTF-8 character in there which is messing things up.
Namely the “ character. Replace it with the standard " character and it should work.
So I have a Ruby script (using Ruby because we have a library of pre-existing code that I need to use). From within Ruby I am using backticks to call Linux commands, specifically in this case the "mv" command. I am trying to move one file to another location but I keep getting the error message that x and y are "the same file" even though they are very clearly NOT the same file.
Here is the code in Ruby:
#!/usr/local/rvm/rubies/ruby-2.1.1/bin/ruby
masterFiles=[]
masterFiles << "/mnt/datadrive/Data Capture/QualityControl/UH_HRA_SVY/Scans and DataOutput/Data/UH_HRA_SVY_DATA.txt"
masterFiles << "/mnt/datadrive/Data Capture/QualityControl/UH_HRA_SVY_SPAN/Scans and DataOutput/Data/UH_HRA_SVY_SPAN_DATA.txt"
tm=Time.new.strftime("%Y%m%d")
masterFiles.each do |mf|
if File.exist?(mf)
qmf=39.chr + mf + 39.chr
`cat #{qmf} >> /tmp/QM`
savename=39.chr + \
"/mnt/datadrive/Data Capture/QualityControl/UH_HRA_SVY/Scans and DataOutput/Data/DailyFiles/" + \
File.basename(mf).gsub(".txt","_"+tm) + ".txt" + 39.chr
`mv #{qmf} #{savename}`
end
end
The error that I get is this:
mv: `/mnt/datadrive/Data Capture/QualityControl/UH_HRA_SVY_SPAN/Scans
and DataOutput/Data/UH_HRA_SVY_SPAN_DATA.txt' and `/mnt/datadrive/Data
Capture/QualityControl/UH_HRA_SVY/Scans and
DataOutput/Data/DailyFiles/UH_HRA_SVY_SPAN_DATA_20140530.txt' are the
same file
If I change this line:
`mv #{qmf} #{savename}`
To this:
puts "mv #{qmf} #{savename}"
And then run the output, it works as expected.
I am pretty sure that this has to do with spaces in the path. I have tried every combination of double-quoting, triple-quoting, quadruple-quoting, and back-slashing I can think of to resolve this but no go. I have also tried using FileUtils.mv but get what is basically the same error worded differently.
Can anybody help ? Thanks a lot.
p.s. I realize it's entirely possible that I could be going about this in an entirely wrong-headed way, so feel free to point that out if so. However, I am trying to use the tools which I already have some knowledge of (cat, mv, etc) instead of re-inventing the wheel.
You could use FileUtils.mv
I often do aliases like so:
require 'fileutils'
def mv(from, to)
FileUtils.mv(from, to)
end
And inside the mv() method I do additional safeguards, i.e. if the file does not exist, if there is a lack of permissions and so forth.
If you then still have problems with filenames that have ' ' blank characters, try to put the file into a "" quote like:
your_target_location = "foo/bar bla"
I have the following string that needs to be replaced by an empty character in rails. Followed many tutorials and docs. Please help me achieve this.
String:
/home/<someword>/dbdumps/backup.sql
To be replaced as:
backup
To get the file name from a path, I'd use File#basename
File.basename('/home/<someword>/dbdumps/backup.sql', '.sql')
#=> 'backup'
if "someword" is the only thing that changes you dont even need regex.
Assume
path = "/home/<someword>/dbdumps/backup.sql"
then
path.split("/").last.split(".").first
returns
=> "backup"
The easiest solution would be a gsub (string substitution) like so:
string = "home/<someword>/dbdumps/backup.sql"
new_string = string.gsub(%r{home/(.*)/dbdumps/backup.sql}, 'backup' )
This is a simple example of string substitution.
In a rails app i do a Net:SSH:start( ) and run ssh.exec!('ls /home//dbdumps/.sql'). I am ?sending the output to a string and then i have to display the list of the files. For that I am taking the output into a string and trying to do a gsub. Is this the right approach?
I would not consider it pretty (naive code, no error checking, loops with requests) but something like this could do the job for you. It depends if you want to end up with just the backup names or the full path.
ssh.exec!("ls -l /home/") do |channel, stream, data|
directories << data if stream == :stdout
end
directories.each do |dir|
ssh.exec!("ls -l /home/" + dir + "dbdumps") do |channel, stream, data|
backup_names << /home/" + dir + "/" + data if stream == :stdout
end
end
hope this helps
Trying to parse a CSV file, but still getting the error message Unquoted fields do not allow \r or \n (line 2)..
I found here at SO similar topic, where was a hint to do following:
CSV.open('file.csv', :row_sep => "\r\n") do |csv|
but his unfortunately doesn't works me... I can't change the CSV file, so I would need to fix it in the code.
EDIT sample of CSV file:
A;B;C
1234;...
Is there any way to do it?
Many thanks!
First of all, you should set you column delimiters to ';', since that is not the normal way CSV files are parsed. This worked for me:
CSV.open('file.csv', :row_sep => :auto, :col_sep => ";") do |csv|
csv.each { |a,b,c| puts "#{a},#{b},#{c}" }
end
From the 1.9.2 CSV documentation:
Auto-discovery reads ahead in the data looking for the next \r\n,
\n, or \r sequence. A sequence will be selected even if it occurs
in a quoted field, assuming that you would have the same line endings
there.
Simpler solution if the CSV was touched or saved by any program that may have used weird formatting (such as Excel or Spreadsheet):
Open the file with any plain text editor (I used Sublime Text 3)
Press the enter key to add a new line anywhere
Save the file
Remove the line you just added
Save the file again
Try the import again, error should be gone
For me I was importing LinkedIn CSV and got the error.
I removed the blank lines like this:
def import
csv_text = File.read('filepath', :encoding => 'ISO-8859-1')
#remove blank lines from LinkedIn
csv_text = csv_text.gsub /^$\n/, ''
#csv = CSV.parse(csv_text, :headers => true, skip_blanks: true)
end
In my case I had to provide encoding, and a quote char that was guaranteed to not occur in data
CSV.read("file.txt", 'rb:bom|UTF-16LE', {:row_sep => "\r\n", :col_sep => "\t", :quote_char => "\x00"})
I realize this is an old post but I recently ran into a similar issue with a badly formatted CSV file that failed to parse with the standard Ruby CSV library.
I tried the SmarterCSV gem which parsed the file in no time. It's an external library so it might not be the best solution for everyone but it beats parsing the file myself.
opts = { col_sep: ';', file_encoding: 'iso-8859-1', skip_lines: 5 }
SmarterCSV.process(file, opts).each do |row|
p row[:someheader]
end
Please see this thread Unquoted fields do not allow \r or \n
Solution:
file = open(file.csv).read.gsub!("\r", '')
CSV.open(file, :row_sep => "\r\n") do |csv|
In my case, the first row of the spreadsheet/CSV was a double-quoted bit of introduction text. The error I got was:
/Users/.../.rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/csv.rb:1880:in `block (2 levels) in shift': Unquoted fields do not allow \r or \n (line 1). (CSV::MalformedCSVError)
I deleted the comment with " characters so the .csv ONLY had the .csv data, saved it, and my program worked with no errors.
If you have to deal with files coming from Excel with newlines in cells there is also a solution.
The big disadvantage of this way is, that no semicolons or no double quotes in strings are allowed.
I choose to go with no semicolons
if file.respond_to?(:read)
csv_contents = file.read
elsif file_data.respond_to?(:path)
csv_contents = File.read(file.path)
else
logger.error "Bad file_data: #{file_data.class.name}: #{file_data.inspect}"
return false
end
result = "string"
csv_contents = csv_contents.force_encoding("iso-8859-1").encode('utf-8') # In my case the files are latin 1...
# Here is the important part (Remove all newlines between quotes):
while !result.nil?
result = csv_contents.sub!(/(\"[^\;]*)[\n\r]([^\;]*\")/){$1 + ", " + $2}
end
CSV.parse(csv_contents, headers: false, :row_sep => :auto, col_sep: ";") do |row|
# do whatever
end
For me the solution works fine, if you deal with large files you could run into problems with it.
If you want to go with no quotes just replace the semicolons in the regex with quotes.
Another simple solution to fix the weird formatting caused by Excel is to copy and paste the data into Google spreadsheet and then download it as a CSV.