all. I've got a dynamic fixture CSV file that's generating predictable data for a table in order for my unit tests to do their thing. It's working as expected and filling the table with the data, but when I check the table after the tests run, I'm seeing a number of additional rows of "blank" data (all zeros, etc). Those aren't being created by the fixture, and the unit tests are read-only, just doing selects, so I can't blame the code. There doesn't seem to be any logging done during the fixtures setup, so I can't see when the "blank" data is being inserted. Anyone ever run across this before, or have any ideas of how to log or otherwise see what the fixture setup is doing in order to trace down the source of the blank data?
You could turn on ActiveRecord logging (put ActiveRecord::Base.logger = Logger.new(STDOUT) in your test/test_helper.rb file).
Or, instead of using fixtures (which have gone the way of the dodo for most Rails developers) you could use something more reliable like Factories (thoughtbot's factory_girl) or seed_fu (if you have specific data that must be loaded).
I discovered what the problem was, although not the precise way to prevent it. I was placing ERB into the fixture CSV file, which was working fine, but due to the way it was being parsed and processed, it was causing blank lines to be placed into the resulting CSV output. Fixtures doesn't seem to handle that very well, and as a result it was inserting blank rows into the table. I couldn't prevent the blank lines from being placed into the output CSV because for whatever reason, the <% rubycode -%> doesn't work -- having the closing dash caused ERB parsing errors. Not sure why.
In any case, the eventual workaround was to switch to YML instead of CSV. It tolerates the white space just fine, and no blank rows are being inserted into the table anymore.
As an aside, factory___girl seems potentially interesting, but the YML is now doing just fine so it may be overkill. There's not a huge benefit to using seed_fu I think. In this current case I'm testing reporting code so the data is very specific and needs to be structured in a certain way in order to verify output data for the reports.
Related
I am having a very strange problem with loading a CSV file that is driving my absolutely crazy and doesn't make any sense to me! I am loading a CSV file into my database with the following code:
CSV.foreach(Rails.root.join('public','uploads', '0', csv_file.file_name), :headers => true, :header_converters => lambda { |h| h.try(:downcase)}) do |row|
Exclusion.create!(row.to_hash)
end
I have a file with 14,808 entries. If I try to load the file at once, for some reason it adds all 14,808 entries into the database as expected, but then starts over again from entry 1. It continues to do this in a recursive manner until I stop the server or it crashes. If I break the file down into two files, the individual files get added to the database as expected. I thought it may be a problem with the number of records, but I was able to load a csv file with about 100,000 records without this problem. I am very baffled as to why this is occurring. Oddly, if I comment out the create statement and just put a counter there, it stops at 14,808. Also, if I create a view that prints each row.to_hash, it stops at 14,808. I can't figure out what about saving it into the database would cause it to continue to repeat itself? I am using SQLite3, but again, I dont have a problem with a CSV file with 100,000 records.
Update:
Going through the log, it looks like the CSV file is loaded properly and all records are added to the database. Ruby even redirects to the proper url afterwards, but then seems to receive the "load file" request again. Since my screen has already timed out waiting for the server to process the CSV file, could that be causing an error and leading to either a duplicate request to load the file or the server thinking it hasn't processed the request and starting over?
Browsers sometimes repeat requests. If you have a catch-all route that goes to one action (the one that loads CSV file) then it could also be triggered by the browsers requesting favicon.ico.
However, I'd ask, why are you using a web request for this? Does csv_file come from the user? If not (i.e. if you already have the CSV file you want to load) I'd recommend putting this in a Rake task and just running it manually.
I am in the process of learning Rails and I've ran into an interesting problem tonight.
I was creating a migration that would require an index on the foreign key:
Whenever I would run 'bundle exec rake db:migrate', I would receive this console error:
It looks as if it was trying to create the index before it was creating the reference.
The reason I believe this is because when I change the "subject" reference to a symbol:
The migration then suddenly works as expected!
This may just be the fact that I'm a total newby, but are symbols actually processed faster by Ruby than strings are?
Just curious - thanks!
This isn't a "faster" problem, or a problem of speed. The migrations are executed one line at a time, in order. The way you had it specified before simply didn't create the column correctly, hence when it got to the line where you create the index, the names didn't match up.
My guess is, with the string version it created the column name exactly as you spelled it, "subject" as opposed to subject_id when you use a symbol. Either way, you definitely had a name mismatch between when the column was created, and when the index was being built.
Always use symbols for this in your migrations, and you should be fine. Always check your schema.rb file, or browse the database using a GUI tool, after a migration to make sure the columns got created the way you expect, and with the data types you think they are, and you should be good.
Per my requirements, I have created models for querying external database (different from the one that the rails app uses) for some of the data.
I am trying to write tests around these models and want to separate the "sample test data" from the actual tests.
I thought I could put the data in a yml file and load it into a hash, but it did work out :(
Added sample test data to a fixture file name 'external_database.yml'
Put the below code in setup, in the test file
ext_data = YAML.load_file(Rails.root.to_s + "/test/fixtures/ext_data.yml")
But I am stuck with the below error
1) Error:
test_should_errorout_for_invalid_market_zip(ExtDBTest):
ActiveRecord::StatementInvalid: Mysql::Error: Table 'rails_app_db.ext_data' doesn't exist: DELETE FROM ext_data
What is the best way to do what I want done?
I guess your problem is that the schema of that external database is not contained in your schema.rb-file and/or migrations. These are used to setup your test-database before you run the tests.
So the attempt is made to write those fixtures into non-existing tables - with the result you see above.
Multiple database-connections in unit tests are generally a pain. Consider creating an sqlite-file for the data of the external dependencies and configure your test-environment to use this file - or a copy of it, in case you need to mutate data.
I have an app that I've converted over from another cms. The old URLs were being stored in the database like so:
/this-is-an-old-permalink/
And I need them to be like this:
this-is-an-old-permalink
Note the absence of forward slashes. What is the easiest way to go about removing them?
I'm not necessarily looking for the exact code to do it (although that'd be nice!) -- I'm asking also as a Rails newb: What is the best method to go about doing things like this? I've only really worked with Rails in setting up a model, controller, views and outputting data. I haven't had to do any processing like this. Would it go in the model? Any help is appreciated!
edit
Do I need to get all records, loop through them, do regex on that one field and then save it?
Since you're likely only going to write this once, your best bet is to create a script for it within lib, or to write a migration for it. I recommend the latter, because it will then be executed automatically with rake db:migrate if you restore from your old backup at a later date. You can then use all your standard Model processing tricks (like you would use on a Controller) within the migration without exposing the substitution code to a Controller.
EDIT:
You can add the following to a new file within lib/tasks to create a new rake task for this called db:substitute_slashes:
namespace :db do
desc "Remove slashes from old-style URLs"
task :substitute_slashes => :environment do
Modelname.find(:all).each do |obj|
obj.fieldname.gsub!(/regex here/,'')
obj.save!
end
end
end
The exclamation on the end of save! means it will throw an exception if the resulting object fails validation, which is a good thing to check for in this case.
You would then be able to execute this with the command rake db:substitute_slashes.
This may well be a duplicate, but I couldn't find anyone asking quite this question.
My understanding* is that if I want to migrate data from an outside source to my Rails app's database, I should do so using a migration. It seems from my preliminary research that what I could do is use a tool like FasterCSV to parse a CSV file (for example) right in the migration (.rb) file itself.
Is this the correct approach? And if so, where should I actually put that CSV file -- it seems that if migrations are, after all, meant to be reversible/repeatable, that CSV data ought to be kept in a stable location.
*Let me know if I am completely mistaken about how to even go about this as I am still new to RoR.
You can write this to a rake job without FasterCSV, though I use both.
Write rows to 'csvout' file.
outfile = File.open('csvout', 'wb')
CSV::Writer.generate(outfile) do |csv|
csv << ['c1', nil, '', '"', "\r\n", 'c2']
...
end
outfile.close
This file will output where the rake file is written. In your case, you can put it in a seperate folder for CSV's. I would personally keep it out of the rest of the app structure.
You may want to look into seed_fu to manage it. It has the benefit of being able to easily update the data already in database. You can convert the CSV into a seed file, which is just a Ruby code (example code is provided there).