I'm new to ruby and rspec.
I have a module that interacts with S3 in ruby.
In my code I :
create a new S3 instance : s3 = AWS::S3.new()
Get my bucket : #s3bucket = s3.buckets[#bucket]
Retrieve my S3Object : object = #s3bucket.objects[key]
Finally, I save the object to a local file:
File.open(local_filename, 'wb') do |s3file|
object.read do |chunk|
return completed if stop?
s3file.write(chunk)
end
My code works well, but I'm having problems unit testing it,
specifically I'm having problems mocking the object.read do |chunk| part.
No matter what I try the chunk turns out empty.
Can some one help?
Thanks.
try this:
s3_object = class_double("S3Object")
allow(s3_object).to receive(:read).and_return(<what ever you want>)
If you need to need to store API responses in your tests without making multiple calls, check out https://github.com/vcr/vcr
If you want to mock this:
create a new S3 instance : s3 = AWS::S3.new()
you do
allow_any_instance_of(AWS::S3).to_receive(:new).and_return(<the return value of the method>)
You can use VCR as suggested previously but you will run into issues if you are working on a team and you run your tests at the same time as another team member if you both have deleted the same cassette.
Related
When the client changes his profile picture it hits the update method, which responds with update.js.erb. This is a fast and straightforward process. However, behind the scenes on the server, a bunch of files (10 of them) is generated from the profile picture and these need to be uploaded to an Amazon bucket from the server. This a lengthy process and I don't want to make the client wait until it is finished. Moreover, the file uploads often fail with a RequestTimeoutException because they take longer than 15 seconds.
All this raises many questions:
How do you do the 10 file generation/upload after the update method has exited? Threads are killed after the main method has finished.
How do you catch an exception inside a thread? The following code does not catch the timeout exceptions.
threads = []
threads << Thread.new {
begin
# upload file 1 ....
rescue Rack::Timeout::RequestTimeoutException => e
# try to upload again ....
else
ensure
end
}
threads << Thread.new {
begin
# upload file 2 ....
rescue Rack::Timeout::RequestTimeoutException => e
# try to upload again ....
else
ensure
end
}
threads.each { |thr|
thr.join
}
What's the best way to try to upload a file again if it timed out?
What is the best solution to this problem?
You need to use delayed_job or whenever gem for background task, but I would like suggest sidekiq
I also faced the same problem in a project. I came accross a solution using AWS lambda. You can use carrierwave gem/ rails 5 active storage module if you are using rails to upload image on S3. If you are not using rails then use AWS-SDK for ruby to upload files on S3. You can bind events whenever a file created/modified on S3. Whenever a file created it will invoke lambda function and your work is done. can bind them to lambda function. In lambda function you can write logic to create files and upload it back to s3. You can write lambda code in ruby, node and python.
This strategy may help you.
I am trying to check whether a particular pdf file exists on AWS S3 using aws-sdk gem (version 2) inside ruby on rails application.
I have the AWS connection established and currently using exists? method:
puts #bucket.objects(prefix:"path/sample_100.pdf").exists?
on running the above statement, I get the below no method error:
undefined method 'exists?' for Aws::Resources::Collection
Checked few documents but of not much help. Is there any other way to achieve the same?
Thanks in advance
I'm not a Ruby developer myself, but I might be able to suggest something.
The usual way to check whether an object exists in Amazon S3 is using the HEAD Object operation. Basically, it returns the metadata (but no content) of an object if it exists, or a 404 error if it doesn't. It's like GET Object, but without the contents of the object.
I just looked up in the AWS SDK for Ruby API Reference and found this method:
http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html#head_object-instance_method
Take a look at that, it's probably what you are looking for.
I'd recommend you to use the much simpler S3 gem: https://github.com/qoobaa/s3 If you only need to deal with S3. You'll be able to do it this way:
object = bucket.objects.find("example.pdf")
As mentioned by Bruno, you can use head_object to get info on the file, without actually fetching it. If it is not found (or other problems, such as permissions), an exception will be raised. So if head_object returns, the file exists.
Here's a file that exists:
> head = s3.head_object(bucket: bucket, key: path)
=> #<struct Aws::S3::Types::HeadObjectOutput last_modified=2020-06-05 16:18:05 +0000, content_length=553, etc...>
And here's one that does not exist, and the exception it raises:
> path << '/not-really'
=> "example/file/not-really"
> head = s3.head_object(bucket: bucket, key: path)
Aws::S3::Errors::NotFound
Traceback (most recent call last):
1: from (irb):18
Aws::S3::Errors::NotFound ()
And here's how you can roll your own s3_exists? method:
def s3_exists?(bucket, path)
s3.head_object(bucket: bucket, key: path)
true
rescue
false
end
I have a block of code that creates Tempfiles
#tmp_file = Tempfile.new("filename")
I keep them closed after creation,
#tmp_file.close unless #tmp_file.closed?
When there is a need to add data to the temp files I open them and add data as below
def add_row_to_file(row)
#tmp_file.open
#tmp_file.read
#tmp_file.print(row.to_json + "\n")
end
All is well, but for testing the same I have stubbed tempfile as below and is creating an error when the test case runs into add_row_to_file(row)
buffers = {}
Tempfile.stub(:new) do |file_name|
buffer = StringIO.new
buffers[file_name] = buffer
end
Error message is :
Failure/Error: ]],
NoMethodError:
private method `open' called for #<StringIO:0x00000010b867c0>
I want to keep the temp files closed on creation as there is a max temp files open issue at OS level (I have to deal with uploading lot of tempfiles to S3)
but for testing I have a problem accessing the private method of StringIO.
Any idea how to solve this problem? Thanks.
I have a work around, which is to skip closing the StringIO when in test environment.
#tmp_file.close unless #tmp_file.closed? || Rails.env.test?
and update add_row_to_file(row) as below
def add_row_to_file(row)
#tmp_file.open unless Rails.env.test?
#tmp_file.read unless Rails.env.test?
#tmp_file.print(row.to_json + "\n")
end
Apart from the work-around provided if we want 100% code coverage while in test we can try the below.
Do not close the stream if it is StringIO
#tmp_file.close unless #tmp_file.is_a?(StringIO)
so we dont need to open it while in test.
def add_row_to_file(row)
#tmp_file = File.open(#tmp_file, 'a+') unless #tmp_file.is_a?(StringIO)
#tmp_file.print(row.to_json + "\n")
end
In this way we can achieve the Tempfile testing without actually creating Files while in test environment and having 100% code coverage while in test.
The gem I'm using to integrate OpenTok in my Rails application is at: https://github.com/opentok/Opentok-Ruby-SDK. I based the core of the application on this example: http://www.tokbox.com/blog/building-a-video-party-app-with-ruby-on-rails.
In the relevant part of code, I'm creating an #opentok object in the config_opentok method:
def config_opentok
if #api_key.nil? or #api_secret.nil?
if Rails.env.development?
#api_key = API_KEY
#api_secret = API_SECRET
else
#api_key = ENV['API_KEY']
#api_secret = ENV['API_SECRET']
end
end
if #opentok.nil?
#opentok = OpenTok::OpenTokSDK.new(#api_key, #api_secret)
end
end
And I'm creating a session with the following code:
config_opentok
if Rails.env.development?
session = #opentok.create_session('localhost')
else
session = #opentok.create_session(request.remote_addr)
end
The trouble is, the create_session seems to throw an error
SocketError: getaddrinfo: nodename nor servname provided, or not known
whenever I run my Rspec tests without an internet connection. So I'd like to stub that method so that it returns just a hash {:sessionId => 1}. But I'm having trouble figuring out how to stub the method. I can't just stub the OpenTok module or the OpenTok::OpenTokSDK class. How would I go about stubbing the create_session method?
here's what I've been doing that works:
First, what I tend to do is to initialize the OpenTok object when the app loads so I'm not creating an OpenTok object on every request. To do this, I create a ruby file (apis.rb) in my config/initializers folder.
My apis.rb looks like this:
TB_KEY = ENV['TB_KEY']
TB_SECRET = ENV['TB_SECRET']
OTSDK = OpenTok::OpenTokSDK.new TB_KEY, TB_SECRET
In my controller, to generate a session I'll simply call OTSDK.createSession, similar to what you already have.
To test with rspec, you can simply write in your test file:
OTSDK.stub(:createSession).and_return( {:sessionId => "1MX_2A3453095J0TJ30..."} )
If you run rspec with wifi turned off calling createSession should no longer throw an error.
Here's the documentation for rspec stubbing: http://rubydoc.info/gems/rspec-mocks/frames
Good Luck!
The trouble is, the create_session seems to throw an error whenever I run my Rspec tests without an internet connection.
Instead of attempting to stub, why not give your tests a mock internet connection with VCR?
After initial set up, VCR lets you run all of your tests as if you were actively connected to the internet. This allows you to run tests offline, speeds up all the tests that needed an active connection, and gives you a consistent set of results.
If you have a subscription to RailsCasts, Ryan made a video about VCR in episode 291
My application accepts file uploads, with some metadata being stored in the DB, and the file itself on the file system. I am trying to make the metadata visible in the application before the file upload and post-processing are finished, but because saves are transactional, I have had no success. I have tried the callbacks and calling create_or_update() instead of save(), all to no avail. Is there a way to do this without re-writing the guts of ActiveRecord::Base? I've even attempted naming the method make() instead of save(), but perplexingly that had no effect.
The code below "works" fine, but the database is not modified until everything else is finished.
def save(upload)
uploadFile = upload['datafile']
originalName = uploadFile.original_filename
self.fileType = File.extname(originalName)
create_or_update()
# write the file
File.open(self.filePath, "wb") { |f| f.write(uploadFile.read) }
begin
musicFile = TagLib::File.new(self.filePath())
self.id3Title = musicFile.title
self.id3Artist = musicFile.artist
self.id3Length = musicFile.length
rescue TagLib::BadFile => exc
logger.error("Failed to id track: \n #{exc}")
end
if(self.fileType == '.mp3')
convertToOGG();
end
create_or_update()
end
Any ideas would be quite welcome, thanks.
Have you considered processing the file upload as a background task? Save the metadata as normal and then perform the upload and post-processing using Delayed Job or similar. This Railscast has the details.
You're getting the meta-data from the file, right? So is the problem that the conversion to OGG is taking too long, and you want the data to appear before the conversion?
If so, John above has the right idea -- you're going to need to accept the file upload, and schedule a conversion to occur sometime in the future.
The main reason why is that your rails thread will process the OGG conversion and can't respond to any other web-requests until it's complete. Blast!
Some servers compensate for this by having multiple rails threads, but I recommend a background queue (use BJ if you host yourself, or Heroku's background jobs if you host there).