Multiple HTTP requests matchable in one VCR cassette for rspec tests - ruby-on-rails

I have a spec file with an expectation that a controller action will return success.
The POST api/v1/users/:id/features/block action in the controller calls two HTTP calls on an external API, the only difference being in the body.
I've put the two requests and responses in the same VCR cassette, but when the cassette is being used, only the first request ever gets compared against and fails when it should be matching the second, causing the tests to fail.
What I'm looking for is a way of having the multiple requests match so the controller action completes and returns successfully.
The error I'm getting is at the end.
describe "POST /api/v1/users/:id/features/block" do
before(:each) do
#user = FactoryGirl.create(:user)
post :block, user_id: #user.id, block: "0"
end
it "should return 200 OK" do
expect(response).to be_success
end
end
Simplified versions of my VCR configuration and RSpec configuration follow:
VCR.configure do |c|
c.hook_into :webmock
c.default_cassette_options = {
match_requests_on: [:method, :uri, :body_regex]
}
c.register_request_matcher :body_regex do |request_1, request_2|
# Match body against regex if cassette body is "--ruby_regex /regexhere/"
if request_2.body[/^--ruby_regex\s*\//]
regex = request_2.body.gsub(/^--ruby_regex\s*\//, '').gsub(/\/$/, '')
request_1.body[/#{regex}/] ? true : false
else
true # No regex defined, continue processing
end
end
end
RSpec.configure do |c|
c.around(:each) do |example|
options = example.metadata[:vcr] || {}
name = example.metadata[:full_description].split(/\s+/, 2).join("/").underscore.gsub(/[^\w\/]+/, "_")
VCR.use_cassette(name, options, &example)
end
end
end
A summarized version of the cassette being used in this comparison that I'm having trouble with is:
---
http_interactions:
- request:
method: post
uri: https://upstream/api
body:
string: --ruby_regex /query1.+block/
response:
status:
code: 200
body:
string: { "response": "SUCCESS" }
- request:
method: post
uri: https://upstream/api
body:
string: --ruby_regex /query2.+block/
response:
status:
code: 200
body:
string: { "response": "SUCCESS" }
recorded_at: Fri, 05 Sep 2014 08:26:12 GMT
recorded_with: VCR 2.8.0
Error during tests:
An HTTP request has been made that VCR does not know how to handle
...
VCR is using the current cassette: (Correct cassette file path)
...
Under the current configuration VCR can not find a suitable HTTP interaction to replay and is prevented from recording new requests.
I don't want to record new requests because then the second one overwrites the first instead of adding the second request to the end of the cassette.

Related

Mock GoogleAPI request

I am using the google-maps gem.
I am trying to mock/stub api requests unsuccessfully.
module GoogleMap
class Route
attr_accessor :start_location, :end_location
def initialize(start_location, end_location)
#start_location = start_location
#end_location = end_location
end
def driving_duration_in_seconds
route.duration.value
end
def driving_distance_in_meters
route.distance.value
end
def driving_distance_hash
return unless start_location && end_location
{ distance_in_meters: driving_distance_in_meters, duration_in_seconds: driving_duration_in_seconds }
end
private
def coordinates_as_strings(location)
"#{location.latitude},#{location.longitude}"
end
def route
#route ||= Google::Maps.route(coordinates_as_strings(start_location), coordinates_as_strings(end_location))
end
end
end
I need to stub:
WebMock::NetConnectNotAllowedError:
Real HTTP connections are disabled. Unregistered request: GET https://maps.googleapis.com/maps/api/directions/json?destination=37.19687112,116.43791248&key=<apikey>&language=en&origin=2.15362819,-81.63712649 with headers {'Accept'=>'*/*', 'Date'=>'Sat, 12 Feb 2022 21:35:55 GMT', 'User-Agent'=>'HTTPClient/1.0 (2.8.3, ruby 2.7.2 (2020-10-01))'}
You can stub this request with the following snippet:
stub_request(:get, "https://maps.googleapis.com/maps/api/directions/json?destination=37.19687112,116.43791248&key=<apikey>&language=en&origin=2.15362819,-81.63712649").
with(
headers: {
'Accept'=>'*/*',
'Date'=>'Sat, 12 Feb 2022 21:35:55 GMT',
'User-Agent'=>'HTTPClient/1.0 (2.8.3, ruby 2.7.2 (2020-10-01))'
}).
to_return(status: 200, body: "", headers: {})
If I try a most basic stub I get an error:
stub_request(:any, /maps.googleapis.com/).
to_return(status: 200, body: '', headers: {})
Google::Maps::InvalidResponseException:
unknown error: 783: unexpected token at ''
# .../gems/ruby-2.7.2/gems/google-maps-3.0.7/lib/google_maps/api.rb:64:in `rescue in response'
# .../gems/ruby-2.7.2/gems/google-maps-3.0.7/lib/google_maps/api.rb:60:in `response'
# .../.rvm/gems/ruby-2.7.2/gems/google-maps-3.0.7/lib/google_maps/api.rb:27:in `query'
I think it is erroring out because I am not passing a key in. But I don't see why I should have to pass in a valid api key into a webmock.
I also would not have my route defined by anything. And in order to test that route can return route.distance.value etc, I would need to mock with something.
For other tests I was successful in mocking instances, but to test this lib that it actually works, I feel like mocking instance methods and not that an api was actually called is a waste of a test. Maybe this is just a waste of time, and I should assume it works because I am using a gem.
But I was expecting something like this:
RSpec.describe GoogleMap do
let(:start_location) { create(:location) }
let(:end_location) { create(:location) }
context 'GoogleMaps::Route.new(start_location, end_location)' do
let(:subject) { GoogleMap::Route.new(start_location, end_location) }
# I have not been successful in stubbing this with the correct regex
# stub_request(:get, "https://maps.googleapis.com/maps/api/directions/json?destination=<lat>,<long>key=<key>&language=en&origin=<lat>,<long>").
# with(
# headers: {
# 'Accept'=>'*/*',
# 'Date'=>'Thu, 10 Feb 2022 21:09:02 GMT',
# 'User-Agent'=>'HTTPClient/1.0 (2.8.3, ruby 2.7.2 (2020-10-01))'
# }).
# to_return(status: 200, body: "", headers: {})
# stub_request(:get, %r{https:\/\/maps\.googleapis\.com\/maps\/api\/directions\/json\?destination=.+,.+&key=.+&language=en&origin=.+,.+}).
# stub_request(:any, /maps.googleapis.com/).
# to_return(status: 200, body: '', headers: {})
xit 'gives driving distance in seconds'
xit 'gives driving duration in meters'
end
end
Your WebMock is working fine. Google::Maps::InvalidResponseException is raised after WebMock has replaced the network call. At the point that exception is raised, the Google Maps API client is trying to parse what the network call returned, which is ''.
It's expecting some valid JSON to be returned. If you have your mock return {} is should get past that line. It may well stumble on some other exception later though, as the gem expects a certain schema.
You can dig that out and add in a valid response if you wanted to continue down this path. However, I'd recommend not mocking the network request as that's an implementation detail of a third party piece of code which could change at any time - making your test fail. Instead, I would mock out Google::Maps.route to return what you need it to.

How do I test whether a Sidekiq worker is sending the right data to an external API?

I have a Sidekiq worker that reaches out to an external API to get some data back. I am trying to write tests to make sure that this worker is designed and functioning correctly. The worker grabs a local model instance and examines two fields on the model. If one of the fields is nil, it will send the other field to the remote API.
Here's the worker code:
class TokenizeAndVectorizeWorker
include Sidekiq::Worker
sidekiq_options queue: 'tokenizer_vectorizer', retry: true, backtrace: true
def perform(article_id)
article = Article.find(article_id)
tokenizer_url = ENV['TOKENIZER_URL']
if article.content.nil?
send_content = article.abstract
else
send_content = article.content
end
# configure Faraday
conn = Faraday.new(tokenizer_url) do |c|
c.use Faraday::Response::RaiseError
c.headers['Content-Type'] = 'application/x-www-form-urlencoded'
end
# get the response from the tokenizer
resp = conn.post '/tokenize', "content=#{URI.encode(send_content)}"
# the response's body contains the JSON for the tokenized and vectorized article content
article.token_vector = resp.body
article.save
end
end
I want to write a test to ensure that if the article content is nil that the article abstract is what is sent to be encoded.
My assumption is that the "right" way to do this would be to mock responses with Faraday such that I expect a specific response to a specific input. By creating an article with nil content and an abstract x I can mock a response to sending x to the remote API, and mock a response to sending nil to the remote API. I can also create an article with x as the abstract and z as the content and mock responses for z.
I have written a test that generically mocks Faraday:
it "should fetch the token vector on ingest" do
# don't wait for async sidekiq job
Sidekiq::Testing.inline!
# stub Faraday to return something without making a real request
allow_any_instance_of(Faraday::Connection).to receive(:post).and_return(
double('response', status: 200, body: "some data")
)
# create an attrs to hand to ingest
attrs = {
data_source: #data_source,
title: Faker::Book.title,
url: Faker::Internet.url,
content: Faker::Lorem.paragraphs(number: 5).join("<br>"),
abstract: Faker::Book.genre,
published_on: DateTime.now,
created_at: DateTime.now
}
# ingest an article from the attrs
status = Article.ingest(attrs)
# the ingest occurs roughly simultaneously to the submission to the
# worker so we need to re-fetch the article by the id because at that
# point it will have gotten the vector saved to the DB
#token_vector_article = Article.find(status[1].id)
# we should've saved "some data" as the token_vector
expect(#token_vector_article.token_vector).not_to eq(nil)
expect(#token_vector_article.token_vector).to eq("some data")
end
But this mocks 100% of uses of Faraday with :post. In my particular case, I have no earthly idea how to mock a response of :post with a specific body...
It's also possible that I'm going about testing this all wrong. I could be instead testing that we are sending the right content (the test should check what is being sent with Faraday) and completely ignoring the right response.
What is the correct way to test that this worker does the right thing (sends content, or sends abstract if content is nil)? Is it to test what's being sent, or test what we are getting back as a reflection of what's being sent?
If I should be testing what's coming back as a reflection of what's being sent, how do I mock different responses from Faraday depending on the value of something being sent to it/
** note added later **
I did some more digging and thought, OK, let me test that I'm sending the request I expect, and that I'm processing the response correctly. So, I tried to use webmock.
it "should fetch token vector for article content when content is not nil" do
require 'webmock/rspec'
# don't wait for async sidekiq job
Sidekiq::Testing.inline!
request_url = "#{ENV['TOKENIZER_URL']}/tokenize"
# webmock the expected request and response
stub = stub_request(:post, request_url)
.with(body: 'content=y')
.to_return(body: 'y')
# create an attrs to hand to ingest
attrs = {
data_source: #data_source,
title: Faker::Book.title,
url: Faker::Internet.url,
content: "y",
abstract: Faker::Book.genre,
published_on: DateTime.now,
created_at: DateTime.now
}
# ingest an article from the attrs
status = Article.ingest(attrs)
# the ingest occurs roughly simultaneously to the submission to the
# worker so we need to re-fetch the article by the id because at that
# point it will have gotten the vector saved to the DB
#token_vector_article = Article.find(status[1].id)
# we should have sent a request with content=y
expect(stub).to have_been_requested
# we should've saved "y" as the token_vector
expect(#token_vector_article.token_vector).not_to eq(nil)
expect(#token_vector_article.token_vector).to eq("y")
end
But I think that webmock isn't getting picked up inside the sidekiq job, because I get this:
1) Article tokenization and vectorization should fetch token vector for article content when content is not nil
Failure/Error: expect(stub).to have_been_requested
The request POST https://zzzzz/tokenize with body "content=y" was expected to execute 1 time but it executed 0 times
The following requests were made:
No requests were made.
============================================================
If I try to include webmock/rspec in any of the other places, for example, at the beginning of my file, random things start to explode. For example, if I have these lines in the beginning of this spec file:
require 'spec_helper'
require 'rails_helper'
require 'sidekiq/testing'
require 'webmock/rspec'
Then I get:
root#c18df30d6d22:/usr/src/app# bundle exec rspec spec/models/article_spec.rb:174
database: test
Run options: include {:locations=>{"./spec/models/article_spec.rb"=>[174]}}
There was an error creating the elasticsearch index for Article: #<NameError: uninitialized constant Faraday::Error::ConnectionFailed>
There was an error removing the elasticsearch index for Article: #<NameError: uninitialized constant Faraday::Error::ConnectionFailed>
Which I am guessing is because the test suite is trying to initialize stuff, but webmock is interfering...
I ended up abandoning Faraday and a more complicated test as an approach. I decomposed the worker into both a Service class and a worker. The worker simply invokes the Service class. This allows me to test the service class directly, and then just validate that the worker calls the service class correctly, and that the model calls the worker correctly.
Here's the much simpler service class:
require 'excon'
# this class is used to call out to the tokenizer service to retrieve
# a tokenized and vectorized JSON to store in an article model instance
class TokenizerVectorizerService
def self.tokenize(content)
tokenizer_url = ENV['TOKENIZER_URL']
response = Excon.post("#{tokenizer_url}/tokenize",
body: URI.encode_www_form(content: content),
headers: { 'Content-Type' => 'application/x-www-form-urlencoded' },
expects: [200])
# the response's body contains the JSON for the tokenized and vectorized
# article content
response.body
end
end
Here's the test to see that we are calling the right destination:
require 'rails_helper'
require 'spec_helper'
require 'webmock/rspec'
RSpec.describe TokenizerVectorizerService, type: :service do
describe "tokenize" do
it "should send the content passed in" do
request_url = "#{ENV['TOKENIZER_URL']}/tokenize"
# webmock the expected request and response
stub = stub_request(:post, request_url).
with(
body: {"content"=>"y"},
headers: {
'Content-Type'=>'application/x-www-form-urlencoded',
}).
to_return(status: 200, body: "y", headers: {})
TokenizerVectorizerService.tokenize("y")
expect(stub).to have_been_requested
end
end
end

How to stub HTTP request on Mechanize in Rails?

I have some codebase like this, and I wanna use rspec test favicon_href, but as you like, the favicon_href will call the page function, I know I can mock page function, but for this stage I wanna mock the HTTP request from the given url, so I use WebMock gem's syntax to stub HTTP request, but it seems WebMock is not compatibility with Mechanize, it always show the error in the below despite I relleay have done the stub, anyone know how can solve it or any gem can stub HTTP request on Mechanize?
Code
def favicon_href
#favicon_href ||=
begin
page.at(FAVICON_DOM).attributes['href'].value # finding <link> elements
rescue Exception
'/favicon.ico' # there are some situation the favicon's not <link>'
end
end
def page
#page ||= mechanize.get(url)
end
def mechanize
#mechanize ||= Mechanize.new
end
Error
Failure/Error: #page ||= mechanize.get(valid_url(url))
WebMock::NetConnectNotAllowedError:
Real HTTP connections are disabled. Unregistered request: GET https://tsaohucn.wordpress.com/ with headers {'Accept'=>'*/*', 'Accept-Charset'=>'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'Accept-Encoding'=>'gzip,deflate,identity', 'Accept-Language'=>'en-us,en;q=0.5', 'Connection'=>'keep-alive', 'Host'=>'tsaohucn.wordpress.com', 'Keep-Alive'=>'300', 'User-Agent'=>'Mechanize/2.7.5 Ruby/2.3.1p112 (http://github.com/sparklemotion/mechanize/)'}
You can stub this request with the following snippet:
stub_request(:get, "https://tsaohucn.wordpress.com/").
with(headers: {'Accept'=>'*/*', 'Accept-Charset'=>'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'Accept-Encoding'=>'gzip,deflate,identity', 'Accept-Language'=>'en-us,en;q=0.5', 'Connection'=>'keep-alive', 'Host'=>'tsaohucn.wordpress.com', 'Keep-Alive'=>'300', 'User-Agent'=>'Mechanize/2.7.5 Ruby/2.3.1p112 (http://github.com/sparklemotion/mechanize/)'}).
to_return(status: 200, body: "", headers: {})
registered request stubs:
stub_request(:get, "https://tsaohucn.wordpress.com/").
with(headers: {'Accept'=>'*/*', 'User-Agent'=>'Ruby'})
stub_request(:any, "http://api.stripe.com/")
stub_request(:any, "/api.stripe.com/")
============================================================
There exists an incompatibility between WebMock and net-http-persistent.
See
https://github.com/bblimke/webmock#connecting-on-nethttpstart
Add
WebMock.allow_net_connect!(:net_http_connect_on_start => true)
to your test set up.

Can I log the unhandled VCR request body?

I'm using the VCR gem to mock HTTP queries. I've recorded cassettes, but then I had to change some stuff around, and now I'm getting an error:
An HTTP request has been made that VCR does not know how to handle:
POST http://api.endpoint.here/path.json
Now, because it's a POST request, the VCR is configured to match those on body as well as path. Can I log or dump the body of the unhandled request so I can tweak the cassettes accordingly? Thank you.
Won't callback achieve what you need?
VCR.configure do |c|
c.after_http_request do |request, response|
if request.method == :post
puts "POST Request:#{request.uri}"
puts "#{request.to_hash}" # or request.body
end
end
c.allow_http_connections_when_no_cassette = true
end

Webmock and VCR, allow Http Connections if there is no cassette

I have a problem, I can run a test that uses vcr on its own and it works, it creates the cassette and it uses that on the next test. Great.
The problem is when I run all my tests together this particular test fails, because webmock disables http connections, I have seen this example on the Github repo page that explains how to expect real and not stubbed requests
My question is how Do I say: Allow Http connections for requests UNLESS there is a cassette. It should also CREATE the cassette when HTTP connections are allowed.
The VCR Settings
require 'vcr'
VCR.configure do | c |
if !ARGV.first.nil?
c.default_cassette_options = { :record => :new_episodes, :erb => true }
c.filter_sensitive_data('<BLACKBIRD_API_KEY>') {YAML.load(File.read('config/application.yml'))['BLACKBIRD_API_KEY'].to_s}
c.filter_sensitive_data('<BLACKBIRD_API_URL>') {YAML.load(File.read('config/application.yml'))['BLACKBIRD_API_URL'].to_s}
c.debug_logger = File.open(ARGV.first, 'w')
c.cassette_library_dir = 'spec/vcr'
c.hook_into :webmock
end
end
the above if statement exists because not EVERY test creates a cassette. So we want them to run when a cassette isn't needed.
The Test
require 'spec_helper'
describe Xaaron::Publishers::Users do
context "publish created users" do
before(:each) do
Xaaron.configuration.reset
no_user_member_roles_relation
Xaaron.configuration.publish_to_black_bird = true
Xaaron.configuration.black_bird_api_url = YAML.load(File.read('config/application.yml'))['BLACKBIRD_API_URL']
Xaaron.configuration.black_bird_api_key =YAML.load(File.read('config/application.yml'))['BLACKBIRD_API_KEY']
end
it "should publish to blackbird" do
VCR.use_cassette 'publisher/create_user_response' do
expect(
Xaaron::Publishers::Users.publish_new_user({user: {
first_name: 'adsadsad', user_name: 'sasdasdasdsa' ,
email: 'asdassad#sample.com', auth_token: 'asdsadasdasdsa'
}}).code
).to eql 200
end
end
end
end
Runs fine on its own, creates the cassette, fails when run with all other tests due to webmock.
The Failure
Failure/Error: Xaaron::Publishers::Users.publish_new_user({user: {
WebMock::NetConnectNotAllowedError:
Real HTTP connections are disabled. Unregistered request: GET some_site_url_here with headers {'Http-Authorization'=>'api_key_here', 'User-Agent'=>'Typhoeus - https://github.com/typhoeus/typhoeus'}
You can stub this request with the following snippet:
stub_request(:get, "some site url here").
with(:headers => {'Http-Authorization'=>'some api key here', 'User-Agent'=>'Typhoeus - https://github.com/typhoeus/typhoeus'}).
to_return(:status => 200, :body => "", :headers => {})

Resources