I'm trying to run cucumber using VCR gem, chrome driver, and webmock but it's not working. One problem is that none of the requests are getting recorded. I do have VCR set to ignore_localhost which I believe is needed due to javascript as explained in this SO answer.
But why doesn't my step request to visit a url (say www.test.com) get recorded? I see the chrome browser loading up and the address in the URL so I know the cucumber step is going there.
If i turn off ignore_localhost then I see some recordings but the uri's are all 127.0.0.1. The uri I'd expect is showing up in body: string instead of the uri. Why is this happening?
request:
method: post
uri: http://127.0.0.1:9592/session/87bb12aa8b5b1230399b98bcc7d0ba11/url
body:
encoding: US-ASCII
string: ! '{"url":"http://www.test.com"}'
headers:
Accept:
- application/json
Content-Type:
- application/json; charset=utf-8
Content-Length:
- '46'
User-Agent:
- Ruby
Incase it helps, here's my VCR config
VCR.configure do |c|
c.cassette_library_dir = 'features/support/vcr_cassettes'
c.hook_into :webmock
c.ignore_localhost = true
end
First of all, set your configuration like this:
VCR.configure do |c|
c.cassette_library_dir = 'features/support/vcr_cassettes'
c.hook_into :webmock
c.ignore_localhost = true
c.allow_http_connections_when_no_cassette = false
end
With the last option you tell VCR that no real http requests are allowed, without a cassette. Now run your tests. I bet that it'll throw an error! Now you have to tell VCR when to use which cassette. Have a look at the cucumber helpers for VCR here https://www.relishapp.com/vcr/vcr/v/2-4-0/docs/test-frameworks/usage-with-cucumber
VCR includes some helpers for other testing frameworks too https://www.relishapp.com/vcr/vcr/v/2-4-0/docs/test-frameworks
Related
I'm trying to configure Faraday tracing in a Rails application using Datadog.
I've set up the Faraday -> Datadog connection:
require 'faraday'
require 'ddtrace'
Datadog.configure do |c|
c.use :faraday
end
Faraday.post("http://httpstat.us/200", {foo: 1, bar: 2}.to_json)
Faraday.get("http://httpstat.us/201?foo=1&bar=2")
It works well, the requests are being logged to Datadog.
But those logs do not contain any request parameters, nevertheless GET or POST.
Any adviŅe on how to get the request params/body logged to Datadog?
So, by default the only things sent to Datadog from Faraday as part of the span in terms of the HTTP request are are:
span.set_tag(Datadog::Ext::HTTP::URL, env[:url].path)
span.set_tag(Datadog::Ext::HTTP::METHOD, env[:method].to_s.upcase)
span.set_tag(Datadog::Ext::NET::TARGET_HOST, env[:url].host)
span.set_tag(Datadog::Ext::NET::TARGET_PORT, env[:url].port)
Source: https://github.com/DataDog/dd-trace-rb/blob/e391d2eb64d3c6151a4bdd2710c6a8c7c1d57457/lib/ddtrace/contrib/faraday/middleware.rb#L54
The body of the request is not set in the http part of the span by default, only the URL, HTTP method, host and port are.
However, with manual instrumentation you can add anything you want to the span, so you could write an extension or monkey-patch to the Faraday middleware to add the body and parameters to the span:
span.set_tag("http.body", env[:body])
span.set_tag("http.params", env[:params])
An example monkey patch:
require 'faraday'
require 'ddtrace'
require 'ddtrace/contrib/faraday/middleware'
module FaradayExtension
private
def annotate!(span, env, options)
# monkey patch to add body to span
span.set_tag("http.body", env[:body]) unless env[:body].to_s.strip.empty?
span.set_tag("http.query", env[:url].query) if env[:url].query
super
end
end
Datadog::Contrib::Faraday::Middleware.prepend(FaradayExtension)
Datadog.configure do |c|
c.use :faraday
end
Faraday.post("http://httpstat.us/200", {foo: 1, bar: 2}.to_json)
Faraday.get("http://httpstat.us/201?foo=1&bar=2")
This worked for me in my testing:
NB: I am a Datadog employee, but not on the engineering team, just wanted to be transparent!
I'm using Net::HTTP with Ruby to crawl an URL.
I don't want to crawl streaming audio such as: http://listen2.openstream.co/334
in fact i only want to crawl Html content, so no pdfs, video, txt..
Right now, I have both open_timeout and read_timeout set to 10, so even if I do crawl these streaming audio pages they will timeout.
url = 'http://listen2.openstream.co/334'
path = uri.path
req= Net::HTTP::Get.new(path, {'Accept' => '*/*', 'Content-Type' => 'text/plain; charset=utf-8', 'Connection' => 'keep-alive','Accept-Encoding' => 'Identity'})
uri = Addressable::URI.parse(url)
resp = Net::HTTP.start(uri.host, uri.inferred_port) do |httpRequest|
httpRequest.open_timeout = 10
httpRequest.read_timeout = 10
#how can I read the headers here before it's streaming the body and then exit b/c the content type is audio?
httpRequest.request(req)
end
However, is there a way to check the header BEFORE I read the body of a http response to see if it's an audio? I want to do so without sending a separate HEAD request.
net/http supports streaming, you can use this to read the header before the body.
Code example,
url = URI('http://stackoverflow.com/questions/41306082/ruby-nethttp-read-the-header-before-the-body-without-head-request')
Net::HTTP.start(url.host, url.port) do |http|
request = Net::HTTP::Get.new(url)
http.request(request) do |response|
# check headers here, body has not yet been read
# then call read_body or just body to read the body
if true
response.read_body do |chunk|
# process body chunks here
end
end
end
end
I will add a ruby example later tonight. However, for a quick response. There is a simple trick to do this.
You can use HTTP Range header to indicate if which range of bytes you want to receive from the server. Here is an example:
curl -XGET http://www.sample-videos.com/audio/mp3/crowd-cheering.mp3 -v -H "Range: bytes=0-1"
The above example means the server will return data from 0 to 1 byte range.
See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests
Since I did not find a way to properly do this in Net::HTTP, and I saw that you're using the addressable gem as an external dependency already, here's a solution using the wonderful http gem:
require 'http'
response = HTTP.get('http://listen2.openstream.co/334')
# Here are the headers
puts response.headers
# Everything ok? Start streaming the response
body = response.body
body.stream!
# now just call `readpartial` on the body until it returns `nil`
# or some other break condition is met
Sorry if you're required to use Net::HTTP, hopefully someone else will find an answer. A separate HEAD request might indeed be the way to go in that case.
You can do a whole host of net related things without using a gem. Just use the net/http module.
require 'net/http'
url = URI 'http://listen2.openstream.co/334'
Net::HTTP.start(url.host, url.port){|conn|
conn.request_get(url){|resp|
resp.each{|k_header, v_header|
# process headers
puts "#{k_header}: #{v_header}"
}
#
# resp.read_body{|body_chunk|
# # process body
# }
}
}
Note: while processing headers, just make sure to check the content-type header. For audio related content it would normally contain audio/mpeg value.
Hope, it helped.
Ruby 1.9.3, RSpec 2.13.0, WebMock 1.17.4, Rails 3
I am writing tests for a company app. The controller in question displays a table of a customer's placed calls, and allows for sort/filter options.
EDIT The test fails because with my current setup, the path does not render, because the recorder_server is either not running locally, OR not setup correctly. Please help with this, too.
A Errno::ECONNREFUSED occurred in recordings#index:
Connection refused - connect(2)
/usr/local/lib/ruby/1.9.1/net/http.rb:763:in `initialize'
-------------------------------
Request:
-------------------------------
* URL : http://www.recorder.example.com:8080/recorded_calls
* IP address: 127.0.0.1
* Parameters: {"controller"=>"recordings", "action"=>"index"}
* Rails root: /var/www/rails/<repository>
As a call is placed, its data joins an xml file, created by an external API, called Recorder
The RecordingsController takes the xml file, and parses it into a hash.
When you visit the associated path, you see the results of the hash -- a table of placed calls, their attributes, and parameters for sort/filter.
Here is my spec so far.
require 'spec_helper'
include Helpers
feature 'Exercise recordings controller' do
include_context "shared admin context"
background do
canned_xml = File.open("spec/support/assets/canned_response.xml").read
stub_request(:post, "http://recorder.example.com:8080/recorder/index").
with(body: {"durations"=>["1"], "durations_greater_less"=>["gt"], "filter_from_day"=>"29", "filter_from_hour"=>"0", "filter_from_minute"=>"0", "filter_from_month"=>"12", "filter_from_year"=>"2014", "filter_prefix"=>true, "filter_to_day"=>"29", "filter_to_hour"=>"23", "filter_to_minute"=>"59", "filter_to_month"=>"12", "filter_to_year"=>"2014"}, # "shared_session_id"=>"19f9a08807cc70c1bf41885956695bde"},
headers: {'Accept'=>'*/*', 'Content-Type'=>'application/x-www-form-urlencoded', 'User-Agent'=>'Ruby'}).
to_return(status: 200, body: canned_xml, headers: {})
uri = URI.parse("http://recorder.example.com:8080/recorder/index")
visit recorded_calls_path
end
scenario 'show index page with 1 xml result' do
#page.save_and_open_page
expect(title).to eq("Recorded Calls")
end
end
And here is the RecordingsController
class RecordingsController < ApplicationController
# before_filter options
def index
test_session_id = request.session_options[:id]
#Make request to recording app for xml of files
uri = URI.parse("http://#{Rails.application.config.recorder_server}:#{Rails.application.config.recorder_server_port}/recorder/index")
http = Net::HTTP.new(uri.host, uri.port)
xml_request = Net::HTTP::Post.new(uri.request_uri)
xml_request_data = Hash.new
# sorting params
xml_request_data[:shared_session_id] = request.session_options[:id]
xml_request.set_form_data(xml_request_data)
response = http.request(xml_request)
if response.class == Net::HTTPOK
#recordings_xml = XmlSimple.xml_in(response.body)
#recordings_sorted = #recordings_xml["Recording"].sort { |a,b| Time.parse("#{a["date"]} #{a["time"]}") <=> Time.parse("#{b["date"]} #{b["time"]}") } unless #recordings_xml["Recording"].nil?
else #recordings_xml = Hash.new
end
end
# other defs
end
Any and all advice is much appreciated. Thank you.
How I configured WebMock
I am answering my own question, with the help of B-Seven and a string of comments. File by file, I will list the changes made in order to properly use WebMock.
Add WebMock to Gemfile under group :test, :development.
bundle install to resolve dependencies
my current setup included Ruby 1.9.3, Rails 2.13.0, WebMock 1.17.4
Setup spec_helper.rb to disable "Real HTTP connections". (This was a backtrace error received later on in this puzzling process.) This allows, to my understanding, all "real connections" to translate into localhost connections and work offline... Which is great since, ideally, I do not want the external app's server to run simultaneously.
require 'webmock/rspec'
WebMock.disable_net_connect!(allow_localhost: true)
In my test.rb environment file, the configurations for recorder_server and port were commented out... If left uncommented, the controller would raise an exception stating uninitialized constants. I used the test server/port (substituting the company name for example) as my layout for the spec stubbing.
In recordings_controller_spec.rb, I had already figured out how to make a canned XML response. With these changes above, my spec was able to correctly stub a response on an external, secondary app, and use such response to correctly render the view associated with the controller being tested.
require 'spec_helper'
include Helpers
feature "Exercise recordings_controller" do
include_context "shared admin context"
# A background is currently not used, because I have 3 scenario types... No xml
# results, 1 result, and 2 results. I will later DRY this out with a background,
# but the heavy lifting is over, for now.
scenario "show index page with 1 xml result" do
canned_xml_1 = File.open("spec/support/assets/canned_response_1.xml").read
stub_request(:post, "http://recorder.example.com:8080/recorder/index").
with(headers: {'Accept'=>'*/*', 'User-Agent'=>'Ruby'}).
to_return(status: 200, body: canned_xml_1, headers: {})
uri = URI.parse("http://recorder.example.com:8080/recorder/index")
visit recorded_calls_path
title.should == "Recorded Calls"
page.should have_content("Search Results")
page.should have_content("Inbound", "5551230000", "175", "December 24 2014", "12:36:24", "134")
end
end
Advice/Resources that helped
With B-Seven's suggestion to my original question (see revisions), I was initially stubbing localhost:3000. He said this was incorrect. After further research, I agree since stubbing with WebMock is typically reserved for outside http connections.
In comments after his answer, B-Seven listed articles to refer to. I will list the ones that helped me the most.
http://robots.thoughtbot.com/how-to-stub-external-services-in-tests
http://railscasts.com/episodes/275-how-i-test
https://github.com/bblimke/webmock
http://www.agileventures.org/articles/testing-with-rspec-stubs-mocks-factories-what-to-choose
It is very important to read the backtrace generated from an errors. What took me so long to figure out how to mock was mainly reading them incorrectly. As you can see from my question, I was making a :get stub request. A coworker pointed out that the backtrace suggested to use :post. That was the final piece to make my spec pass.
I decided not to input the configuration variables as my stub request, for it would result in long lines of code. Instead, this is why I needed to uncomment out those configurations in test.rb.
Why are you stubbing localhost? I think you want to
stub_request(:get, "http://#{Rails.application.config.recorder_server}:#{Rails.application.config.recorder_server_port}/recorder/index").
In my cucumber support directory I have the following in vcr.rb:
require 'vcr'
VCR.configure do |c|
c.cassette_library_dir = 'fixtures/vcr_cassettes'
c.hook_into :webmock
c.ignore_localhost = true
c.default_cassette_options = { record: :new_episodes }
end
I am geocoding city names which makes calls to the Google Maps API. I'm trying to record and stub these requests, but it keeps recording the same requests to the same yml file:
- request:
method: get
uri: http://maps.googleapis.com/maps/api/geocode/json?address=Miami,%20FL&language=en&sensor=false
body:
encoding: US-ASCII
string: ''
headers:
Accept-Encoding:
- gzip;q=1.0,deflate;q=0.6,identity;q=0.3
Accept:
- ! '*/*'
User-Agent:
- Ruby
# response...
- request:
method: get
uri: http://maps.googleapis.com/maps/api/geocode/json?address=Miami,%20FL&language=en&sensor=false
body:
encoding: US-ASCII
string: ''
headers:
Accept-Encoding:
- gzip;q=1.0,deflate;q=0.6,identity;q=0.3
Accept:
- ! '*/*'
User-Agent:
- Ruby
It's the same URL and very same request, shouldn't VCR stub the request? How can I prevent my specs from hitting the API every time I try to search for the same city?
It's hard to say for sure what's going on with just what you've posted, but I can explain a bit more about how VCR works and make some guesses about possible reasons for this behavior.
VCR uses request matchers to try to find a previously recorded HTTP interaction to play back. During a single cassette session, when an HTTP interaction is played back, it is considered to be "used" and it will not be played back again (unless you use the allow_playback_repeats option).
So...here are a couple possibilities that come to mind:
Maybe VCR is unable to match your HTTP requests. What request matchers are you using? There are some easy ways to troubleshoot this (see below).
If you're not using :allow_playback_repeats (which is the default, and how I recommend you use VCR), then the behavior you're seeing could happen if multiple duplicate requests are being made in your test--e.g., maybe the cassette has only one matching request but you're test makes 2 of them--that would play back one and record one (since you're using :new_episodes).
To troubleshoot this, I recommend you use the debug_logger option to get VCR to print what it's doing and how it's trying to match each request. That should give you insight into what's going on. You can also override any of the built in request matchers and provide your own logic and/or set a breakpoint in the matcher:
VCR.configure do |c|
c.register_request_matcher :uri do |request_1, request_2|
debugger # so you can inspect the requests here
request_1.uri == request_2.uri
end
end
You may also have run into a VCR bug, although comparing the URIs (using String#==) is such a basic operation that I have a hard time imagining a bug there. Feel free to open a github issue (hopefully with the debug logger output and/or a code sample that triggers this) if you can't figure it out.
On a side note, I recommend you use the :once record mode (the default) rather than :new_episodes. :once will never record additional HTTP interactions to an existing cassette--it only allows the cassette to be recorded once. If a request fails to match, it'll raise an error alerting you to the fact it couldn't match. :new_episodes, on the other hand, will record any request that it can't find a match for, which is the behavior you're seeing.
When I had a similar problem, I fixed it by making the match_requests_on setting more specific:
VCR.configure do |c|
c.default_cassette_options = {
match_requests_on: [:uri, :body, :method]
}
end
I experienced similar behaviour, so what I do is basically keeping to set to :none. If any new requests come up, I use :any, run the part of the test suite that did the requests and set it back to :none.
Seems like :new_episodes uses some strange heuristics to detemermine what new requests are and what requests already occurred. In our case it marked two different requests to the payment gateways as the same ones, leading to endless hours of debugging - because we got a RefundOk answer to a CaptureRequest and things like that. Better don't use :new_episodes...
Using Elasticsearch with VCR will always regenerate the cassettes if you not define the match parameters. I've to define the match to only :method because the :uri and :body could change each time that the test is running.
VCR.configure do |c|
c.hook_into :webmock
c.ignore_localhost = true
c.configure_rspec_metadata!
c.cassette_library_dir = 'spec/cassettes'
c.default_cassette_options = { record: :new_episodes,
match_requests_on: [:method] }
end
A client of mine has asked me to integrate a 3rd party API into their Rails app. The only problem is that the API uses SOAP. Ruby has basically dropped SOAP in favor of REST. They provide a Java adapter that apparently works with the Java-Ruby bridge, but we'd like to keep it all in Ruby, if possible. I looked into soap4r, but it seems to have a slightly bad reputation.
So what's the best way to integrate SOAP calls into a Rails app?
I built Savon to make interacting with SOAP webservices via Ruby as easy as possible.
I'd recommend you check it out.
We used the built in soap/wsdlDriver class, which is actually SOAP4R.
It's dog slow, but really simple. The SOAP4R that you get from gems/etc is just an updated version of the same thing.
Example code:
require 'soap/wsdlDriver'
client = SOAP::WSDLDriverFactory.new( 'http://example.com/service.wsdl' ).create_rpc_driver
result = client.doStuff();
That's about it
We switched from Handsoap to Savon.
Here is a series of blog posts comparing the two client libraries.
I also recommend Savon. I spent too many hours trying to deal with Soap4R, without results. Big lack of functionality, no doc.
Savon is the answer for me.
Try SOAP4R
SOAP4R
Getting Started with SOAP4R
And I just heard about this on the Rails Envy Podcast (ep 31):
WS-Deathstar SOAP walkthrough
Just got my stuff working within 3 hours using Savon.
The Getting Started documentation on Savon's homepage was really easy to follow - and actually matched what I was seeing (not always the case)
Kent Sibilev from Datanoise had also ported the Rails ActionWebService library to Rails 2.1 (and above).
This allows you to expose your own Ruby-based SOAP services.
He even has a scaffold/test mode which allows you to test your services using a browser.
I have used HTTP call like below to call a SOAP method,
require 'net/http'
class MyHelper
def initialize(server, port, username, password)
#server = server
#port = port
#username = username
#password = password
puts "Initialised My Helper using #{#server}:#{#port} username=#{#username}"
end
def post_job(job_name)
puts "Posting job #{job_name} to update order service"
job_xml ="<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:ns=\"http://test.com/Test/CreateUpdateOrders/1.0\">
<soapenv:Header/>
<soapenv:Body>
<ns:CreateTestUpdateOrdersReq>
<ContractGroup>ITE2</ContractGroup>
<ProductID>topo</ProductID>
<PublicationReference>#{job_name}</PublicationReference>
</ns:CreateTestUpdateOrdersReq>
</soapenv:Body>
</soapenv:Envelope>"
#http = Net::HTTP.new(#server, #port)
puts "server: " + #server + "port : " + #port
request = Net::HTTP::Post.new(('/XISOAPAdapter/MessageServlet?/Test/CreateUpdateOrders/1.0'), initheader = {'Content-Type' => 'text/xml'})
request.basic_auth(#username, #password)
request.body = job_xml
response = #http.request(request)
puts "request was made to server " + #server
validate_response(response, "post_job_to_pega_updateorder job", '200')
end
private
def validate_response(response, operation, required_code)
if response.code != required_code
raise "#{operation} operation failed. Response was [#{response.inspect} #{response.to_hash.inspect} #{response.body}]"
end
end
end
/*
test = MyHelper.new("mysvr.test.test.com","8102","myusername","mypassword")
test.post_job("test_201601281419")
*/
Hope it helps. Cheers.
I have used SOAP in Ruby when i've had to make a fake SOAP server for my acceptance tests. I don't know if this was the best way to approach the problem, but it worked for me.
I have used Sinatra gem (I wrote about creating mocking endpoints with Sinatra here) for server and also Nokogiri for XML stuff (SOAP is working with XML).
So, for the beginning I have create two files (e.g. config.rb and responses.rb) in which I have put the predefined answers that SOAP server will return.
In config.rb I have put the WSDL file, but as a string.
##wsdl = '<wsdl:definitions name="StockQuote"
targetNamespace="http://example.com/stockquote.wsdl"
xmlns:tns="http://example.com/stockquote.wsdl"
xmlns:xsd1="http://example.com/stockquote.xsd"
xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
xmlns="http://schemas.xmlsoap.org/wsdl/">
.......
</wsdl:definitions>'
In responses.rb I have put samples for responses that SOAP server will return for different scenarios.
##login_failure = "<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Body>
<LoginResponse xmlns="http://tempuri.org/">
<LoginResult xmlns:a="http://schemas.datacontract.org/2004/07/WEBMethodsObjects" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
<a:Error>Invalid username and password</a:Error>
<a:ObjectInformation i:nil="true"/>
<a:Response>false</a:Response>
</LoginResult>
</LoginResponse>
</s:Body>
</s:Envelope>"
So now let me show you how I have actually created the server.
require 'sinatra'
require 'json'
require 'nokogiri'
require_relative 'config/config.rb'
require_relative 'config/responses.rb'
after do
# cors
headers({
"Access-Control-Allow-Origin" => "*",
"Access-Control-Allow-Methods" => "POST",
"Access-Control-Allow-Headers" => "content-type",
})
# json
content_type :json
end
#when accessing the /HaWebMethods route the server will return either the WSDL file, either and XSD (I don't know exactly how to explain this but it is a WSDL dependency)
get "/HAWebMethods/" do
case request.query_string
when 'xsd=xsd0'
status 200
body = ##xsd0
when 'wsdl'
status 200
body = ##wsdl
end
end
post '/HAWebMethods/soap' do
request_payload = request.body.read
request_payload = Nokogiri::XML request_payload
request_payload.remove_namespaces!
if request_payload.css('Body').text != ''
if request_payload.css('Login').text != ''
if request_payload.css('email').text == some username && request_payload.css('password').text == some password
status 200
body = ##login_success
else
status 200
body = ##login_failure
end
end
end
end
I hope you'll find this helpful!
I was having the same issue, switched to Savon and then just tested it on an open WSDL (I used http://www.webservicex.net/geoipservice.asmx?WSDL) and so far so good!
https://github.com/savonrb/savon