I am trying to make concurrent asynchronous HTTP calls using EventMachine::Iterator. I see too many calls ended up in errcallback. Is there any way to retry them? I see all these have deferred_status failed. How to avoid getting into that failed state.
urls = ["http://www.google.com"]*1000 #using this as an example
EventMachine.run do
EM::Iterator.new(urls, 1000).map(proc { |url, iter|
res = EventMachine::HttpRequest.new(url).get
sleep(0.100)
res.callback {
puts "#{res.req.uri} #{res.response_header.status}"
response =(JSON.parse(res.response)['response']) if res.response_header.status == 200
iter.return(response)
}
res.errback{
puts "Err => #{res.req.uri} #{res.response_header.status}"
iter.return(res.response_header.status)}
}, proc { |responses|
all = responses.flatten
puts 'all done!'
EM.stop
})
end
Related
Setup
I'm aware of how to do a Rails.exist ? Rails.read : fetch & rails.write but rails has a nice Rails.cache.fetch syntax that will automatically check if the cache key exists and is valid, return that if true, else run through the block and save to the cache.
Examples
For example (long way)
def search(param1)
if Rails.cache.exist?("key", namespace: "example")
response = Rails.cache.read("key", namespace: "example")
else
conn = faraday_helper(url: "search/url")
response = conn.post do |req|
req.body = { key: param1 }
end
Rails.cache.write("key", response, namespace: "example", expires_in: 1.hour) if response.success?
end
response
end
Short hand using fetch syntax
Rails.cache.fetch("key", namespace: "example") do
conn = faraday_helper(url: "search/url")
response = conn.post do |req|
req.body = { key: value }
end
response
end
This nice short hand does the same thing as the long way, except for the if response.success? which is what I'm interested in. If I make a call to this api, and the response is a 400 with a body of {"error": "invalid value for <key>"} the short way will cache that error response, which is no good. Edit for clarity: I do want that response body with the error, but I don't want to cache.
Question
Does anyone know of a way to pass a lambda or something to conditionally cache using the shorthand fetch syntax? I'd rather not have this method return nil when the cache fails because I want that error message in the response body, and deleting the cache if the response isn't a success seems to defeat the purpose of the entire thing (unless it's faster?)
How I've done it using fetch syntax
def search(param1:)
bailed_resp = nil
cached_resp = Rails.cache.fetch("key", namespace: "example", skip_nil: true, expires_in: 1.hour) do
conn = faraday_helper(url: "search/url")
response = conn.post do |req|
req.body = { key: param1 }
end
# Save the response for later and bail on caching unless response.success is true
bailed_resp = response
break unless response.success?
response
end
# If we bailed on the fetch, use the bailed response, otherwise use the new/fetched cache.
cached_resp == nil ? ResponseWrappers::Service.new(bailed_resp) : ResponseWrappers::Service.new(cached_resp)
end
Though this does work, I fail to see how it's any different than the long form syntax, which for reference:
def search(param1:)
if Rails.cache.exist?("key", namespace: "example")
response = Rails.cache.read("key", namespace: "example")
else
conn = faraday_helper(url: "search/url")
response = conn.post do |req|
req.body = { key: param1 }
end
Rails.cache.write("key", response, namespace: "example", expires_in: 1.hour) if response.success?
end
response
end
Is anyone able to give additional information on the differences between the two and/or if it's negligible?
You can just break from this block
Rails.cache.fetch("key", namespace: "example") do
conn = faraday_helper(url: "search/url")
response = conn.post do |req|
req.body = { key: value }
end
break unless response.success?
response
end
In this case nothing will be written by this key for failure response
But if you try to repeat this code and response will be ok, it will be written
If you want to use this construction in some method and need this method to return response, you change it to:
def search
Rails.cache.fetch("key", namespace: "example") do
conn = faraday_helper(url: "search/url")
response = conn.post do |req|
req.body = { key: value }
end
return response unless response.success?
response
end
end
And process result outside the method
I have implemented PayPal checkout API in my rails application by using the SmartButtons and by creating the order in the server-side.
I have used the payouts-ruby-sdk gem and my code is as follows:-
index.html.erb
<!-- Set up a container element for the button -->
<div id="paypal-button-container"></div>
<!-- Include the PayPal JavaScript SDK -->
<script src="https://www.paypal.com/sdk/js?client-id=xyz¤cy=USD"></script>
<script>
// Render the PayPal button into #paypal-button-container
paypal.Buttons({
// Call your server to set up the transaction
createOrder: function(data, actions) {
return fetch('/orders', {
method: 'post'
}).then(function(res) {
return res.json();
}).then(function(orderData) {
return orderData.orderID;
});
},
// Call your server to finalize the transaction
onApprove: function(data, actions) {
return fetch('/orders/' + data.orderID + '/capture', {
method: 'post'
}).then(function(res) {
return res.json();
}).then(function(orderData) {
// Three cases to handle:
// (1) Recoverable INSTRUMENT_DECLINED -> call actions.restart()
// (2) Other non-recoverable errors -> Show a failure message
// (3) Successful transaction -> Show a success / thank you message
// Your server defines the structure of 'orderData', which may differ
var errorDetail = Array.isArray(orderData.details) && orderData.details[0];
if (errorDetail && errorDetail.issue === 'INSTRUMENT_DECLINED') {
// Recoverable state, see: "Handle Funding Failures"
// https://developer.paypal.com/docs/checkout/integration-features/funding-failure/
return actions.restart();
}
if (errorDetail) {
var msg = 'Sorry, your transaction could not be processed.';
if (errorDetail.description) msg += '\n\n' + errorDetail.description;
if (orderData.debug_id) msg += ' (' + orderData.debug_id + ')';
// Show a failure message
return alert(msg);
}
// Show a success message to the buyer
alert('Transaction completed');
});
}
}).render('#paypal-button-container');
</script>
orders_controller.rb
class OrdersController < ApplicationController
skip_before_action :verify_authenticity_token
def index
end
def create
# Creating Access Token for Sandbox
client_id = 'xyz'
client_secret = 'abc'
# Creating an environment
environment = PayPal::SandboxEnvironment.new(client_id, client_secret)
client = PayPal::PayPalHttpClient.new(environment)
request = PayPalCheckoutSdk::Orders::OrdersCreateRequest::new
request.request_body({
intent: "CAPTURE",
purchase_units: [
{
amount: {
currency_code: "USD",
value: "10.00"
}
}
]
})
begin
# Call API with your client and get a response for your call
# debugger
response = client.execute(request)
puts response.result.id
render json: {success: true, orderID: response.result.id}
rescue PayPalHttp::HttpError => ioe
# Something went wrong server-side
puts ioe.status_code
puts ioe.headers["debug_id"]
end
end
def execute_payment
client_id = 'xyz'
client_secret = 'abc'
# Creating an environment
environment = PayPal::SandboxEnvironment.new(client_id, client_secret)
client = PayPal::PayPalHttpClient.new(environment)
request = PayPalCheckoutSdk::Orders::OrdersCaptureRequest::new(session[:orderID])
begin
# Call API with your client and get a response for your call
response = client.execute(request)
# If call returns body in response, you can get the deserialized version from the result attribute of the response
order = response.result
puts order
rescue PayPalHttp::HttpError => ioe
# Something went wrong server-side
puts ioe.status_code
puts ioe.headers["debug_id"]
end
end
end
Now I want to implement the Paypal's Payouts API and I know that paypal-ruby-sdk is available for it but I am confused where to fit this code and how to integrate it with the front end. Any ideas? Thanks in advance :)
Your code above is Checkout code, for both front-end (JavaScript), and back-end (Ruby).
Payouts has nothing to do with Checkout, neither front-end Checkout nor back-end Checkout.
Payouts is strictly a backend API operation, where you send money from your account to another account.
Payouts does not connect to any front-end UI. You can build your own UI to trigger a payout, if you need one. Presumably you know who you want to send money from your account to, and what process should trigger this action.
EDIT: here https://github.com/wujek-srujek/reactor-retry-test is a repository with all the code.
I have the following Spring WebClient code to POST to a remote server (Kotlin code without imports for brevity):
private val logger = KotlinLogging.logger {}
#Component
class Client(private val webClient: WebClient) {
companion object {
const val maxRetries = 2L
val firstBackOff = Duration.ofSeconds(5L)
val maxBackOff = Duration.ofSeconds(20L)
}
fun send(uri: URI, data: Data): Mono<Void> {
return webClient
.post()
.uri(uri)
.contentType(MediaType.APPLICATION_JSON)
.bodyValue(data)
.retrieve()
.toBodilessEntity()
.doOnSubscribe {
logger.info { "Calling backend, uri: $uri" }
}
.retryExponentialBackoff(maxRetries, firstBackOff, maxBackOff, jitter = false) {
logger.debug { "Call to $uri failed, will retry (#${it.iteration()} of max $maxRetries)" }
}
.doOnError {
logger.error { "Call to $uri with $maxRetries retries failed with $it" }
}
.doOnSuccess {
logger.info { "Call to $uri succeeded" }
}
.then()
}
}
(It returns an empty Mono as we don't expect an answer, nor do we care about it.)
I would like to test 2 cases, and one of them is giving me headaches, namely the one in which I want to test that all the retries have been fired. We are using MockWebServer (https://github.com/square/okhttp/tree/master/mockwebserver) and the StepVerifier from reactor-test. (The test for success is easy and doesn't need any virtual time scheduler magic, and works just fine.) Here is the code for the failing one:
#JsonTest
#ContextConfiguration(classes = [Client::class, ClientConfiguration::class])
class ClientITest #Autowired constructor(
private val client: Client
) {
lateinit var server: MockWebServer
#BeforeEach
fun `init mock server`() {
server = MockWebServer()
server.start()
}
#AfterEach
fun `shutdown server`() {
server.shutdown()
}
#Test
fun `server call is retried and eventually fails`() {
val data = Data()
val uri = server.url("/server").uri()
val responseStatus = HttpStatus.INTERNAL_SERVER_ERROR
repeat((0..Client.maxRetries).count()) {
server.enqueue(MockResponse().setResponseCode(responseStatus.value()))
}
StepVerifier.withVirtualTime { client.send(uri, data) }
.expectSubscription()
.thenAwait(Duration.ofSeconds(10)) // wait for the first retry
.expectNextCount(0)
.thenAwait(Duration.ofSeconds(20)) // wait for the second retry
.expectNextCount(0)
.expectErrorMatches {
val cause = it.cause
it is RetryExhaustedException &&
cause is WebClientResponseException &&
cause.statusCode == responseStatus
}
.verify()
// assertions
}
}
I am using withVirtualTime because I don't want the test to take nearly seconds.
The problem is that the test blocks indefinitely. Here is the (simplified) log output:
okhttp3.mockwebserver.MockWebServer : MockWebServer[51058] starting to accept connections
Calling backend, uri: http://localhost:51058/server
MockWebServer[51058] received request: POST /server HTTP/1.1 and responded: HTTP/1.1 500 Server Error
Call to http://localhost:51058/server failed, will retry (#1 of max 2)
Calling backend, uri: http://localhost:51058/server
MockWebServer[51058] received request: POST /server HTTP/1.1 and responded: HTTP/1.1 500 Server Error
Call to http://localhost:51058/server failed, will retry (#2 of max 2)
As you can see, the first retry works, but the second one blocks. I don't know how to write the test so that it doesn't happen. To make matters worse, the client will actually use jitter, which will make the timing hard to anticipate.
The following test using StepVerifier but without WebClient works fine, even with more retries:
#Test
fun test() {
StepVerifier.withVirtualTime {
Mono
.error<RuntimeException>(RuntimeException())
.retryExponentialBackoff(5,
Duration.ofSeconds(5),
Duration.ofMinutes(2),
jitter = true) {
println("Retrying")
}
.then()
}
.expectSubscription()
.thenAwait(Duration.ofDays(1)) // doesn't matter
.expectNextCount(0)
.expectError()
.verify()
}
Could anybody help me fix the test, and ideally, explain what is wrong?
This is a limitation of virtual time and the way the clock is manipulated in StepVerifier. The thenAwait methods are not synchronized with the underlying scheduling (that happens for example as part of the retryBackoff operation). This means that the operator submits retry tasks at a point where the clock has already been advanced by one day. So the second retry is scheduled for + 1 day and 10 seconds, since the clock is at +1 day. After that, the clock is never advanced so the additional request is never made to MockWebServer.
Your case is made even more complicated in the sense that there is an additional component involved, the MockWebServer, that still works "in real time".
Though advancing the virtual clock is a very quick operation, the response from the MockWebServer still goes through a socket and thus has some amount of latency to the retry scheduling, which makes things more complicated from the test writing perspective.
One possible solution to explore would be to externalize the creation of the VirtualTimeScheduler and tie advanceTimeBy calls to the mockServer.takeRequest(), in a parallel thread.
With ActionCable, how can I respond with an error after receiving data from a client?
For example, when the client fails to authenticate, ActionCable throws UnauthorizedError which responds with a 404. I want to respond with a 422, for example, when the data that the client sent is invalid.
ActionCable.server.broadcast "your_channel", message: {data: data, code: 422}
Then in your.coffee file:
received: (res) ->
switch res.code
when 200
# your success code
# use res.data to access your data message
when 422
# your error handler
From what I could gather there's no "Rails way" to do this, the answer given by #Viktor seems correct. In summary: ensure all messages are broadcasted with both data and code, then switch by code in the client.
For a more modern and ES6 example see below:
In rails:
require 'json/add/exception'
def do_work
// Do work or raise
CampaignChannel.broadcast_to(#campaign, data: #campaign.as_json, code: 200)
rescue StandardError => e
CampaignChannel.broadcast_to(#campaign, data: e.to_json, code: 422) # Actually transmitting the entire stacktrace is a bad idea!
end
In ES6:
campaignChannel = this.cable.subscriptions.create({ channel: "CampaignChannel", id: campaignId }, {
received: (response) => {
const { code, data } = response
switch (code) {
case 200:
console.log(data)
break
default:
console.error(data)
}
}
})
In a grails integration test, I have code that resembles this:
def ctrlA = new MyController()
... make some request that returns 'ok' ...
assert ctrlA.response.json.status == 'ok'
def ctrlB = new MyController()
... make some request that returns 'error' ...
assert ctrlB.response.json.status == 'error' // fails; status still equals 'ok'
Problem: Even when ctrlB actually does return a json response that looks like { status: 'error' }, I'm actually seeing { status: 'ok' }, the value that was in ctrlA.response.json!! My logs in the controller indicate that 'error' is most definitely being returned.
Why is this?
Ah. Don't need the separate ctrlA and ctrlB at all. Just call ctrl.response.reset() in between.