I'm writing trading bot on ruby and I need to constantly do some calculations based on Exchange's orderbook depth data across several daemons (daemons gem).
The problem is that right now I'm fetching data via Exchange's API separately in each daemon, so I ran into API calls limit (40 requests/second). That's why I'm trying to utilize ruby Drb to share orderbook data across several processes (daemons) in order to not sending unnecessary API calls.
Although I m not sure how to constantly consume API on the server side and provide the latest actual data to the client process. In the sample code below, client will get data actual at the moment when I started server
server_daemon.rb
require 'drb'
exchange = exchange.new api_key: ENV['APIKEY'], secret_key: ENV['SECRET']
shared_orderbook = exchange.orderbook limit: 50
DRb.start_service('druby://127.0.0.1:61676', shared_orderbook)
puts 'Listening for connection… '
DRb.thread.join
client_daemon.rb
require 'drb'
DRb.start_service
puts shared_data = DRbObject.new_with_uri('druby://127.0.0.1:61676')
Related
Taking the file upload and websocket examples from Fast API, consider a POST route:
#app.post("/uploadfiles/")
async def create_upload_files(files: List[UploadFile] = File(...)):
data = [file.filename for file in files]
return process_data(data)
And consider the websocket
#app.websocket("/ws/{client_id}")
async def websocket_endpoint(websocket: WebSocket, client_id: int):
await manager.connect(websocket)
try:
while True:
data = await websocket.receive_text()
await manager.send_personal_message(f"You wrote: {data}", websocket)
# Replace with update based on process_data() status
except WebSocketDisconnect:
manager.disconnect(websocket)
How can I update the websocket on the progress of any post requests for a given client_id?
I've looked through dependency injections and background tasks, and I can't seem to figure out a way to do it.
Ultimately, the tool I'm trying to build:
User uploads multiple files on client side
Server processes the files (which can take minutes to hours), and will ultimately return a single file to the user (e.g., redirect to download the file)
Meanwhile, I'd like the user to be updated on the progress on the data processing (e.g., via websockets).
If the user closes the tab (client side disconnect), the original data proecss job is stopped.
I've read about polling and other approaches, but it seems like web sockets ought to work.
I just want to test the Front-End part. So, here is my problem:
Background
I have a robust Ruby on Rails (V3.2) backend app and an entiry new and separate front-end app with ReactJs (V16.4).
Problem
We begin to test React app with the help of Selenium-Webdriver and JestJs, we managed to try several views, but the problem arose when we made POST requests to the rails API.
I don't want to fill my database (development) with garbage because of the tests.
Ex: What happens when I want to test the creation of a new user?.
Possible solutions thought
I was thinking in 3 solutions:
Intercept the API calls and mock them by imitating their response (ex: at submitting click using selenium-webdriver).
Make use of Rails test environment through React
Just revert the call of the API doing the opposite, this would mean creating often undesirable actions in the controller. (ex: doing a delete for each post)
It depends if you want to test the whole stack (frontend/backend) or only the frontend part.
Frontend tests
If you only want to test the frontend part go with your first solution : mock API calls.
You will be limited if you just use the selenium-webdriver directly. I would recommend using nightwatch or testcafe. Testcafe does not depend on selenium. This is also optional in the latest versions of Nightwatch.
Testcafe includes a Request mocking API : http://devexpress.github.io/testcafe/documentation/test-api/intercepting-http-requests/mocking-http-responses.html
With Nightwatch you could use nock. See Nightwatch Mock HTTP Requests
Full stack tests
If you want to test the whole stack, you may use this approach : implement a custom API endpoint to allow for resetting your database in a clean state before or after tests execution. (like "/myapi/clean")
You should disable access to this endpoint in production environments.
You can then implement test hooks (before/after) to call your custom api endpoint :
http://nightwatchjs.org/guide#using-before-each-and-after-each-hooks
http://devexpress.github.io/testcafe/documentation/test-api/test-code-structure.html#test-hooks
You could have a test environment. From my experience, garbage data generated by tests is not such a big deal. You can periodically clean it up. Or you can spin up a new environment for every test run.
Finally I decided to use enzyme with jest and sinon.
example code:
import { mount } from "enzyme";
import sinon from "sinon";
beforeAll(() => {
server = sinon.fakeServer.create();
const initialState = {
example: ExampleData,
auth: AuthData
};
wrapper = mount(
<Root initialState={initialState}>
<ExampleContainer />
</Root>
);
});
it("example description", () => {
server.respondWith("POST", "/api/v1/example", [
200,
{ "Content-Type": "application/json" },
'message: "Example message OK"'
]);
server.respond();
expect(wrapper.find(".response").text().to.equal('Example message OK');
})
In the code above we can see how to intercept API calls using the test DOM created by the enzyme and then mock API responses using sinon.
The text here: https://developers.google.com/fusiontables/docs/v2/migration_guide implies that the 10MB limit not in effect for API v2, or that an alternative service "Media download" could be used for large responses.
The API Reference here: https://developers.google.com/fusiontables/docs/v2/reference/ does not have any information regarding the 10MB limit, or how you use "media download" to recieve your request.
How do I work around the 10MB limit for Fusion Tables API v2? I can't seem to find documentation that explains it.
To use media-download simply add the parameter alt=media to the URL
For those who use Google's API Client Libraries, the 'media download' is specified by using a specific method. For the Python library, there are two versions of the SQL query methods: sql*() and sql*_media() (and this is very likely true for the other client libraries as well).
Example usage:
# Build the googleapiclient service
FusionTables = build('fusiontables', 'v2', credentials=credentials);
query = 'select * from <table id>';
# "standard" query, returning fusiontables#sqlresponse JSON:
jsonRequest = FusionTables.query().sqlGet(sql = query);
jsonResponse = jsonRequest.execute();
# alt=media query, returning a CSV-formatted bytestring (in Python, at least):
bytestrRequest = FusionTables.query().sqlGet_media(sql = query);
byteResponse = bytestrRequest.execute();
As Kerry mentions here, media format queries that are too large to be sent as a GET request will fail (while regular format queries of the same length succeed provided the query result is less than 10 MB). In the python client, this failure appears as a HTTP 502: Bad Gateway error.
Also note that ROWIDs are currently not returned in the media response format.
I'm using rspec to test my application and I'm having a hard time figuring out how to test this. The Slack::Notifier's job is to send a post request to a webhook. Once I call this method in Rspec, I don't know how to see the response. Also, is it possible to match the format of this text to an expected text somewhere? My method is below. Thanks.
def notify
offset = 14400 #UTC to EST
notifier = Slack::Notifier.new Rails.application.secrets.slack_organization_name, Rails.application.secrets.slack_token, channel: "##{Rails.application.secrets.slack_channel}", username: Rails.application.secrets.slack_user_name
notifier.ping(":white_check_mark: *USAGE SUMMARY for #{(Time.now - offset).to_formatted_s(:long) }*")
count = 0
current_time = Time.now.to_i
live_response.each do |r|
if r["properties"]["time"] > ((current_time - offset) - 60) #&& r["properties"]["$initial_referring_domain"] == "capture.com"
notifier.ping("
*Name:* #{r["properties"]["$name"]}
*Event:* #{r["event"]}
*Keywords:* #{r["properties"]["keywords"]}
*Organization:* #{r["properties"]["organizationName"]}
*Email:* #{r["properties"]["$email"]}
*Time:* #{Time.at(r["properties"]["time"] + offset).utc.to_datetime.in_time_zone("Eastern Time (US & Canada)").to_formatted_s(:long_ordinal)}
*More Data:* #{ANALYTICS_URL}#{r["properties"]["distinct_id"]}
__________________________________________________
")
count +=1
end
end
notifier.ping("*There were #{count} events in this report.*")
end
Testing network communications (like API calls) is a tricky thing. Personally I would rely on programming by contract and testing in isolation - i.e. assume the external service is working fine and it responds positively for valid request.
Then you test your client code by checking that you are actually sending a valid request. For this stub the method where control exits your code into a library/system code. For example if you are making a HTTP GET request using a gem like HTTParty, then stub HTTParty.get i.e. HTTParty.stub(:get) and in that stub verify that correct parameters were sent.
On the other side of the spectrum you should also simulated both positive and negative responses from the web service and make sure your client code handles it in expected manner.
If you are making a real then you are introducing a lot of dependencies on your test : a test setup of external service, risk of network issues (timeout, n/w breakdown, etc) problems with external service and may be more.
If you yourself are writing that webservice too then test that one also in isolation, i.e by simulating valid and invalid inputs and making sure they are handled properly. This part is pretty much your controller specs or request specs.
Once again, this is my opinion. Suggestions to do this in a better way and constructive criticism on the shortcomings of this approach are definitely welcome.
I have a Rails application that has a Document with the flag available. The document is uploaded to an external server where it is not immediately available (takes time to propogate). What I'd like to do is poll the availability and update the model when available.
I'm looking for the most performant solution for this process (service does not offer callbacks):
Document is uploaded to app
app uploads to external server
app polls url (http://external.server.com/document.pdf) until available
app updates model Document.available = true
I'm stuck on 3. I'm already using sidekiq in my project. Is that an option, or should I use a completely different approach (cron job).
Documents will be uploaded all the time and so it seems relevant to first poll the database/redis to check for Documents which are not available.
See this answer: Making HTTP HEAD request with timeout in Ruby
Basically you set up a HEAD request for the known url and then asynchronously loop until you get a 200 back (with a 5 second delay between iterations, or whatever).
Do this from your controller after the document is uploaded:
Document.delay.poll_for_finished(#document.id)
And then in your document model:
def self.poll_for_finished(document_id)
document = Document.find(document_id)
# make sure the document exists and should be polled for
return unless document.continue_polling?
if document.remote_document_exists?
document.available = true
else
document.poll_attempts += 1 # assumes you care how many times you've checked, could be ignored.
Document.delay_for(5.seconds).poll_for_finished(document.id)
end
document.save
end
def continue_polling?
# this can be more or less sophisticated
return !document.available || document.poll_attempts < 5
end
def remote_document_exists?
Net::HTTP.start('http://external.server.com') do |http|
http.open_timeout = 2
http.read_timeout = 2
return "200" == http.head(document.path).code
end
end
This is still a blocking operation. Opening the Net::HTTP connection will block if the server you're trying to contact is slow or unresponsive. If you're worried about it use Typhoeus. See this answer for details: What is the preferred way of performing non blocking I/O in Ruby?