I have an API url that is a stream of data with the content type: text/event-stream.
How is it possible to listen to the stream? Like subsribe to each event to print the data? I have tried to use the ruby libary em-eventsource
My test.rb file:
require "em-eventsource"
EM.run do
source = EventMachine::EventSource.new("my_api_url_goes_here")
source.message do |message|
puts "new message #{message}"
end
source.start
end
When I visit my api url I can see the data updated each second. But when I run the ruby file in the terminal it does not print any data/messages.
Set a timer to check source.ready_state, it seems like it does not connect to api for some reason
EDIT: it seems your problem is in https' SNI, which is not supported by current eventmachine release, so upon connecting eventsource tries to connect to default virtual host on api machine, not the one the api is on, thus the CONNECTING state
Try using eventmachine from master branch, it already states to have support for SNI, that is going to be released in 1.2.0:
gem 'eventmachine', github:'eventmachine/eventmachine'
require 'eventmachine'
require 'em-http'
require 'json'
http = EM::HttpRequest.new("api_url", :keepalive => true, :connect_timeout => 0, :inactivity_timeout => 0)
EventMachine.run do
s = http.get({'accept' => 'application/json'})
s.stream do |data|
puts data
end
end
I used EventMachine http libary.
Related
I have a data scraper in ruby that retrieves article data.
Another dev on my team needs my scraper to spin up a webServer he can make a request to so that he may import the data on a Node Application he's built.
Being a junior, I do not understand the following :
a) Is there a proper convention in Rails that tells me where to place my scraper.rb file
b) Once that file is properly placed, how would i get the server to accept connections with the scrapedData
c)What (functionally) is the relationship between the ports, sockets, and routing
I understand this may be a "rookieQuestion" but I honestly dont know.
Can someone please BREAK THIS DOWN.
I have already:
i) Setup a server.rb file and have it linking to localhost:2000 but Im not sure how to create a proper route or connection that allows someone to use Postman for a valid route and connect to my data.
require 'socket'
require 'mechanize'
require 'awesome_print'
port = ENV.fetch("PORT",2000).to_i
server = TCPServer.new(port)
puts "Listening on port #{port}..."
puts "Current Time : #{Time.now}"
loop do
client = server.accept
client.puts "= Running Web Server ="
general_sites = [
"https://www.lovebscott.com/",
"https://bleacherreport.com/",
"https://balleralert.com/",
"https://peopleofcolorintech.com/",
"https://afrotech.com/",
"https://bossip.com/",
"https://www.itsonsitetv.com/",
"https://theshaderoom.com/",
"https://shadowandact.com/",
"https://hollywoodunlocked.com/",
"https://www.essence.com/",
"http://karencivil.com/",
"https://www.revolt.tv/"
]
holder=[]
agent = Mechanize.new
general_sites.each do |site|
page=agent.get(site);
newRet = page.search('a')
newRet.each do |e|
data = e.attr('href').to_s
if(data.length > 50)
holder.push(data)
end
end
pp holder.length.to_s + " [ posts total] ==> Now Scraping --> " + site
end
client.write(holder)
client.close
end
In Rails you don't spin up a web server manually, as it's done for you using rackup, unicorn, puma or any other compatible application server.
Rails itself is never "talking" to the HTTP clients directly, it is just a specific application that exposes a rack-compatible API (basically have an object that responds to call(hash) and returns [integer, hash, enumerable_of_strings]); the app server will get the data from unix/tcp sockets and call your application.
If you want to expose your scraper to an external consumer (provided it's fast enough), you can create a controller with a method that accepts some data, runs the scraper, and finally renders back the scraping results in some structured way. Then in the router you connect some URL to your controller method.
# config/routes.rb
post 'scrape/me', to: 'my_controller#scrape'
# app/controllers/my_controller.rb
class MyController < ApplicationController
def scrape
site = params[:site]
results = MyScraper.run(site)
render json: results
end
end
and then with a simple POST yourserver/scrape/me?site=www.example.com you will get back your data.
I am consuming an API using a gem in Ruby on Rails. The gem makes the API call for me and returns the status code as an integer (for example 200, 201 e.t.c.) and the data response from the API.
def get_cars
status_code, data = MyGem::Cars.get_cars
if status_code in SUCCESSFUL_RESPONSE_CODES
# Perform data manipulation
else
raise "There was an error processing the request. Status code #{status_code}"
end
end
Now I have manually initialised SUCCESSFUL_RESPONSE_CODES as a list containing integers of successful codes I found here.
Is this list defined somewhere in Ruby/RoR to avoid manually defining it?
I would expect any widely supported gem to use standard HTTP response codes to determine if the HTTP response was a success. For example:
require 'open-uri' # Rails loads this by default.
res = open('http://example.com')
res.status
=> ["200","OK"]
status.include?'OK'
=>true
status.include?'200'
=> true
So long as you trust the gem code making your request to handle standard HTTP response codes, you should be ok. Here's another example using HTTParty gem
require 'HTTParty'
res = HTTParty.get('https://example.com')
res.success?
=> true
How can I send data to a WebSocket using Ruby in a Background Process?
Background
I already have a separate ruby file running a Websocket server using the websocket-eventmachine-server gem. However, within my Rails application, I want to send data to the websocket in a background task.
Here is my WebSocket server:
EM.run do
trap('TERM') { stop }
trap('INT') { stop }
WebSocket::EventMachine::Server.start(host: options[:host], port: options[:port]) do |ws|
ws.onopen do
puts 'Client connected'
end
ws.onmessage do |msg, type|
ws.send msg, type: type
end
ws.onclose do
puts 'Client disconnected'
end
end
def stop
puts 'Terminating WebSocket Server'
EventMachine.stop
end
end
However, in my background process (I'm using Sidekiq), I'm not sure how to connect to the WebSocket and send data to it.
Here's my Sidekiq worker:
class MyWorker
include Sidekiq::Worker
def perform(command)
100.times do |i|
# Send 'I am on #{i}' to the Websocket
end
end
end
I was hoping to be able to do something like EventMachine::WebSocket.send 'My message!' but I don't see an API for that or something similar. What is the correct way to send data to a WebSocket in Ruby?
Accepted Answer:
Af your keeping your current websocket server:
You can use Iodine as a simple websocket client for testing. It runs background tasks using it's own reactor pattern based code and has a websocket client (I'm biased, I'm the author).
You could do something like this:
require 'iodine/http'
Iodine.protocol = :timers
# force Iodine to start immediately
Iodine.force_start!
options = {}
options[:on_message] = Proc.new {|data| puts data}
100.times do |i|
options[:on_open] = Proc.new {write "I am number #{i}"}
Iodine.run do
Iodine::Http.ws_connect('ws://localhost:3000', options)
end
end
P.S.
I would recommend using a framework, such as Plezi, for your websockets (I'm the author). Some frameworks let you run their code within a Rails/Sinatra app (Plezi does that and I think Faye, although not strictly a framework, does that too).
Using EM directly is quite hardcore and there are a lot of things to manage when dealing with Websockets, which a good framework helps you manage.
EDIT 3:
Iodine WebSocket client connections are (re)supported starting with Iodine 0.7.17, including TLS connections when OpenSSL >= 1.1.0.
The following code is an updated version of the original answer:
require 'iodine'
class MyClient
def on_open connection
connection.subscribe :updates
puts "Connected"
end
def on_message connection, data
puts data
end
def on_close connection
# auto-reconnect after 250ms.
puts "Connection lost, re-connecting in 250ms"
Iodine.run_after(250) { MyClient.connect }
end
def self.connect
Iodine.connect(url: "ws://localhost:3000/path", handler: MyClient.new)
end
end
Iodine.threads = 1
Iodine.defer { MyClient.connect if Iodine.master? }
Thread.new { Iodine.start }
100.times {|i| Iodine.publish :updates, "I am number #{i}" }
EDIT 2:
This answer in now outdated, since Iodine 0.2.x doesn't include a client any longer. Use Iodine 0.1.x or a different gem for websocket clients.
websocket-eventmachine-server is a websockets server.
If you want to connect to a websocket server using ruby, you can do it with some gems, like
https://github.com/igrigorik/em-websocket: Both server and client, also based on eventmachine.
ruby-websocket-client: Client only
I'm new with Rubyonrails and Koala gem and I'm sure I'm doing something wrong. I've been tuning my code to the minimun expression but the problem persist. Then I tried to do the same without koala gem, but the problem persisted.
This is the code:
require 'koala'
require 'open-uri'
puts Time.now
#graph = Koala::Facebook::API.new
resp = #graph.graph_call("cocacola", {}, "get", {})
puts resp
puts Time.now
coke_url = "https://graph.facebook.com/cocacola"
response = open coke_url
response = JSON.parse response.read
puts response.inspect
puts Time.now
I have to wait always 21 seconds the Facebook's response. If I put the https://graph.facebook.com/cocacola on my browser, the response is instantaneous ¿is not the same?
thanks
I'm using Koala in my application and did not have such experience. The only difference is that I don't use it anonymously. Instead I created an app and I'm using an access token to access the Facebook API. This might be the root cause, as I've found this post that also seems relates.
Finally, it was a DNS problem (thanks jpgeek).
When I did just a GET request to any website, the response was after 21 seconds, but using the IP of the same website, the response was instantaneously.
I found on google the solution: http://www.mikeperham.com/2010/02/10/asynchronous-dns-resolution/
I have use this personal solution in Gemfile (I'm not sure if it's the best):
group :development do
require 'resolv'
require 'resolv-replace'
end
Now it's working fine!
A client of mine has asked me to integrate a 3rd party API into their Rails app. The only problem is that the API uses SOAP. Ruby has basically dropped SOAP in favor of REST. They provide a Java adapter that apparently works with the Java-Ruby bridge, but we'd like to keep it all in Ruby, if possible. I looked into soap4r, but it seems to have a slightly bad reputation.
So what's the best way to integrate SOAP calls into a Rails app?
I built Savon to make interacting with SOAP webservices via Ruby as easy as possible.
I'd recommend you check it out.
We used the built in soap/wsdlDriver class, which is actually SOAP4R.
It's dog slow, but really simple. The SOAP4R that you get from gems/etc is just an updated version of the same thing.
Example code:
require 'soap/wsdlDriver'
client = SOAP::WSDLDriverFactory.new( 'http://example.com/service.wsdl' ).create_rpc_driver
result = client.doStuff();
That's about it
We switched from Handsoap to Savon.
Here is a series of blog posts comparing the two client libraries.
I also recommend Savon. I spent too many hours trying to deal with Soap4R, without results. Big lack of functionality, no doc.
Savon is the answer for me.
Try SOAP4R
SOAP4R
Getting Started with SOAP4R
And I just heard about this on the Rails Envy Podcast (ep 31):
WS-Deathstar SOAP walkthrough
Just got my stuff working within 3 hours using Savon.
The Getting Started documentation on Savon's homepage was really easy to follow - and actually matched what I was seeing (not always the case)
Kent Sibilev from Datanoise had also ported the Rails ActionWebService library to Rails 2.1 (and above).
This allows you to expose your own Ruby-based SOAP services.
He even has a scaffold/test mode which allows you to test your services using a browser.
I have used HTTP call like below to call a SOAP method,
require 'net/http'
class MyHelper
def initialize(server, port, username, password)
#server = server
#port = port
#username = username
#password = password
puts "Initialised My Helper using #{#server}:#{#port} username=#{#username}"
end
def post_job(job_name)
puts "Posting job #{job_name} to update order service"
job_xml ="<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:ns=\"http://test.com/Test/CreateUpdateOrders/1.0\">
<soapenv:Header/>
<soapenv:Body>
<ns:CreateTestUpdateOrdersReq>
<ContractGroup>ITE2</ContractGroup>
<ProductID>topo</ProductID>
<PublicationReference>#{job_name}</PublicationReference>
</ns:CreateTestUpdateOrdersReq>
</soapenv:Body>
</soapenv:Envelope>"
#http = Net::HTTP.new(#server, #port)
puts "server: " + #server + "port : " + #port
request = Net::HTTP::Post.new(('/XISOAPAdapter/MessageServlet?/Test/CreateUpdateOrders/1.0'), initheader = {'Content-Type' => 'text/xml'})
request.basic_auth(#username, #password)
request.body = job_xml
response = #http.request(request)
puts "request was made to server " + #server
validate_response(response, "post_job_to_pega_updateorder job", '200')
end
private
def validate_response(response, operation, required_code)
if response.code != required_code
raise "#{operation} operation failed. Response was [#{response.inspect} #{response.to_hash.inspect} #{response.body}]"
end
end
end
/*
test = MyHelper.new("mysvr.test.test.com","8102","myusername","mypassword")
test.post_job("test_201601281419")
*/
Hope it helps. Cheers.
I have used SOAP in Ruby when i've had to make a fake SOAP server for my acceptance tests. I don't know if this was the best way to approach the problem, but it worked for me.
I have used Sinatra gem (I wrote about creating mocking endpoints with Sinatra here) for server and also Nokogiri for XML stuff (SOAP is working with XML).
So, for the beginning I have create two files (e.g. config.rb and responses.rb) in which I have put the predefined answers that SOAP server will return.
In config.rb I have put the WSDL file, but as a string.
##wsdl = '<wsdl:definitions name="StockQuote"
targetNamespace="http://example.com/stockquote.wsdl"
xmlns:tns="http://example.com/stockquote.wsdl"
xmlns:xsd1="http://example.com/stockquote.xsd"
xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
xmlns="http://schemas.xmlsoap.org/wsdl/">
.......
</wsdl:definitions>'
In responses.rb I have put samples for responses that SOAP server will return for different scenarios.
##login_failure = "<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Body>
<LoginResponse xmlns="http://tempuri.org/">
<LoginResult xmlns:a="http://schemas.datacontract.org/2004/07/WEBMethodsObjects" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
<a:Error>Invalid username and password</a:Error>
<a:ObjectInformation i:nil="true"/>
<a:Response>false</a:Response>
</LoginResult>
</LoginResponse>
</s:Body>
</s:Envelope>"
So now let me show you how I have actually created the server.
require 'sinatra'
require 'json'
require 'nokogiri'
require_relative 'config/config.rb'
require_relative 'config/responses.rb'
after do
# cors
headers({
"Access-Control-Allow-Origin" => "*",
"Access-Control-Allow-Methods" => "POST",
"Access-Control-Allow-Headers" => "content-type",
})
# json
content_type :json
end
#when accessing the /HaWebMethods route the server will return either the WSDL file, either and XSD (I don't know exactly how to explain this but it is a WSDL dependency)
get "/HAWebMethods/" do
case request.query_string
when 'xsd=xsd0'
status 200
body = ##xsd0
when 'wsdl'
status 200
body = ##wsdl
end
end
post '/HAWebMethods/soap' do
request_payload = request.body.read
request_payload = Nokogiri::XML request_payload
request_payload.remove_namespaces!
if request_payload.css('Body').text != ''
if request_payload.css('Login').text != ''
if request_payload.css('email').text == some username && request_payload.css('password').text == some password
status 200
body = ##login_success
else
status 200
body = ##login_failure
end
end
end
end
I hope you'll find this helpful!
I was having the same issue, switched to Savon and then just tested it on an open WSDL (I used http://www.webservicex.net/geoipservice.asmx?WSDL) and so far so good!
https://github.com/savonrb/savon