Currently I have a lighttpd server with flup and webpy. If you make enough requests fast enough (say clicking a link repeatedly many times or doing an apache bench) Lighttpd throws a 500 internal server error. At this point it is pretty easy to exploit (I can break it with several clicks of a link).
Lighttpd code:
fastcgi.server = (
"/sm" => (
( "host" => "127.0.0.1",
"port" => 7000,
"check-local" => "disable",
)
)
)
And the error in my lighttpd error logs:
2012-11-09 16:17:32: (mod_fastcgi.c.3005) got proc: pid: 0 socket: tcp:127.0.0.1:7000 load: 15
2012-11-09 16:17:32: (mod_fastcgi.c.2494) unexpected end-of-file (perhaps the fastcgi process died): pid: 0 socket: tcp:127.0.0.1:7000
2012-11-09 16:17:32: (mod_fastcgi.c.3325) response not received, request sent: 1252 on socket: tcp:127.0.0.1:7000 for /sm , closing connection
2012-11-09 16:17:32: (mod_fastcgi.c.1515) released proc: pid: 0 socket: tcp:127.0.0.1:7000 load: 14
This makes me feel like lighttpd is breaking because flup didn't respond. Now, I can simply throw more threads at the problem and it goes away (or at least makes it harder to exploit). flup server code:
#!/usr/bin/python
from apps.main import app as main_app
# run as fastcgi
from flup.server.fcgi import WSGIServer
params = {
'multiplexed': False,
'bindAddress': ('127.0.0.1', 7000),
'maxThreads': 9, <---- If I move this up to 20 no more problems
}
server = WSGIServer(main_app.wsgifunc(), **params)
server.run()
Another reason I think this is a flup problem is because I can bypass the flup server and just do a proxy that sends requests directly to webpy and I don't have the problem. Now, I would rather not just up the threads if there is a more elegant solution out there. Does anyone know what could be causing flup to break? Or are my conclusions thus far misplaced?
Related
I have the cowboy example eventsource running on a local Debian server. For the code please see --> https://github.com/ninenines/cowboy/tree/master/examples/eventsource
After about 60 seconds there is always a 'eventsource was closed' then a 'eventsource connected' message.
I am testing on the latest chrome browser on win10.
I cannot see any reason for this and I wondered if this something built into the SSE standard, that the connection is dropped at regular intervals.
The chrome debugger shows the following error message:
GET http://192.168.1.100:8080/eventsource net::ERR_INCOMPLETE_CHUNKED_ENCODING 200 (OK)
Any thoughts?
MPC
You're most likely hitting the idle timeout.
You can change it with the following snippet in the cowboy example:
{ok, _} = cowboy:start_clear(http, [{port, 8080}], #{
idle_timeout => 15000,
env => #{dispatch => Dispatch}
}),
You have more information about ProtocolOpts in the cowboy doc
I am trying to establish the max number of retries from my app to rabbit broker.
I have the retry interceptor,
#Bean
public RetryOperationsInterceptor retryOperationsInterceptor() {
return RetryInterceptorBuilder.stateless()
.maxAttempts(CommonConstants.MAX_AMQP_RETRIES)
.backOffOptions(500, 2.0, 3000)
.build();
}
and this is used inside listener container,
container.setAdviceChain(new Advice[]{retryOperationsInterceptor()});
However, after a couple of retries, the consumer attempts connection all over again in an endless loop,
2017-02-21 15:03:12.229 WARN 9292 --- [nsumerThread_92] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused: connect
2017-02-21 15:03:12.229 INFO 9292 --- [nsumerThread_92] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer: tags=[{}], channel=null, acknowledgeMode=AUTO local queue size=0
2017-02-21 15:03:13.245 WARN 9292 --- [nsumerThread_93] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused: connect
2017-02-21 15:03:13.245 INFO 9292 --- [nsumerThread_93] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer: tags=[{}], channel=null, acknowledgeMode=AUTO local queue size=0
2017-02-21 15:03:13.261 ERROR 9292 --- [nsumerThread_83] o.s.a.r.l.SimpleMessageListenerContainer : Failed to check/redeclare auto-delete queue(s).
I want the app to fail and error out because of lack of connectivity to the broker after a MAX_RETRY # limit.
thanks for the help
EDIT
As suggested by #artem-bilan, I ended up using a Component
public class BrokerFailureEventListener implements ApplicationListener<ListenerContainerConsumerFailedEvent>
In this class the onApplicationEvent I counted the number of failures and then take appropriate action.
In case of producer-side, it's a little different scenario. As explained by #artem-bilan, the application would need to take care of any issues. I explored using netflix-hystrix and added a fallback method for the production method and will go with that route. thanks much again.
Well, you misunderstood a bit container.setAdviceChain(new Advice[]{retryOperationsInterceptor()});. It is for the business errors during messages processing:
Business exception handling, as opposed to protocol errors and dropped connections, might need more thought and some custom configuration, especially if transactions and/or container acks are in use. Prior to 2.8.x, RabbitMQ had no definition of dead letter behaviour, so by default a message that is rejected or rolled back because of a business exception can be redelivered ad infinitum. To put a limit in the client on the number of re-deliveries, one choice is a StatefulRetryOperationsInterceptor in the advice chain of the listener. The interceptor can have a recovery callback that implements a custom dead letter action: whatever is appropriate for your particular environment.
In contradiction to the:
In fact it loops endlessly trying to restart the consumer, and only if the consumer is very badly behaved indeed will it give up. One side effect is that if the broker is down when the container starts, it will just keep trying until a connection can be established.
What you need is ListenerContainerConsumerFailedEvent, which is emitted as:
private void logConsumerException(Throwable t) {
if (logger.isDebugEnabled()
|| !(t instanceof AmqpConnectException || t instanceof ConsumerCancelledException)) {
logger.warn(
"Consumer raised exception, processing can restart if the connection factory supports it",
t);
}
else {
logger.warn("Consumer raised exception, processing can restart if the connection factory supports it. "
+ "Exception summary: " + t);
}
publishConsumerFailedEvent("Consumer raised exception, attempting restart", false, t);
}
So, you can listen for those events and stop your application when some condition is reached.
I have a Rails Application with websocket-rails gem.
Inside my application there is a Daemon that I launch with rails runner MyDaemon.start
I'm using websocket-rails Synchronization, so my config/initializers/websocket_rails.rb looks like this:
WebsocketRails.setup do |config|
config.log_internal_events = false
config.standalone = false
config.synchronize = true
end
Inside MyDaemon, using synchronization, I can trigger event that will reach both my WebsocketRails::BaseController and my javascript WebSocketRails.
What I'm trying to do is to find a way to bind to events from my MyDaemon.
I've tried to implement a plain WebSocket client using both faye-websocket-ruby and websocket-client-simple, but after banging my head on my keyboard for some time, I figured out that there is some kind of "handshake" process using connection_id from the client_connected message. Basically none of the solutions provided in this other so question works for me.
I need to understand if inside my MyDaemon I can subscribe directly to some WebsocketRails callback, even inside an EventMachine, or how should I implement a Websocket Client in Ruby itself.
My last attempt to have a ruby client can be found in this gist, and this is a sample output:
ruby client.rb ws://localhost:3000/websocket
[:open, {"upgrade"=>"websocket", "connection"=>"Upgrade", "sec-websocket-accept"=>"zNTdGvxFKJeP+1PyGf27T4x2PGo="}]
JSON message is
[["client_connected", {"id"=>nil, "channel"=>nil, "user_id"=>nil, "data"=>{"connection_id"=>"4b7b91001befb160d17b"}, "success"=>nil, "result"=>nil, "token"=>nil, "server_token"=>nil}]]
client id is 4b7b91001befb160d17b
[:message, "[[\"client_connected\",{\"id\":null,\"channel\":null,\"user_id\":null,\"data\":{\"connection_id\":\"4b7b91001befb160d17b\"},\"success\":null,\"result\":null,\"token\":null,\"server_token\":null}]]"]
JSON message is
[["websocket_rails.ping", {"id"=>nil, "channel"=>nil, "user_id"=>nil, "data"=>{}, "success"=>nil, "result"=>nil, "token"=>nil, "server_token"=>nil}]]
Sending ["pong",{}]
[:message, "[[\"websocket_rails.ping\",{\"id\":null,\"channel\":null,\"user_id\":null,\"data\":{},\"success\":null,\"result\":null,\"token\":null,\"server_token\":null}]]"]
[:close, 1006, ""]
While the log of websocket-rails is:
I [2015-06-27 02:08:45.250] [ConnectionManager] Connection opened: #<Connection::2b3dddaf3ec4ed5e3550>
I [2015-06-27 02:08:45.251] [Dispatcher] Started Event: client_connected
I [2015-06-27 02:08:45.251] [Dispatcher] Name: client_connected
I [2015-06-27 02:08:45.251] [Dispatcher] Data: {"connection_id"=>"2b3dddaf3ec4ed5e3550"}
I [2015-06-27 02:08:45.251] [Dispatcher] Connection: #<Connection::2b3dddaf3ec4ed5e3550>
I [2015-06-27 02:08:45.251] [Dispatcher] Event client_connected Finished in 0.000174623 seconds
I [2015-06-27 02:09:05.252] [ConnectionManager] Connection closed: #<Connection::2b3dddaf3ec4ed5e3550>
I [2015-06-27 02:09:05.252] [Dispatcher] Started Event: client_disconnected
I [2015-06-27 02:09:05.252] [Dispatcher] Name: client_disconnected
I [2015-06-27 02:09:05.252] [Dispatcher] Connection: #<Connection::2b3dddaf3ec4ed5e3550>
I [2015-06-27 02:09:05.253] [Dispatcher] Event client_disconnected Finished in 0.000236669 seconds
Probably I'm missing somethig very stupid, so I'm here to ask your help!
You can use Iodine as a websocket client (I'm the author):
require 'iodine/http'
# prevents the Iodine's server from running
Iodine.protocol = :timer
# starts Iodine while the script is still running
Iodine.force_start!
options = {}
options[:on_open] = Proc.new {puts 'Connection Open'; write "Hello World!" }
options[:on_close] = Proc.new {puts 'Connection Closed'}
options[:on_message] = Proc.new {|data| puts "I got: #{data}" }
# connect to an echo server for demo. Use the blocking method:
websocket = Iodine::Http::WebsocketClient.connect "wss://echo.websocket.org/", options
websocket << "sending data"
sleep 0.5
websocket.close
As an aside note, reading around I noticed that the websocket-rails gem isn't being updated all that much. See this question
As an alternative, you can run websockets inside your Rails app by using the Plezi framework (I'm the author).
It's quite easy to use both frameworks at the same time on the same server. This way you can use your Rails model's code inside your Plezi Websocket controller.
Because Plezi will manage the websockets and Rails will probably render the 404 Not Found page, Plezi's routes will take precedence... but as long as your routes don't override each other, you're golden.
Notice that to allow both apps to run together, Plezi will force you to use Iodine server as your Rack server. To avoid this you can use the Placebo API and run Plezi on a different process.
You can read more the framework's README file.
I have a custom device with a TCP/IP stack implemented that's sending a byte each 5 seconds to a remote IP.
On that remote IP, I'm building a site with rails 3.1.3 that will have to receive, store and display the data sent by the custom device.
I was thinking on having a TCP Socket running in the background, something like this, but i don't have a clue on how to integrate this with a rails site. Where to place it, how to start it and how to propagate the data to the views.
Does anybody have a clue on how shall I proceed?
To solve this I created a raketask that starts a TCP Server that will handle messages.
Note: This code has more than a year so I'm not 100% sure how it behaves, but I think the core part is this:
#server = TCPServer.new(#host, port)
loop do
Thread.start(#server.accept) do |tcpSocket|
port, ip = Socket.unpack_sockaddr_in(tcpSocket.getpeername)
begin
loop do
line = tcpSocket.recv(100).strip # Read lines from the socket
handle_message line # method to handle messages from the socket
end
rescue SystemCallError
#close the sockets logic
end
end
end
I'm seriously struggling to solve this one, any help would be appreciated!
I have two Rails apps, let's call them Client and Service, all very simple, normal REST interface - here's the basic scenario:
Client makes a POST /resources.json request to the Service
The Service runs a process which creates the resource and returns an ID to the Client
Again, all very simple, just that Service processing is very time-intensive and can take several minutes. If that happens, an EOFError is raised on the Client, exactly 60s after the request was made (no matter what the ActiveResource::Base.timeout is set to) while the service correctly processed the request and responds with 200/201. This is what we see in the logs (chronologically):
C 00:00:00: POST /resources.json
S 00:00:00: Received POST /resources.json => resources#create
C 00:01:00: EOFError: end of file reached
/usr/ruby1.8.7/lib/ruby/1.8/net/protocol.rb:135:in `sysread'
/usr/ruby1.8.7/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill'
/usr/ruby1.8.7/lib/ruby/1.8/timeout.rb:62:in `timeout'
...
S 00:02:23: Response POST /resources.json, 201, after 143s
Obviously the service response never reached the client. I traced the error down to the socket level and recreated the scenario in a script, where I open a TCPSocket and try to retrieve data. Since I don't request anything, I shouldn't get anything back and my request should time out after 70 seconds (see full script at the bottom):
Timeout::timeout(70) { TCPSocket.open(domain, 80).sysread(16384) }
These were the results for a few domain:
www.amazon.com => Timeout after 70s
github.com => EOFError after 60s
www.nytimes.com => Timeout after 70s
www.mozilla.org => EOFError after 13s
www.googlelabs.com => Timeout after 70s
maps.google.com => Timeout after 70s
As you can see, some servers allowed us to "wait" for the full 70 seconds, while others terminated our connection, raising EOFErrors. When we did this test against our service, we (expectedly) got an EOFError after 60 seconds.
Does anyone know why this happens? Is there any way to prevent these or extend the server-side time-out? Since our service continues "working", even after the socket was closed, I assume it must be terminated on the proxy-level?
Every hint would be greatly appreciated!
PS: The full script:
require 'socket'
require 'benchmark'
require 'timeout'
def test_socket(domain)
puts "Connecting to #{domain}"
message = nil
time = Benchmark.realtime do
begin
Timeout::timeout(70) { TCPSocket.open(domain, 80).sysread(16384) }
message = "Successfully received data" # Should never happen
rescue => e
message = "Server terminated connection: #{e.class} #{e.message}"
rescue Timeout::Error
message = "Controlled client-side timeout"
end
end
puts " #{message} after #{time.round}s"
end
test_socket 'www.amazon.com'
test_socket 'github.com'
test_socket 'www.nytimes.com'
test_socket 'www.mozilla.org'
test_socket 'www.googlelabs.com'
test_socket 'maps.google.com'
I know this is nearly a year old, but in case anyone else finds this, I wanted to add a possible culprit.
Amazon's ELB will terminate idle connections at 60 seconds, so if you are using EC2 behind ELB, then ELB could be the server side problem.
the only "documentation" I could find here is https://forums.aws.amazon.com/thread.jspa?threadID=33427&start=50&tstart=50, but it's better than nothing
Each server decides when to close the connection. It depends on the server side software and its settings. You can't control that.