I just started using haproxy (1.4) for loadbalance. Through reading the Configuration Manual, i could using cookie insert to insert a cookie in the server responses, but this requires the browser to enable cookies. Besides after googled hours , i could not find anything about the Arguments details of cookie rewrite. So i tried liked this in the haproxy.cfg
backend app
balance roundrobin
cookie SERVERID rewrite
mode http
server web1 192.168.1.133:80 cookie web1 maxconn 32 check inter 5000 rise 2 fall 5 weight 1
server web2 192.168.1.134:80 cookie web2 maxconn 32 check inter 5000 rise 2 fall 5 weight 2
but i can't find anything in the firebug related cookies.
So i wondered if anyone can give something about cookie-based persistence in a backend, no matter whether the browser enabled cookies or not, without using anything tools like memcached , etc, just configuring the configuration file.
You can look at doing something with a "stick table" https://code.google.com/p/haproxy-docs/wiki/stick_table
You will have to make sure that your stick table is synced to all load balancers or make sure that the end user traffic lands on the same load balancer every time. I know that later versions of haproxy (1.5.17+) support syncing the stick table, I do not know about earlier versions.
Related
We have a multiservice architecture consisting of HAProxy front end ( we can change this to another proxy if required), a mongodb database, and multiple instances of a backend app running under Docker Swarm.
Once an initial request is routed to an instance ( container ) of the backend app we would like all future requests from mobile clients to be routed to the same instance. The backend app uses TCP sockets to communicate with a VoIP PBX.
Ideally we would like to control the number of instances of the backend app using the replicas key in the docker-compose file. However if a container died and was recreated we would require mobile clients continue routing to the same container. The reason for this is each container is holding state info.
Is this possible with Docker swarm? We are thinking each instance of the backend app when created gets an identifier which is then used to do some sort of path based routing.
HAproxy has what you need. This article explains all.
As a conclusion of the article, you may choose from two solutions:
IP source affinity to server and Application layer persistence. The latter solution is stronger/better than the first but it requires cookies.
Here is an extras from the article:
IP source affinity to server
An easy way to maintain affinity between a user and a server is to use user’s IP address: this is called Source IP affinity.
There are a lot of issues doing that and I’m not going to detail them right now (TODO++: an other article to write).
The only thing you have to know is that source IP affinity is the latest method to use when you want to “stick” a user to a server.
Well, it’s true that it will solve our issue as long as the user use a single IP address or he never change his IP address during the session.
Application layer persistence
Since a web application server has to identify each users individually, to avoid serving content from a user to an other one, we may use this information, or at least try to reproduce the same behavior in the load-balancer to maintain persistence between a user and a server.
The information we’ll use is the Session Cookie, either set by the load-balancer itself or using one set up by the application server.
What is the difference between Persistence and Affinity
Affinity: this is when we use an information from a layer below the application layer to maintain a client request to a single server
Persistence: this is when we use Application layer information to stick a client to a single server
sticky session: a sticky session is a session maintained by persistence
The main advantage of the persistence over affinity is that it’s much more accurate, but sometimes, Persistence is not doable, so we must rely on affinity.
Using persistence, we mean that we’re 100% sure that a user will get redirected to a single server.
Using affinity, we mean that the user may be redirected to the same server…
Affinity configuration in HAProxy / Aloha load-balancer
The configuration below shows how to do affinity within HAProxy, based on client IP information:
frontend ft_web
bind 0.0.0.0:80
default_backend bk_web
backend bk_web
balance source
hash-type consistent # optional
server s1 192.168.10.11:80 check
server s2 192.168.10.21:80 check
Session cookie setup by the Load-Balancer
The configuration below shows how to configure HAProxy / Aloha load balancer to inject a cookie in the client browser:
frontend ft_web
bind 0.0.0.0:80
default_backend bk_web
backend bk_web
balance roundrobin
cookie SERVERID insert indirect nocache
server s1 192.168.10.11:80 check cookie s1
server s2 192.168.10.21:80 check cookie s2
Is it possible to increase CloudFlare's time-out? If yes, how?
My code takes a while to execute and I wasn't planning on Ajaxifying it the coming days.
No, CloudFlare only offers that kind of customisation on Enterprise plans.
CloudFlare will time out if it fails to establish a HTTP handshake after 15 seconds.
CloudFlare will also wait 100 seconds for a HTTP response from your server before you will see a 524 timeout error.
Other than this there can be timeouts on your origin web server.
It sounds like you need Inter-Process Communication. HTTP should not be used a mechanism for performing blocking tasks without sending responses, these kind of activities should instead be abstracted away to a non-HTTP service on the server. By using RabbitMQ (or any other MQ) you can then pass messages from the HTTP element of your server over to the processing service on your webserver.
I was in communication with Cloudflare about the same issue, and also with the technical support of RabbitMQ.
RabbitMQ suggested using Web Stomp which relies on Web Sockets. However Cloudflare suggested...
Websockets would create a persistent connection through Cloudflare and
there's no timeout as such, but the best way of resolving this would
be just to process the request in the background and respond asynchronously, and serve a 'Loading...' page or similar, rather than having the user to wait for 100 seconds. That would also give a better user experience to the user as well
UPDATE:
For completeness, I will also record here that
I also asked CloudFlare about running the report via a subdomain and "grey-clouding" it and they replied as follows:
I will suggest to verify on why it takes more than 100 seconds for the
reports. Disabling Cloudflare on the sub-domain, allow attackers to
know about your origin IP and attackers will be attacking directly
bypassing Cloudflare.
FURTHER UPDATE
I finally solved this problem by running the report using a thread and using AJAX to "poll" whether the report had been created. See Bypassing CloudFlare's time-out of 100 seconds
Cloudflare doesn't trigger 504 errors on timeout
504 is a timeout triggered by your server - nothing to do with Cloudflare.
524 is a timeout triggered by Cloudflare.
See: https://support.cloudflare.com/hc/en-us/articles/115003011431-Troubleshooting-Cloudflare-5XX-errors#502504error
524 error? There is a workaround:
As #mjsa mentioned, Cloudflare only offers timeout settings to Enterprise clients, which is not an option for most people.
However, you can disable Cloudflare proxing for that specific (sub)domain by turning the orange cloud into grey:
Before:
After:
Note: it will disable extra functionalities for that specific (sub)domain, including IP masking and SSL certificates.
As Cloudflare state in their documentation:
If you regularly run HTTP requests that take over 100 seconds to
complete (for example large data exports), consider moving those
long-running processes to a subdomain that is not proxied by
Cloudflare. That subdomain would have the orange cloud icon toggled to
grey in the Cloudflare DNS Settings . Note that you cannot use a Page
Rule to circumvent Error 524.
I know that it cannot be treated like a solution but there is a 2 ways of avoiding this.
1) Since this timeout is often related to long time generating of something, this type of works can be done through crontab or if You have access to SSH you can run a PHP command directly to execute. In this case connection is not served through Cloudflare so it goes as long as your configuration allows it to run. Check it on Google how to run scripts from command line or how to determine them in crontab by using /usr/bin/php /direct/path/to/file.php
2) You can create subdomain that is not added to cloudlflare and move Your script there and run them directly through URL, Ajax call or whatever.
There is a good answer on Cloudflare community forums about this:
If you need to have scripts that run for longer than around 100 seconds without returning any data to the browser, you can’t run these through Cloudflare. There are a couple of options: Run the scripts via a grey-clouded subdomain or change the script so that it kicks off a long-running background process and quickly returns a status which the browser can poll until the background process has completed, at which point the full response can be returned. This is the way most people do this type of action as keeping HTTP connections open for a long time is unreliable and can be very taxing also.
This topic on Stackoverflow is high in SERPs so I decided to write down this answer for those who will find it usefull.
https://support.cloudflare.com/hc/en-us/articles/115003011431-Troubleshooting-Cloudflare-5XX-errors#502504error
Cloudflare 524 error results from a web page taking more than 100 seconds to completely respond.
This can be overridden to (up to) 600 seconds ... if you change to "Enterprise" Cloudflare account. The cost of Enterprise is roughtly $40k per year (annual contract required).
If you are getting your results with curl, you could use the resolve option to directly access your IP, not using the Cloudflare proxy IP:
For example:
curl --max-time 120 -s -k --resolve lifeboat.com:443:127.0.0.1 -L https://lifeboat.com/blog/feed
The simplest way to do this is to increase your proxy waiting timeout.
If you are using Nginx for instance you can simply add this line in your /etc/nginx/sites-availables/your_domain:
location / {
...
proxy_read_timeout 600s; # this increases it by 10mins; feel free to change as you see fit with your needs.
...
}
If the issue persists, make sure you use let's encrypt to secure your server alongside Nginx and then disable the orange cloud on that specific subdomain on Cloudflare.
Here are some resources you can check to help do that
installing-nginx-on-ubuntu-server
secure-nginx-with-let's-encrypt
I have a website that is 100% https and will only work as https. My site is an asp.net mvc application running on IIS 7.5.
It is on multiple servers with traffic distributed via a load balancer.
I am not in control of the hardware.
For http requests, I was hoping that it could be stopped at the load balancer and a redirect to https at this point.
However the hardware company wont do this for me, and instead I need to do the redirect from http to https within IIS on the server. Therefore unencrypted traffice can enter the inner network with redirect being at the server level. I would feel more comfortable with such a transfer happening at the load balancer.
Do I have valid concerns?
Threat model:
HTTP request:
Attacker
| Security Boundary
V V
Client -- http request --> Load Balancer
|
Client <-- redirect -----------+
Threats which occur from allowing HTTP redirects regardless of methodology:
Spoofing: Client could connect to MITM spoofed HTTP server which does not pass through the redirect, but instead proxies connections to the actual HTTPS server
Tampering: Client could receive redirect URL from MITM spoofed server which directs them to another action (E.g.: client receives redirect to https://yoursite.com/login.aspx?redirect=/deleteAllDocuments)
Information Disclosure: Initial HTTP request is disclosed, any information in POST or GET is available unencrypted to eavesdroppers.
Arguments for performing redirect on server other than target server:
Firewall can be limited to HTTPS data, limiting risk of unencrypted data due to misconfiguration
Configuration and liability could become "Someone else's problem", from at least a political perspective
Vulnerabilities in HTTP server would be isolated and could not be used to attack HTTPS server or underlying application
Arguments for performing redirect on something other than the load balancer:
Load balancers are not servers
Load balancers are not servers, and therefore might have lightly used code paths when used as servers, which could be more prone to undiscovered bugs or performance problems
Configuration is not available to you, but you (or your company) is still probably liable for any misconfigurations which occur (from a legal perspective)
In light of the above analysis, for highest security with lowest risk:
I would not put the redirect on the target server, nor the load balancer, but instead on a VM which only serves to redirect pages. A minimal Linux or windows box should be able to be tightly locked down to limit exposure.
I would not allow redirects with a query-string or POST data (E.g.: show 404 for any non GET / HTTP/1.1 request)
I would call the possibility of spoofing and tampering an acceptable risk, or show a page to the user explaining that the site must be accessed using HTTPS instead of using a redirect
But, if you can assume the following conditions are met, placing the HTTP server co-resident with the HTTPS server should not reduce security.
Any bug in the HTTP server is present in the HTTPS server
The HTTP server is correctly configured to disallow access to protected resources (set up as a separate site in IIS, for example. Secure site still has no HTTP binding)
No other application is able to create a server on HTTP (netsh urlacl only has IIS, for example)
Configuration is audited to ensure the above configurations are properly maintained (Periodic pen tests, manual configuration review, configuration change management, and an IDS or IPS system)
In some cases, the reduced complexity may even be easier to secure than a separate server. Additionally, if the administrator is unfamiliar with the load balancer's configuration, they may be more prone to make a critical error in configuration than if they were to make the same configuration in a product they know well.
Do I have valid concerns?
Do you have a valid concern about the initial connection being over HTTP? Sure. The initial request can be intercepted and the response spoofed in a MitM attack. The attacker can then either prevent the user from using HTTP (adding ssl/tls between the attacker and your server and relaying to the victim in the clear) or can create an imposter SSL session with the client that terminates at the attacker before being re-encrypted on it's way to you (using various spoofing techniques to make the attack less obvious to the casual user).
However, if such an attack were launched, I would be far more worried about the transit from the client to your load balancer, not between your load balancer and IIS. If you suspect that you have malicious systems behind your load balancer, you have an entirely different set of problems.
See my answer over on security.so for some relevant information regarding redirects from HTTP to HTTPS.
Struts 2 support stop double-submission of forms by generate a unique random token and store it in the session, and use token tag pass the token to the client form then verify the tokens from session and form.
As far as i know, this solution can only work in a single JVM because the session is separated from each other. I cannot find something useful about how to use this solution in a distributed Java environment. We use Nginx proxy HTTP requests to multiple JVMs and Nginx does not guarantee to proxy the same request to the same JVM every time.
Can someone give me some help?
BTW, i am trying to use this solution to stop CSRF attack.
You have two choices (neither of which really has anything to do with Struts 2, but has everything to do with session management in a distributed environment):
Use Session Affinity - so when a user creates a session, Nginx remembers which backend server the user went to, and that session is bound to that server for all subsequent requests. (This is the more typical solution). might get you started.
Depending on your application server, there may be the possibility of sharing the session data between servers. For example, in Tomcat 6, the configuration directions are.
I've been catching up with the Scaling Rails screencasts. In episode 11 which covers advanced HTTP caching (using reverse proxy caches such as Varnish and Squid etc.), they recommend only considering using a reverse proxy cache once you've already exhausted the possibilities of page, action and fragment caching within your Rails application (as well as memcached etc. but that's not relevant to this question).
What I can't quite understand is how using an HTTP reverse proxy cache can provide a performance boost for an application that already uses page caching. To simplify matters, let's assume that I'm talking about a single host here.
This is my understanding of how both techniques work (maybe I'm wrong):
With page caching the Rails process is hit initially and then generates a static HTML file that is served directly by the Web server for subsequent requests, for as long as the cache for that request is valid. If the cache has expired then Rails is hit again and the static file is regenerated with the updated content ready for the next request
With an HTTP reverse proxy cache the Rails process is hit when the proxy needs to determine whether the content is stale or not. This is done using various HTTP headers such as ETag, Last-Modified etc. If the content is fresh then Rails responds to the proxy with an HTTP 304 Not Modified and the proxy serves its cached content to the browser, or even better, responds with its own HTTP 304. If the content is stale then Rails serves the updated content to the proxy which caches it and then serves it to the browser
If my understanding is correct, then doesn't page caching result in less hits to the Rails process? There isn't all that back and forth to determine if the content is stale, meaning better performance than reverse proxy caching. Why might you use both techniques in conjunction?
You are right.
The only reason to consider it is if your apache sets expires headers. In this configuration, the proxy can take some of the load off apache.
Having said this, apache static vs proxy cache is pretty much an irrelevancy in the rails world. They are both astronomically fast.
The benefits you would get would be for your none page cacheable stuff.
I prefer using proxy caching over page caching (ala heroku), but thats just me, and a digression.
A good proxy cache implementation (e.g., Squid, Traffic Server) is massively more scalable than Apache when using the prefork MPM. If you're using the worker MPM, Apache is OK, but a proxy will still be much more scalable at high loads (tens of thousands of requests / second).
Varnish for example has a feature when the simultaneous requests to the same URL (which is not in cache) are queued and only single/first request actually hits the back-end. That could prevent some nasty dog-pile cases which are nearly impossible to workaround in traditional page caching scenario.
Using a reverse proxy in a setup with only one app server seems a bit overkill IMO.
In a configuration with more than one app server, a reverse proxy (e.g. varnish, etc.) is the most effective way for page caching.
Think of a setup with 2 app servers:
User 'Bob'(redirected to node 'A') posts a new message, the page gets expired and recreated on node 'A'.
User 'Cindy' (redirected to node 'B') requests the page where the new message from 'Bob' should appear, but she can't see the new message, because the page on node 'B' wasn't expired and recreated.
This concurrency problem could be solved with a reverse proxy.