The logs don't show anything different, and the computer is four times faster than the last one. Anyone know any common reasons why making a request to localhost would take a very long time?
I am using Mongrel.
Hard to give a solution based on the little information you give, so try to narrow it down. I would say that these three causes seem the most likely:
the database is slow. You can check this if your queries take a long time (check the logs). Perhaps you are using a slow connector (i.e. the default Ruby MySQL library), or your indexes haven't made it to your new machine.
Mongrel is slow. Check by starting it with Webrick and see if that's any better
your computer is slow. Perhaps it's running something else that's taking up CPU or memory. See your performance monitor (application to use for this differs per OS).
Could be a conflict between IPv4 and IPv6. If you're running Apache you have to take special steps to make it work nicely with IPv6 (my information here might be out of date.) I've found that an IPv6-enabled client would try to talk IPv6 to the server, and Apache would not receive the request. After it timed out the client would retry on IPv4.
Related
Does iOS store DNS and if so, how do I reset it after a server transfer? I have an App that is released, and before I migrate by server to a different IP, I want to make sure I know if the people using my App will have any issues with the API calls in the App. Any knowledge would be appreciated. Thank You for your time!
I'm a DNS expert, but not so much for iOS. So this answer is how the DNS is intended to work. I can't say whether or not iOS got this right (but they have a better reputation for rightness than some others I won't mention).
In the DNS each RR (that's Resource Record, the basic unit in the DNS, your address [assuming you only have one] is in a single RR) has a TTL, which specifies how long it will be good for. In normal operation you set these to a fairly long time (at least a few hours, often several days). When things aren't changing, this says how long the DNS client should hold the data before querying again. When you're contemplating a change, you set the TTL down low (say a few minutes) in advance, and then you can make the change at the right time and since no hosts should hold on to the old data for longer than that lowered TTL, you know that everyone should have the new data within that time. The reason you use higher TTLs normally is to lower the demands on both your server and your (and everyone who connects to you) Internet connection.
I have a problem that membase is being very slow on my environment.
I am running several production servers (Passenger) on rails 2.3.10 ruby 1.8.7.
Those servers communicate with 2 membase machines in a cluster.
the membase machines each have 64G of memory and a100g EBS attached to them, 1G swap.
My problem is that membase is being VERY slow in response time and is actually the slowest part right now in all of the application lifecycle.
my question is: Why?
the rails gem I am using is memcache-northscale.
the membase server is 1.7.1 (latest).
The server is doing between 2K-7K ops per second (for the cluster)
The response time from membase (based on NewRelic) is 250ms in average which is HUGE and unreasonable.
Does anybody know why is this happening?
What can I do inorder to improve this time?
It's hard to immediately say with the data at hand, but I think I have a few things you may wish to dig into to narrow down where the issue may be.
First of all, do your stats with membase show a significant number of background fetches? This is in the Web UI statistics for "disk reads per second". If so, that's the likely culprit for the higher latencies.
You can read more about the statistics and sizing in the manual, particularly the sections on statistics and cluster design considerations.
Second, you're reporting 250ms on average. Is this a sliding average, or overall? Do you have something like max 90th or max 99th latencies? Some outlying disk fetches can give you a large average, when most requests (for example, those from RAM that don't need disk fetches) are actually quite speedy.
Are your systems spread throughout availability zones? What kind of instances are you using? Are the clients and servers in the same Amazon AWS region? I suspect the answer may be "yes" to the first, which means about 1.5ms overhead when using xlarge instances from recent measurements. This can matter if you're doing a lot of fetches synchronously and in serial in a given method.
I expect it's all in one region, but it's worth double checking since those latencies sound like WAN latencies.
Finally, there is an updated Ruby gem, backwards compatible with Fauna. Couchbase, Inc. has been working to add back to Fauna upstream. If possible, you may want to try the gem referenced here:
http://www.couchbase.org/code/couchbase/ruby/2.0.0
You will also want to look at running Moxi on the client-side. By accessing Membase, you need to go through a proxy (called Moxi). By default, it's installed on the server which means you might make a request to one of the servers that doesn't actually have the key. Moxi will go get it...but then you're doubling the network traffic.
Installing Moxi on the client-side will eliminate this extra network traffic: http://www.couchbase.org/wiki/display/membase/Moxi
Perry
Our company provides web-based management software (servicedesk, helpdesk, timesheet, etc) for our clients.
One of them have been causing a great headache for some months complaining about the connection speed with our servers.
In our individual tests, the connection and response speeds are always great.
Some information about this specific client :
They have about 300 PC's on their local network, all using the same bandwith/server for internet access.
They dont allow us to ping their server, so we cant establish a trace route.
They claim every other site (google, blogs, news, etc) are always responding fast. We know for a fact they have no intention to mislead us and know this to be true.
They might have up to 100 PC's simulateneously logged in our software at any given time. They have a need to increase that amount up to 300 so this is a major issue.
They are helpfull and colaborative in this issue we are trying to resolve for a long time.
Some information about our server and software :
We have been able to allocate more then 400 users at a single time without major speed losses for other clients.
We have gone extensive lengths to make good use of data caching and opcode caching in the software itself, and we did notice the improvement (from fast to faster)
There are no database, CPU or memory bottlenecks or leaks. Other clients are able to access the server just fine.
We have little to no knowledge on how to do some analyzing on specific end-user problems (Apache running under Windows server), and this is where I could use a lot of help.
Anything that might be related to Apache configuration would also be helpfull.
While all signs points to it being an internal problem in this specific client network, we are dedicating this effort to solve that too, if that is the case, but do not have capable or instructed professionals to deal with network problems (they do, however, while their main argument is that 'all other sites are fast, only yours is slow')
you might want to have a look at the tools from google "page speed family": http://code.google.com/speed/page-speed/docs/overview.html
your customer should maybe run the page speed extension for you. maybe then you can find out what is the problem: http://code.google.com/speed/page-speed/docs/extension.html
How much time would be saved in skipping the request for the IP of a domain name? Would it be noticeable. Before anyone gets to excited i know there could be problems with using an IP as opposed to a URL in web dev like if you have multiple servers, but im just curious.
Assuming they have to use the domain name to get to your site in the first place, their PC will cache the name and address, so anything you do after that won't make much difference. It's not like it does a lookup over the net for every click.
DNS resolution is cached at multiple levels. You would just be saving yourself a single roundtrip per session. Considering the hassles involved (IPs change more often than we'd like, and often not when we have control over them), I would seriously advice against it.
About the only thing it will do for you is insulate you from DNS failure. When your user experiences DNS problems, your site will be one the few that mysteriously keeps working for them!
You can still run multiple servers and load balance them behind a single IP, so that needn't be a worry. Generally though, there are far less pros than cons to this approach.
You could also add the ip and hostname to you local hosts file if you want to be save from DNS failure and slow lookups.
Although I don't think there is much merit in it, since it kind of defeats the purpose of having dns :)
We're running a Rails site at http://hansard.millbanksystems.com, on a dedicated Accelerator. We currently have Apache setup with mod-proxy-balancer, proxying to four mongrels running the application.
Some requests are rather slow and in order to prevent the situation where other requests get queued up behind them, we're considering options for proxying that will direct requests to an idle mongrel if there is one.
Options appear to include:
recompiling mod_proxy_balancer for Apache as described at http://labs.reevoo.com/
compiling nginx with the fair proxy balancer for Solaris
compiling haproxy for Open Solaris (although this may not work well with SMF)
Are these reasonable options? Have we missed anything obvious? We'd be very grateful for your advice.
Apache is a bit of a strange beast to use for your balancing. It's certainly capable but it's like using a tank to do the shopping.
Haproxy/Nginx are more specifically tailored for the job. You should get higher throughput and use fewer resources at the same time.
HAProxy offers a much richer set of features for load-balancing than mod_proxy_balancer, nginx, and pretty much any other software out there.
In particular for your situation, the log output is highly customisable so it should be much easier to identify when, where and why slow requests occur.
Also, there are a few different load distribution algorithms available, with nice automatic failover capabilities too.
37Signals have a post on Rails and HAProxy here (originally seen here).
if you want to avoid Apache, it is possible to deploy a Mongrel cluster with an alternative web server, such as nginx or lighttpd, and a load balancer of some variety such as Pound or a hardware-based solution.
Pounds (http://www.apsis.ch/pound/) worked well for me!
The only issue with haproxy and SMF is that you can't use it's soft-restart feature to implement the 'refresh' action, unless you write a wrapper script. I wrote about that in a bit more detail here
However, IME haproxy has been absolutely bomb-proof on solaris, and I would recommend it highly. We ship anything from a few hundred GB to a couple of TB a day through a single haproxy instance on solaris 10 and so far (touch wood) in 2+ years of operation we've not had any problems with it.
Pound is an HTTP load balancer that I've used successfully in the past. It includes a dynamic scaling feature that may help with your specific problem:
DynScale (0|1): Enable or disable
the dynamic rescaling code (default:
0). If enabled Pound will periodically
try to modify the back-end priorities
in order to equalise the response
times from the various back-ends. This
value can be overridden for specific
services.
Pound is small, well documented, and easy to configure.
I've used mod_proxy_balancer + mongrel_cluster successfully (small traffic website).