I'm trying to get down and dirty with DevOps and I'm running into a health check request timed out failure. The problem is my Elastic Load Balancer sends a health check to my EC2 instance and gets a network timeout. I'm not sure what I did wrong. I am following this tutorial and I have completed all the steps up to and including "Using a Elastic Load Balancer". My EC2 instance seems to be working fine and I am able to successfully curl localhost on port 9292 from within the EC2 instance.
EC2 instance security group setup:
Elastic Load Balancer setup:
My target group for the ELB routing has port 9292 open via HTTP and here's a screenshot of the target in my target group that is unhealthy.
Health check config:
I have a VPC that my EC2 instance is a part of and my ELB is connected to the same VPC. I do not have Apache installed and I do not have nginx installed. To my understanding, I do not need these. I have a Rails Puma server running and I can send successful curl requests to the server.
My hunch is that my ELB is not allowed to reach my EC2 instance, resulting in a network timeout and a failed health check. I'm unable to find the cause for this. Any ideas? This SO post didn't help much. Are my security groups misconfigured? What else could potentially block a routing request from ELB to my EC2 instance?
Also, is there a way to view network requests / logs for my EC2 instance? I keep seeing VPC flow logging but I feel like there are simpler alternatives.
Here's something I posted in the AWS forums but to no avail.
UPDATE: I can curl the private IP of target just fine from within an EC2 instance. I don't think it's the target instance, I think it's something to do with the security group setup. I am unable to identify why though because I have basically allowed all traffic from the Load Balancer to the EC2 instance.
I made my mistake during the "Setup your VPC" step. I finished creating a subnet for an RDS instance. I proceeded to start an instance and the default subnet that AWS chose when I switched to my VPC was the subnet I made for my RDS, which was NOT a public subnet. Therefore, any attempts, from any EC2 instance or my load balancer, would not be able to reach it because I had only set up my public subnet to take requests.
The solution was to create a new instance and this time, pick the correct public subnet. My original EC2 instance was associated with a private subnet while the load balancer was pointing to the public subnet.
Here's a link to a hand drawn image that helped me pin point my problem, hopefully can help anyone else who's having trouble setting up. I didn't put image here directly because it's bigger than 2MB.
Glad to answer any further questions too!
Related
I am working backend server launched on ECS cluster, hosted on an EC2 instance using docker.
the ECS is running great, exposed by IP address and port, but to be used with my ios app it needs to be served over https.
How do serve my ECS container over https? I have read a couple of things regarding using a load balancer, but tutorials are outdated and I can't find one that shows configuration after the ecs cluster has already been created.
Please point me to the right direction so I can get it served over https.
You need to have the following resources:
DNS address
Valid SSL Certificate
Load Balancer
Load balancer security group
Target Group
The target group will mediate between your server and your load balancer.
Also, in the security group define all the rules you currently have in the server security group, and in the server's security group ad a rule that open is open to all traffic in all ports with the security group instead of id.
This guide can help you:https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-create-https-ssl-load-balancer.html
(look at Create an HTTPS/SSL load balancer using the console)
I am trying to get the Bitnami Neo4j image running on OpenShift (testing on my local Minishift), but I am unable to connect. I am following the steps outlined in this issue (now closed), however, now I cannot access the external IP for the load balancer.
Here are the steps I have taken:
Deploy Image (bitnami/neo4j)
Create service for the load balancer,
using the YAML supplied in the issue mentioned
Get the external IP
address for the LB (oc get services) The command in step 3 lists 2
of the same IP addresses, and when I attempt to go to this IP in my
browser it times out.
I can create a route that points to port 7374 on the IP of the LB, but
then I get the same error as reported in the aforementioned issue.
(ServiceUnavailable: WebSocket connection failure. Due to security
constraints in your web browser, the reason for the failure is not
available to this Neo4j Driver. Please use your browsers development
console to determine the root cause of the failure. Common)
Configure neo4j to accept non-local connections. E.g.:
dbms.connector.bolt.address=0.0.0.0:7687
Source: https://neo4j.com/developer/kb/explanation-of-error-websocket-connection-failure/
I've been using docker swarm for a while and I'm really pleased with how simple it is to set up a swarm cluster and to run replicated services. However I've faced a problem that seems like a blocker in my use case.
I'm using docker 1.12 and swarm mode.
My problem is that the internal IPVS load balancer sends request to tasks that have "status health: starting" and whereas my application is not properly started.
My application takes some time to start but docker swarm load balancer starts sending requests as soon as the container is in "state running".
After running some tests I realized that If I scale up one instance, the instance is available to the load balancer immediately and the client may get a connection refused response if the load balancer sends the request to the starting server.
I've implemented the health check and I was expecting a particular instance to only become available to the load balancer after the first successful health check.
Is there any way to configure the load balancer or the scheduler to only send request to instance that are properly started?
Best Regards,
Bruno Vale
I have rails running on an Amazon EC2 instance but I'm having trouble accessing my rails server (port 3000) from public IPs. This was working as recently as last week but today when I started up my rails server I was unable to get a response from an outside IP. Even when I created a brand new rails project I was not able to access the server once I had started it.
Despite this I can access the Apache server by going through port 80. Moreover SSL continues to work.
I have already
ensured that the port 3000 is open on my security group in the Amazon EC2 console
checked my iptables on the Amazon instance. There are no rules and the default for the INPUT chain is to accept packets.
banged my head against a wall.
Does anyone have any ideas?
Did you verify the port 3000 is opened for every IPs addresses and not only for your workplace's IP?
Also, your Amazon EC2 virtual machine's IP could have changed.
I have a rails 3.2 app running on Heroku which needs to proxy requests to an external API from a static IP address. Since Heroku doesn't offer elastic IPs, and Proximo is too expensive and limiting for the number of requests I need to make, I set up a simple forward proxy on an AWS EC2 micro instance in US-East using mod proxy.
I can proxy requests from my app's local environment just fine. However, requests from heroku time out. My thinking is that, since I can proxy from my local environment, the point of failure must be the connection between Heroku and my Proxy box. I've tried the answer given here: Security settings between ec2 and heroku but it didn't work. I've even tried allowing all inbound access on port 80 (even though that's terrible for the internet).
So, my question is, what are the security settings that I should enable for my ec2 instance in order to allow Heroku to proxy through it?
Heroku dynos are all running on machines within Amazon EC2 us-east-1 data center. They do not have any restrictions/firewalls on outgoing connections.
As long as you have the proper Security Group settings to allow the connections from your dynos to your own EC2 instance, you should be good.
It sounds like you haven't correctly opened up access from within us-east-1 to your instance. Double check your security group.
Information on how to edit the correct security group:
Check what security group you are using for your instance. See value
of Security Groups column in row of your instance. It's important -
I changed rules for default group, but my instance was under
quickstart-1 group when I had similar issue.
Go to Security Groups tab, go to Inbound tab, select HTTP in Create
a new rule combo-box, leave 0.0.0.0/0 in source field and click Add
Rule, then Apply rule changes.