Can I launch a bastion host through auto-scaling-group, so that I set "MinSize": 1 and "DesiredCapacity": 1.
I understand that normally ASG is used along with ELB or SQS and Cloudwatch from load balancing or scaling purpose. And I feel my purpose here is different -- I want to make my bastion machine up and running, and once it's down, I want to bring it back as soon as possible. (I don't need my bastion host to be "HA", but I'd like it to be able to automatically recover, say within 3 mins)
Is there such an use case for auto scaling group?
Yes, using an Auto Scaling Group in this fashion will ensure that a failed host will be replaced automatically if it fail EC2 health checks.
However, this is not the best and up to date way to solve your problem. EC2 supports Auto Recovery as of last year. Recovery can be configured to perform a variety of actions on an instance that fails EC2 health checks. The advantage it has over Auto Scaling is that things like Elastic IPs can be migrated over to the new instance. The docs contain all the information you'll need to set this up.
Yes, that's a valid use case.
Auto scaling groups force you to setup automatically creatable instances: you define a launch configuration that specifies stuff like instance type and the image you want to launch, and the number of instances in the group.
When you set the desired instances to '1', the autoscaling group (AG) will start enforcing that one instance will be running.
Problem: the instances get assigned a different IP when they boot so you won't know where to reach it.
There are two ways around this:
- use an ELB so you can always reach it at the ELB's address. When only running one instance, this is kind of an overkill
- make the instance assign an elastic ip when it boots. I don't think that Amazon supports this out-of-the box yet, but you can find scripts that do this for you on the web.
Note that this setup won't prevent failure. But once an instance fails, it's a matter of terminating it and a new one will be backup in 5 minutes or so.
Refer following link from amazon on the architecture and best practice for Bastion host - http://docs.aws.amazon.com/quickstart/latest/linux-bastion/architecture.html
Related
I'm having 5 Docker nodes in a cluster [swarm].
Let's say I'll constraint NGINX [it is not about nginx, it is just an example] to be deployed only on Docker node 1.
Can I create that constraint in such a way that if Docker node 1 goes down the constraint to not be available anymore?
Like, having that constraint only when the node is reacheable, when it isn't, automatically remove the constraint?
Yes, you can use the placement-ref to place a spread stratergy to your node.hostname=your.node1.hostname as document here
https://docs.docker.com/engine/reference/commandline/service_create/#specify-service-placement-preferences---placement-pref.
If the nodes in one category (for example, those
with node.labels.datacenter=south) can’t handle their fair share of
tasks due to constraints or resource limitations, the extra tasks will
be assigned to other nodes instead, if possible.
The downside is that when your node 1 is back online, the service won't be update and rebalance until the service has been updated again (manually or the service is down).
Additionally, it's not a good design if your service has to be placed on a special node but it should be designed to be able to work every where so you can balance server load accross all nodes. Regarding, NGINX, it's stateless and you can deploy it to all of your nodes and let the docker routing mess to do the load balancing. If your service is statefull, even that it's re-deploy to a second node, your data will be not available and your total service is interupted too. So my real answer is that your question is possible but not the expectation of how Docker Swarm is designed and may be not good too.
If you have any good reason to stick with your question solution. You can think about a load balancer in front of your NGINX (or others) like another NGINX or HAProxy which will allow you more control to route your requests to a master node and use secondary or more node for backup purpose only and so on. The design will be that you have a stateless Load Balancer deploy in global mode, and your core service is running behind the LB. This design will give you no downtime when your node 1 is down or service is updating or relocating.
Not exactly sure how to ask this question or if this is a valid approach. So I am learning all about docker, containers, etc. From what I have read it is great for creating individual different microservices that perform various tasks such as BasketService, CartService, etc, which can each be contained in their own docker container on a vm which I think the URL calls from my UI (If hosted on a linux vm) would be something along the lines of https://MyLinuxVM/BasketService/{controller}.
My Question:
Now lets say I have only 1 service. We will call it MyService, that needs to have multiple instances. So I could have 4 instances i.e: MyService1, MyService2, MyService3, MyService4. All exactly the same. From my client, would the following assumption be correct?
I can call https://MyLinuxVM/MyService1/{controller} or https://MyLinuxVM/MyService2/{controller} to send to a specific container instance?
Why:
I feel this may help explain why I am doing this and possibly help everyone understand my problem in the first place. I have 4 physical devices I need to communicate with. We will call them Device1, Device2, Device3, Device4. Each device has its own IP Address, and its own set of "Tools" connected to it on various ports of the device (10-20 ports per device).
From our UI, the users can click a button that sets some torque values for the tool in their hand by sending the data to the MVC backend which gets sent to the "Correct" background worker/container which will then transform the data into byte[] and pass it along to its dedicated device. I am not sure if I need multiple background workers in a single container, or just a single configurable container with a single background worker that gets deployed multiple times dependent on number of devices we have running in the shop.
I have read a lot of things on creating different worker services that do different tasks, but I need multiple instances of a worker service that can be configured (preferably from db tables) to send to a specific device.
Picture for additional details / visual:
I want to move all of my operations over to K8S for so long, but am still hesitant to that. This question will likely be broad, but bear with me. Let me first describe the existing system.
I hosts a lot of different websites (>30). A lot of that for my own experimentation, but some are for actual clients. I have 1 VM in New York (I'm using DigitalOcean), with multiple Docker containers, frequently managed using docker-compose. There is 1 container for every site. The request first comes in to front container running HAProxy. This strips away SSL, then forwards the request to 2 proxy container running Nginx. These 2 container then forwards the request to all the other containers for their service. All of my certificates come from LetsEncrypt, and have to be renewed every 3 months. To do so, I stop front, run certbot --apache so it binds to port 80. It gets the certificates, then I stop apache, then recreate front container.
There are several reasons to why I do it this way:
I change site configs a lot, and how all of them are wired together. So front is expected to run forever, unless I'm getting certificates, and proxys are expected to change a lot. I change the proxy image, then stops and recreates the 1st container, then stops and recreates the 2nd container, so that there will be no downtime at all.
I really don't know how to get certificates when there are multiple nodes. In fact, I'm a total noob at the whole certificate thing and LetsEncrypt is pretty much the only way I know of to do this.
I want to directly edit files on the remote server. I have a bad practice of editing production code directly, mainly because I get impatient with setting up dev, staging and production environments. It takes too much time, and the gains feels small. And for clients, they are typically small businesses, with <10 employees, and regularly, they want to have some aesthetic changes to the websites. I can have a video call with them, they tell me exactly what they want, I code that in, it gets uploaded to the server immediately, and they see changes right away. Then they can critique the design, and we can iterate back and forth. If I were to setup different environments, they can't see it right away, and there has to be this long process of committing to git, deploy to staging, then production. This takes a long time, and I don't think is justified.
I realize that my systems are not that well maintained. Images are not getting security updates, I don't know if they are still running or not unless I check for them manually, which is tedious, so I don't do them at all. Furthermore, I have an Asian background, that means I have clients from both the US and Asia, pretty much the farthest place possible from each other, which increases latency by a lot. That means client in Asia has to wait for around 1-2 second for the page to actually load, which is eternal. I have also moved to Asia in the past week, so now, accessing the New York server via ssh is incredibly slow, and my productivity just plummets. So now it might be the best time to revamp everything, and move to K8S once and forever. However, there are major problems in the planning process and currently, K8S seems to lack a lot of stuff that are just deal breakers for me. So please criticize my plans, and improve them however you see fit.
What I plan to do now is this:
There will be 2 servers, 1 at New York, 1 at Singapore. These 2 severs will have 2 different ip addresses. Those 2 will be running K8S Pods. Preferably, they should have exactly the same configs, website containers, database containers, etc. Then for each website DNS record, I will modify A and AAAA records so that they contain 2 ip addresses for the 2 servers.
My question is:
Will DNS always route to Singapore if user is in China, and always route to New York if user is in England?
How to actually get certificates for 2 nodes? My understanding is that when certbot issues a certificate, it associates the domain name with the node ip address. That means 2 nodes can't have the same certificate for the same domain name. Is this correct? If you can get certificates for 2 nodes then how to do that?
How to keep files in sync between servers? Say I edit the file tree in Singapore server, I want that file to also be modified in New York several seconds later. For databases, I can have a master database at either Singapore or New York, then have slave databases at both locations that updates whenever the master updates, and the slaves can serve as a low latency database for each server.
How to actually route requests from servers to containers inside. I initially plan to use NodePort, to direct the request to front Pods, then that can distribute requests to other Pods, but I was heartbroken when NodePort can't attach to ports below 30000. The only other option that I am aware of is to have an external load balancing service that directs traffic to the 2 servers. But that costs like $15/site/month, and because I have >30 sites, doing so will bankrupt me. I can also have 4 servers in total, 2 for the K8S cluster, and 2 serves as a load balancer that will forward to NodePort. Will this plan works? How will automatic renewing of certificates even work here?
Please note that may be my questions are the wrong questions to ask (like, may be I shouldn't use A and AAAA records for directing traffic), and there's a different way to do this entirely, so feel free to ask the right questions.
read your question hats off to write down the whole stuff but half of the stuff is useless.
Answers of your question :
Can we add the same or multiple entries in DNS? example.com with A record multiple times possible?
You might require to set up a regional K8s cluster with regional ingress support. you can use certmanager with letsencrypt which will manage your cert at LB level and terminate it at the front.
If you are looking forward to use two VMs put one LB in front of both and set SSL over there.
if you are using K8s with stateless PODs editing direct file inside container is not a option. better you manage the Github update inside and container get deployed on to both cluster at a same time for that you can setup CI/CD. You are right in case of database server setup with master slave concept you can use read replicas.
To route the traffic from server to internal application of K8s you can an internal LB or exposing services with node ports(above 30000 but change target port in SVC) and route the port if you want to redirect requests on a specific port using the target port.
still, i am not getting "I can also have 4 servers in total, 2 for the K8S cluster, and 2 serves as a load balancer that will forward to NodePort. Will this plan works? How will automatic renewing of certificates even work here?" which server will be in front and which one in the backend.
If all your services are websites (run over http) you could use k8s ingress to route traffic to pods based on Host header (domain name) and use only one LB with one IP address. The most popular ingress controller seems to be the Nginx Ingress Controller
If you don't want to use LB you can use hostPort to expose nginx ingress but as soon as you have k8s cluster with more than one node, use LB because hostPort is generally not advised to use unless you have a very good reason to do so.
Speaking of DNS, you can use sth like AWS route53 routing policies for location routing. You don't necessarily need to use AWS. I just want to show you that there are solutions to this problem, but use whatever you like.
For certificates use cetrmanager with DNS-01 challenge.
From letsencrypt docs about DNS-01 challenge:
It works well even if you have multiple web servers.
cetrmanager will also handle certificate renewal for you.
About keeping files in sync between servers; It depends on files, but for static content it might be best to use CDN that will replicate content from one source to other locations.
For simultanous deploys to 2 separate clusters you can use some CI/CD pipeline like e.g. github actions.
Can anyone tell me in the course, it is possible to override the parameters of individual box.cfg on a running instance. For example, add a replica, for several days I have been trying to deploy three replicas on three hosts via the docker service stack.
When I raise my hands on each server, everything works, through deploy they do not see each other and fall. I've tried all sorts of ways. hung up the endpoint on the target nodes, when requested, it gives the ip of the machine on which the container rises, if the ip matches one of those indicated in SEED, then substitutes the internal ip of the container instead (otherwise it cannot connect to itself).
In theory, it all works as I described, but there are suspicions that everything is not much different, I suppose that the problem is that before the declaration of box.cfg the instance does not reserve the address. Alas, I can not go inside the container because it cannot rise. I got the idea that if all three nodes are declared at the minimum settings and as soon as they rise to listen to the subnet, as soon as the node finds another, it will write it to replication and override box.cfg. Correct me please who had experience.
Some of the box.cfg parameters are dynamic. For example, the box.cfg{listen=}. You can set this one from the Lua code as you wish. In your case, if the container gets its IP address later, you need to specify only the port in listen. This way, Tarantool will listen on all possible interfaces.
The replication_source is a bit trickier. You can set it dynamically, but your first (initializing) call to box.cfg should be with the replication_source. This is because all instances that are initialized without this parameter will create their own replicaset, and it will make it impossible to join them to another replicaset.
You can read more about Tarantool replication architecture here: https://www.tarantool.io/en/doc/latest/book/replication/repl_architecture/
I'm trying to figure out if Kubernetes will work for a certain use case. I understand the networking/clustering concept, and even the load balancing and how that can be used with things like nginx. However, assuming this is not deployed on a public cloud and things like ELB won't be available, could it still be used for a high-speed networking application using DPDK? For example, if we assume the cluster networking provided by k8s is only used for the control/management path, and the containers themselves handle the NIC directly with DPDK, is this something it's commonly used for?
Secondly, I understand the replication controller and petsets feature I think, but I'm not really clear on whether the intent of those features is for high availability or not. It seems that the "pod fails and the RC replaces it on a different node" isn't necessarily for HA, and there aren't really guarantees on how fast it builds a new pod. Am I incorrect?
For the second question, if the replication controller has size large than 1, it is highly available.
For example, you have an service "web-svc" in front of the replication controller "web-app", with size 3, then your request will be load balanced to one of the 3 pod:
web-svc ----> {web-app-pod1, web-app-pod2, web-app-pod3}
If some of the 3 pods fail, kubernetes will replace them with new ones.
And pet set is similar to replication controller, but used for stateful applications like database.