I have a NodeJS app hosted on Cloud Run.
I have set that just 1 and only 1 instance of the service should be running at any given point in time.
However, whenever I make a code change and deploys the new revision, it turns out that the previous revision is still running until after a while then it stops.
How can I make sure even though I am deploying new code changes, multiple instances should never run. The existing running instance should stop immediately I am about to deploy new changes.
Multiple instances is causing duplicate items to be produced in my code and business logic.
Thank you.
Make sure that
Minimum number of instances:0
Maximum number of instances: 1
'Serve this revision immediately' checkbox is selected.
Based on that, 100% of the traffic will be migrated to the revision, overriding all existing traffic splits, if any.
Related
I want to only ever run one instance of my container to run at a given time. For instance say my ECS container is writing data to an EFS. The application I am running is built in such a way that multiple instances can not be writing to this data at a given time. So is there a way to make sure that ECS never starts more than one instance. I was worried that when one container was being torn down or stood up that two containers may end up running simultaneously.
I was also thinking about falling back to EC2 so that I could meet this requirement but did want to try this out in ECS.
I have tried setting the desired instances to 1 but I am worried that this will not work.
Just set min, desired and maximum number of tasks to 1.
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-configure-auto-scaling.html#:~:text=To%20configure%20basic%20Service%20Auto%20Scaling%20parameters
I’m working with a ksqldb server deployed in Kubernetes, and since some time ago it crashed for some reason, I want to implement High Availability as described in https://docs.ksqldb.io/en/latest/operate-and-deploy/high-availability/
We are deploying the server with docker, so the properties that we put inside the config file are:
KSQL_KSQL_STREAMS_NUM_STANDBY_REPLICAS: “2”
KSQL_KSQL_QUERY_PULL_ENABLE_STANDBY_READS: “true”
KSQL_KSQL_HEARTBEAT_ENABLE: “true”
KSQL_KSQL_LAG_REPORTING_ENABLE: “true”
When doing so and restarting the server, I can see that only the first 2 properties are properly set, and I can see the last two (for example with SHOW PROPERTIES from the ksqdb CLI).
Do you have an idea about why I can’t see them?
Do I have to manually deploy a second ksqldb server with the same ksql.service.id?
If this is the case, what is the correct way to do it? Are there particular properties to be set?
I have a Dockerized Django application which have a number of CRON-jobs that need to be executed.
Right now I'm running it with the package Supercronic (which is recommended for running cron-jobs inside containers). This will be deployed on a two servers for redunancy-purposes, i.e. If one goes down the other one need to take over and execute the cron-jobs.
However, the issue is that without any configuration this will result in duplicate cron-jobs being executed, one for each server. I've read that you can set up something called a "lease" for the cron-jobs to retrieve, to avoid duplicates from different servers, but I haven't found any instructions on how to set this up.
Can someone maybe point me in the right direction here?
If you are running Supercron in two different instance, Supercron doesn't know about whether the job gets triggered, Its up to the application to handle the consistency.
You can do it in many ways either controlling the state with File or DB entries or any better way where your docker application can check the status before it start executing the actual process.
I've got a project with 4 components, and every component has hosting set up on Google Cloud Run, separate deployments for testing and for production. I'm also using Google Cloud Build to handle the build & deployment of the components.
Due to lack of good webhook events from source system, I'm currently forced to trigger a rebuild of all components in a project every time there is a new change. In the project this means 8 different images to build and deploy, as testing and production use different build-time settings as well.
I've managed to optimize Cloud Build to handle the 8 concurrent builds pretty nicely, but they all finish around the same time, and then all 8 are pushed to Cloud Run. It often seems like Cloud Run does not like this at all and starts throwing some errors to me that I've been unable to resolve.
First and more serious is that often about 4-6 of the 8 deployments go through as expected, and the remaining ones either are significantly delayed or just fail, often so that the first few go through fine, then a few with significant delays, and the final 1-2 just fail. This seems to be caused by some "reconciliation request quota" being exhausted in the region (in this case europe-north1), as this is the error I can see at the top of the Cloud Run service -view:
Additionally and mostly annoyingly, the Cloud Run dashboard itself does not seem to handle having 8 services deployed, as just sitting on the dashboard view listing the services regularly throws me another error related to some read quotas:
I've tried contacting Google via their recommended "Send feedback" button but have received no reply in ~1wk+ (who knows when I sent it, because they don't seem to confirm receipt).
One option I can do to try and improve the situation is to deploy the "testing" and "production" variants in different regions, however that would be less than optimal, and seems like this is some simple configuration somewhere about the limits. Are there other options for me to consider? Or should I just try to set up some synchronization on these that not all deployments are fired at once?
Optimizing the need to build and deploy all components at once is not really an option in this case, since they have some shared code as well, and when that changes it would still be necessary to support this.
This is an issue with Cloud Run. Developers are expected to be able to deploy many services in parallel.
The bug should be fixed within a few days or couple of weeks.
[update] Bug should now be fixed.
Make sure to use the --async flag if you want to deploy in parrallel: gcloud run deploy $SERVICE --image --async
We use delayed_job in a fairly high traffic Rails app, so there are always jobs in the queue.
After a deploy, when you restart dj, there is potential for the code in already queued jobs to become invalid. For example, if the deploy meant that a class used in the job no longer existed, the job would fail.
Is there a standard practice for getting round this?
Is there a standard practice for getting round this?
Yes, multi-stage deployments (or whatever is the proper name for this).
Step 1: Deploy a new version of code that allows both new and old jobs run at the same time. For example, if you want to remove some class, make new jobs not use it, but don't remove the class yet, so that it's available to old jobs.
Step 2: Wait until all old jobs are processed.
Step 3: Deploy a change that removes the class and the old job version.
Note: you could implement step 1 by copying old job code and giving it a new "version". If you had, say, ProjectJobs::Import, you copy it as ProjectJobs::Import2 (this version won't use class that you want to remove). Since they're different classes, DJ won't have any problems with picking appropriate implementation.
I would say, that you have to:
stop all workers right after they finish their jobs
deploy changes
start workers
I think that in your case, this code might be helpful.