First of all my apologies that the question is under stackoverflow not stack exchange, I don't have enough points to ask it there.
I've created a packer template in which creates my image(the image includes the code for my application, nginx, php-fpm and ...)
If you have used packer before, you will know that at the end of the process it will give you the image_id, I need to use this image id in order to update the template for my cloudformation on aws,
the cloud formation template will create an launch configuration based on the image_id from the packer. later on the launch configuration will be used to create an autoscaling group,which is connected to an ELB(The ELB is not under cloudformation).
Here are my questions:
1-whats the best way to automate the process of getting the id from packer and updating the cloudformation template?(To elaborate more, i need to get the id somehow, for now the only thing that I can think of is a bash command, but this cause an issue if I want to use jenkins later on.what are other alternatives?)
2-Lets say I managed to get the id, now whats the best policy to update the cloudformation template?(Currently aws CLI is my only option any better solution)?
2-How to automate these whole process using jenkins?
I would put a wrapper Python/Ruby script that would run packer, then call cloudformation reading from the packer output.
Related
I'm using SCDF and i was wondering if there was any way to configure default properties for one application?
I got a task application registered in SCDF and this application gets some JDBC properties to access business database :
app.foo.export.datasource.url=jdbc:db2://blablabla
app.foo.export.datasource.username=testuser
app.foo.export.datasource.password=**************
app.foo.export.datasource.driverClassName=com.ibm.db2.jcc.DB2Driver
Do i really need to put this prop in a property file like this : (it's bit weird to define them during the launch)
task launch fooTask --propertiesFile aaa.properties
Also, we cannot use the rest API, credentials would appear in the url.
Or is there another way/place to define default business props for an application ? These props will be only used by this task.
The purpose is to have one place where OPS team can configure url and credentials without playing with the launch command.
Thank you.
Yeah, SCDF feels a bit weird in the configuration area.
As you wrote, you can register an application and create tasks, but all the configuration is passed at the first launch of the task. Speaking other way round, you can't fully install/configure a task without running it.
As soon as a task has run once, you can relaunch it without any configuration and it uses the configuration from before. The whole config is saved in the SCDF database.
However, if you try to overwrite an existing configuration property with a new value, SCDF seems to ignore the new value and continue to use the old one. No idea if this is intended by design or a bug or if we are doing something wrong.
Because we run SCDF tasks on Kubernetes and we are used to configure all infrastructure in YAML files, the best option we found was to write our own Operator for SCDF.
This operator works against the REST interface of SCDF and also compensates the weird configuration issues mentioned above.
For example the overwrite issue is solved by first deleting the configuration and recreate it with the new values.
With this operator we have reached what you are looking for: all our SCDF configuration is in a git repository and all changes are done through merge requests. Thanks to CI/CD, on the next launch, the new configuration is used.
However, a Kubernetes operator should be part of the product. Without it, SCDF on Kubernetes feels quite "alien".
Was hoping to get some advice for the best approach for creating Docker images for deployment over several environments. Assuming we have 3 environments - DEV, UAT & PRD would it be best to name the image accordingly along with a version tag i.e.
my-service-dev:release-10
my-service-uat:release-8
my-service-prd:release-6
or keep the repository the same but use the tag:
my-service:dev-release-10
my-service:uat-release-8
my-service:prd-release-6
We are using Bamboo as a build/deploy pipeline so can easily append build/release variables.
Any advice much appreciated.
Usually you should use the same image in all environments, so that when you push an image to production you're running the exact code you ran in the test environment. You don't need text like "release" in the tag either, and if you look at standard Docker Hub images it's common enough for the tag to just be a number.
my-service:10
If your CI system can generate sequential build numbers these can work fine. If developers are building test images themselves it may be more convenient to use a time stamp or source control commit ID.
my-service:20201111
my-service:g52fc7a0
If you really need different builds in different environments, standard Docker Hub images encode details like that in tags. For example, the python image has variants python:3.9-alpine and python:3.9-buster (Debian). Often a top-level image name is a "unit" in a registry; you might need to use a tool like Terraform to create a new image name in AWS ECR, but you can push arbitrarily many tags once the image name exists.
my-service:10-test
my-service:11-prod
I'm trying to use a Google dataflow template to export data from Bigtable to Google Cloud Storage (GCS). I'm following the gcloud command details here. However, when running I get a warning and associated error where the suggested fix is to add workers (--numWorkers), increase the attached disk size (--diskSizeGb). However, I see no way to execute the Google provided template while passing those parameters. Amy I missing something?
Reviewing a separate question, it seems like there is a way to do this. Can someone explain how?
parameters like numWorkers and diskSizeGb are Dataflow wide pipeline options. You should be able to specify them like so
gcloud dataflow jobs run JOB_NAME \
--gcs-location LOCATION --num-workers=$NUM_WORKERS --diskSizeGb=$DISK_SIZE
Let me know if you have furthr questions
AWS ECS service points to a Task Definition, which holds a name of docker image, including a tag.
So when I create a new version of my docker image I have 2 possibilities:
Update Task definition to the new version and then update Service to point to the new revision of Task Definition
Use some tag to point to the last version, let's say "current" tag will always point to the last version, Task Definition will contain "my-image:current" and then I need just restart ECS service
What is better and why?
Use a unique tag for every build, and update the task definition to point at the new tag (your first option).
The big problem you'll run into is that, in general, Docker-based systems will try to avoid pulling an image they already have. If you restart the service, but the node it restarts on sees it already has my-image:current, it will just re-run it without pulling the updated version. It seems like it can work – How does "latest" tag work in an ECS task definition and container instances pulling from ECR? includes a setup which appears to work – but it's a little hard to tell just from looking at things what exact version you have in use.
A second good reason to avoid a "current" or "latest" tag is to have the ability to roll back. If you use, for example, a timestamp-based tagging system, you deploy build 20200323, and it doesn't work, it's very easy to reset the task definition back to build 20200322 to get back to yesterday's code while your developers debug things. If it was "current" yesterday, and it's "current" today, it's a lot harder to figure out what "not-current" should be.
I've been playing with Docker for the last 5 days. And now that I'm comfortable with it, I wonder if I should put my application, e.g a war file, right when building it, in other words ADD it inside the Dockerfile, or I should just generate an image that will provide the necessary environment for my application? Is there a best practice? The second option seems to make more sense, If, for example you won't to automate the build and deployment in Jenkins. But still I would like to know.