I have a complex project with multiple docker-compose files which get loaded depending on the environment I want to simulate.
I would like to see documentation related to the use of multiple compose files. I have the feeling that some sections are additive (ports) while other sections follow a replace strategy (environment)
I am unable to find documentation about this. Is there an official description of the behaviour?
I could prepare a simple PoC to test this, but I would prefer to refer to an official explanation to avoid reaching the wrong conclussions.
This is all I am able to find:
You can supply multiple -f configuration files. When you supply
multiple files, Compose combines them into a single configuration.
Compose builds the configuration in the order you supply the files.
Subsequent files override and add to their predecessors.
I read that "override and add to their predecessors" as meaning "some sections add, some sections override". I would like to know which ones add, and which ones override.
See the "Adding and overriding configuration" section of the "Extend services in Compose" page (emphasis in original):
For single-value options like image, command or mem_limit, the new value replaces the old value.
For the multi-value options ports, expose, external_links, dns, dns_search, and tmpfs, Compose concatenates both sets of values.
In the case of environment, labels, volumes, and devices, Compose “merges” entries together with locally-defined values taking precedence. For environment and labels, the environment variable or label name determines which value is used.
Entries for volumes and devices are merged using the mount path in the container.
That page also has several examples.
This matches the behavior you're observing: if you have two files that both have ports: they simply get concatenated together, but if you have two files that both have environment: variables, the last definition for each individual variable takes effect.
Related
I'd like to be able to define the variable values in DDEV's version-controlled ./ddev/config.yaml file, using the non-version-controlled .env file. For example (pseudo-code):
name: env($PROJECT_NAME)
# etc...
I can't rely on remembering to swap out config.yaml files or any other manual steps.
The reason for the season is that I need to have multiple DDEV instances of the same site. Each instance would be committing to the same repo, but may (or may NOT) have different branches. In other words, they need to be capable of being merged with each other, without DDEV getting mixed up. Since I have .ddev/config.yaml committed to the repo, I need some other way of having separate DDEV instances.
You probably want to use config.*.yaml for this. By default, config.*.yaml are gitignored. For example, config.local.yaml might have local overrides. See the docs for more info.
I haven't experimented with using the .env file in this context, but I know that config.local.yaml will work fine for this use.
I'm digging into the caching of Docker buildx to try to debug an issue. I'm trying to figure out how, exactly, buildx checks if a layer is available in the local cache. Although I've searched fairly extensively, I can't seem to find any documentation on this.
Looking at the local cache files themselves, I see a bunch of files with hash names. My assumption is that it works as follows (assuming use of type=local,mode=max):
For each line in the Dockerfile, it uses some combination of parameters to calculate a SHA hash.
It checks in the --cache-from directory to see if a file with that hash as the name exists
If it does exist, it uses that file as the layer and doesn't re-build anything (and copies that file to the --cache-to directory.
If it does not exist, it builds the layer and saves it as a file, with that hash as the name, in the --cache-to directory.
This results in an output cache with 1 file for each line in the Dockerfile.
So my questions are:
Is my understanding of this process correct? Am I missing any key elements?
For step (1) above, what are the "parameters" that it uses to calculate the hash? I would think it's the string value of the line itself, plus the value of any files that are copied by the line (e.g. ADD), but does it use anything else? e.g. the last-modified timestamp of any files that it copies?
Is my understanding of this process correct? Am I missing any key elements?
My understanding is roughly along those lines. I'd need to check the code myself to know the specifics.
For step (1) above, what are the "parameters" that it uses to calculate the hash? I would think it's the string value of the line itself, plus the value of any files that are copied by the line (e.g. ADD), but does it use anything else? e.g. the last-modified timestamp of any files that it copies?
In general, caching of Dockerfile steps uses the following (this predates buildkit):
For an ADD/COPY step, a hash of the source files. That hash includes file ownership and permissions. A quick test indicates the modification timestamp is not included in that (the cache is still used after I touched a file being copied).
For a RUN step, any ENV or ARG key/value pairs are included, because they modify the environment, along with the text of the command being run. Docker has no concept of commands pulling from external resources, and it doesn't know what environment variable changes will impact any specific command.
For all steps, the cache requires that the previous step's result plus the current step matches (the new COPY --link may be an exception to this). So if you copy a new file into the image, all remaining steps of that build stage are no longer found in the cache, docker has no way to generically know that a specific file doesn't affect some RUN steps.
I am trying to figure out if there is a difference between the two pre-defined variables in TFS Online 2017: $(Build.Repository.LocalPath) and $(Build.SourcesDirectory). I have a build that uses these two variables and didn't know if I could use them interchangeably or not.
Looking at Microsoft's documentation the descriptions are as follows:
$(Build.SourcesDirectory): The local path on the agent where your source code files are downloaded. For example: c:\agent_work\1\s
By default, new build definitions update only the changed files. You can modify how files are downloaded on the Repository tab.
$(Build.Repository.LocalPath): The local path on the agent where your source code files are downloaded. For example: c:\agent_work\1\s
By default, new build definitions update only the changed files. You can modify how files are downloaded on the Repository tab.
Are these representing the same thing or am I missing something?
They're synonyms. Most standard templates and tasks use the $(Build.SourcesDirectory), so that is what I tend to use.
They often result in the same but not necessarily. As described in the docs:
If you check out multiple repositories, the behavior is as follows (and might differ from the value of the Build.SourcesDirectory variable):
The description for Build.SourcesDirectory on the same page contains a similar note.
Basically if you want to define a custom path for the self checkout and still not need to specify the extra dir, you specifically need Build.Repository.LocalPath.
For clarity, you can still use Build.SourcesDirectory to resolve to the full path if you have the usual
- checkout: self
path: s
and I'd recommend using it whenever possible if so. If you have something like
- checkout: self
path: main_project
then you'd need $(Agent.BuildDirectory)/main_project to reach the same.
I was playing a bit with Cygnus, and I was wondering how to properly configure it. I’ve seen both agent_<id>.conf and cygnus_instance_<id>.conf files are needed. I understand the purpose of the first one, but not the second one. In addition, what about the grouping_rules.conf file? Is there any other configuration file?
File by file:
agent_<id>.conf file is the main configuration file for Cygnus. It inherits the syntax from Apache Flume, the base technology used by Cygnus, thus it is used to declare and specify the sources, channels and sinks of your Cygnus agent.
cygnus_instance_<id>.conf file is used to configure other Cygnus parameters that cannot be configured as part of the agent configuration, such as the logging file, the management interface port, etc. Cygnus service will run as many instances as cygnus_instance_<id>.conf files are configured. That's why an <id> must be provided, since this <id> will be used to find the proper agent_<id>.conf file.
grouping_rules.conf file is used when the Grouping Rules advanced feature is wanted to be used. Usually, this file may be empty (but it must exist) and Cygnus will run after all.
flume-env.sh file has been inherited from Apache Flume, and it is used in order to configure certain Flume paramters such as the classpath overwritting the default one, some Java options (-Xms, -Xmx, etc)...
How do you deal with host-specific configuration for docker containers?
Most importantly production passwords. They can't be put to container for security reasons. Also, they need to be changed on regular basis w/o redeploying anything.
I mostly use Volume containers if I really want to avoid making an image for that: http://docs.docker.com/userguide/dockervolumes/
however, I have done things like
FROM myapp
ADD password.file /etc
RUN do some cfg
to make specific images with password baked in.
That way the specific configuration list passwords is not in the general images which are re-used and known, but they are in an image that is build for and on the host that is running it. (this approach is not really very different to using puppet/chef etc to customise the files on a vm/server - it eventually needs to hit the filesystem in a safe, but repeatable way)
As one comment says, use environment variables you can pass via -e pass=abcdef. You can also save them in a fig.yml config file to avoid having to type them every time.