I was playing a bit with Cygnus, and I was wondering how to properly configure it. I’ve seen both agent_<id>.conf and cygnus_instance_<id>.conf files are needed. I understand the purpose of the first one, but not the second one. In addition, what about the grouping_rules.conf file? Is there any other configuration file?
File by file:
agent_<id>.conf file is the main configuration file for Cygnus. It inherits the syntax from Apache Flume, the base technology used by Cygnus, thus it is used to declare and specify the sources, channels and sinks of your Cygnus agent.
cygnus_instance_<id>.conf file is used to configure other Cygnus parameters that cannot be configured as part of the agent configuration, such as the logging file, the management interface port, etc. Cygnus service will run as many instances as cygnus_instance_<id>.conf files are configured. That's why an <id> must be provided, since this <id> will be used to find the proper agent_<id>.conf file.
grouping_rules.conf file is used when the Grouping Rules advanced feature is wanted to be used. Usually, this file may be empty (but it must exist) and Cygnus will run after all.
flume-env.sh file has been inherited from Apache Flume, and it is used in order to configure certain Flume paramters such as the classpath overwritting the default one, some Java options (-Xms, -Xmx, etc)...
Related
I have a complex project with multiple docker-compose files which get loaded depending on the environment I want to simulate.
I would like to see documentation related to the use of multiple compose files. I have the feeling that some sections are additive (ports) while other sections follow a replace strategy (environment)
I am unable to find documentation about this. Is there an official description of the behaviour?
I could prepare a simple PoC to test this, but I would prefer to refer to an official explanation to avoid reaching the wrong conclussions.
This is all I am able to find:
You can supply multiple -f configuration files. When you supply
multiple files, Compose combines them into a single configuration.
Compose builds the configuration in the order you supply the files.
Subsequent files override and add to their predecessors.
I read that "override and add to their predecessors" as meaning "some sections add, some sections override". I would like to know which ones add, and which ones override.
See the "Adding and overriding configuration" section of the "Extend services in Compose" page (emphasis in original):
For single-value options like image, command or mem_limit, the new value replaces the old value.
For the multi-value options ports, expose, external_links, dns, dns_search, and tmpfs, Compose concatenates both sets of values.
In the case of environment, labels, volumes, and devices, Compose “merges” entries together with locally-defined values taking precedence. For environment and labels, the environment variable or label name determines which value is used.
Entries for volumes and devices are merged using the mount path in the container.
That page also has several examples.
This matches the behavior you're observing: if you have two files that both have ports: they simply get concatenated together, but if you have two files that both have environment: variables, the last definition for each individual variable takes effect.
I'm using SCDF and i was wondering if there was any way to configure default properties for one application?
I got a task application registered in SCDF and this application gets some JDBC properties to access business database :
app.foo.export.datasource.url=jdbc:db2://blablabla
app.foo.export.datasource.username=testuser
app.foo.export.datasource.password=**************
app.foo.export.datasource.driverClassName=com.ibm.db2.jcc.DB2Driver
Do i really need to put this prop in a property file like this : (it's bit weird to define them during the launch)
task launch fooTask --propertiesFile aaa.properties
Also, we cannot use the rest API, credentials would appear in the url.
Or is there another way/place to define default business props for an application ? These props will be only used by this task.
The purpose is to have one place where OPS team can configure url and credentials without playing with the launch command.
Thank you.
Yeah, SCDF feels a bit weird in the configuration area.
As you wrote, you can register an application and create tasks, but all the configuration is passed at the first launch of the task. Speaking other way round, you can't fully install/configure a task without running it.
As soon as a task has run once, you can relaunch it without any configuration and it uses the configuration from before. The whole config is saved in the SCDF database.
However, if you try to overwrite an existing configuration property with a new value, SCDF seems to ignore the new value and continue to use the old one. No idea if this is intended by design or a bug or if we are doing something wrong.
Because we run SCDF tasks on Kubernetes and we are used to configure all infrastructure in YAML files, the best option we found was to write our own Operator for SCDF.
This operator works against the REST interface of SCDF and also compensates the weird configuration issues mentioned above.
For example the overwrite issue is solved by first deleting the configuration and recreate it with the new values.
With this operator we have reached what you are looking for: all our SCDF configuration is in a git repository and all changes are done through merge requests. Thanks to CI/CD, on the next launch, the new configuration is used.
However, a Kubernetes operator should be part of the product. Without it, SCDF on Kubernetes feels quite "alien".
I've got a requirement to create some impostors/stubs/reflectors (pick your own term...) using Apache Camel. These stubs need to:
listen to a bunch of IBM MQ queues
for each queue:
grab messages off the queue when they appear
extract info from the message via simple XPath or regex extracts and construct a response via a template
wait some predefined period
send the response back on another queue
I'm no Camel expert, but I can work out how to do that much...
However, given there's going to be lots of these stubs and I want to use different subsets of these stubs in different circumstances, I want to build a generic Camel Docker image, and apply different sets of stubs to it when I start the Docker container.
If it helps clarify things further, I want to be able to store the stub definitions as uncompiled code (e.g. XML, Simple, whatever) in git repos - separate from the Docker image - and have the Docker/Camel container load those stub definitions via either volume mount/s or environment variables. Once the container starts, those stub images will persist till the container is killed off - I don't need to manipulate the stubs except when the container starts.
Key thing is that the Camel/Docker image has to be generic, not pre-built with a specific set of stub definitions.
I can handle the Docker side of things OK - what I can't work out is how to have Camel load the stub definitions when Camel starts (i.e. when the Docker container is created) rather than have the stubs loaded into e.g. a WAR at compile time.
Thanks for any help or suggestions
It can be easy implemented using the following way:
Camel/Docker image has the main cmd to run camel routes from spring xml (you can use whatever logic you want like camel standalone or spring boot etc.)
Spring XML file is read from some location which is for example hardcoded inside Docker image (like /app/config.xml for example)
You will start docker container and map required spring XML file to the /app/config.xml
If you cannot map XML files to the running container then you can implement you own small boot part that will on startup read XML content from some env variable, store it into the temporary file and run same logic for starting spring using file context. In that case, you can run docker container and pass spring XML as an env. variable.
I have 5 Jenkins servers (for projects) + 1 (for administration) with common a config.xml.
But for each server I want to have specific Dashboard view. These views are defined in the config.xml (tag : hudson.plugins.view.dashboard.Dashboard)
is there a way to have a specific configuration files and import it in the common config.xml (like a import resource="./config-specific.xml) ? (I try it= > don't work))
I can't have a specific config.xml by server, it must be common for each server.
Thanks
XIncludes could be a solution to that.
The only point is that you must enable the XInclude feature for the XML parser that's used by Jenkins (by default, it's disabled). If you're willing to modify core/src/main/java/hudson/XmlFile.java, then adding a JAXP.setXIncludeAware(true) should do the trick.
I didn't figure out yet if it's possible to do the same dynamically by setting a static system property like -D...=true. Also, XIncludes may be lost when Jenkins updates the config files itself.
I'm building a plugin that will contain sharable code and configuration for several applications. One things that I'm trying to share is the data source information. Basically I need the application to not have to define it's own data source and instead use the plugin's data source. The best way that I can think of doing this is to take advantage of the external configuration functionality that's available in Grails (http://grails.org/doc/latest/guide/conf.html#3.4%20Externalized%20Configuration). However, I'm not exactly sure how to do this. All the examples I can find online show you how to do this when using an external file on the file system somewhere. I want to use configuration files from the plugin.
According to the documentation linked to above you can specify a "config script" class to use like this:
grails.config.locations = [com.my.app.MyConfig]
This would probably work, however, I can't find documentation on what a "config script" class actually is and how to create one.
By default, the DataSource file of your plugin will not be used (it's ignored in the package stage) but you can create another file that ends with "DataSource" (eg MyPluginDataSource). This is also true for BootStrap and Config.
You will probably need to leave the application DataSource file empty.