I have certain environment variables I need to utilize within Gitlab CI. I need these variables to be within Rails credentials, not stored on the Gitlab runners via settings.
I have been working on this for about a week now and am struggling to find a working solution. The ideal format would be able to use ERB-style markdown (i.e. - <%= Rails.application.credentials[:my_credential] %>) but of course I’m not able to preprocess the ERB and have Gitlab CI still function.
How can I utilize ENV vars on Gitlab CI that cannot be manually stored in the runner?
You can store them in config files on your app. Specifically you can create a ./config.toml file that can be used by the runners. You could then use ERB in the file to access your Rails credentials. You would just need to set up a script that would parse it during the deploy.
Here is an example of one way of doing it. Note that you need to add the .erb to the file extension.
Related
I am using Travis-CI to test code in a repository. There are quite some files after the testing and I would like to have them at a persistent place. How can I do that under the context of Travis-CI?
As an artificial example, suppose my Travis-CI server runs a C program that stores a large number of integers in a specific file. The file can be found at the Travis-CI server after the build. But how can I get that file? In my use case, this file is large and it would not make sense to read it from the console of Travis-CI; in other words, I would not consider using "cat ..." in .travis.yml.
After some search, here is what I got:
The most convenient way seems to deploy the generated files to GitHub pages. The process is explained here: https://docs.travis-ci.com/user/deployment/pages/. In short:
first, create a GitHub page from the repository under test. This can be done through the Github web of the repository. The outcome includes an additional remote branch called gh-=pages generated.
then, in .travis.yml, use the deploy section to specify the condition to do the deployment.
I am currently doing a POC on Jenkins pipeline to figure out how to configure my product in a CI environment. The requirements of the pipeline are:
Checkout code from SVN
Compile the program
Deploy to a predefined location on the server
Change DB configurations (& maybe even other configs not identified yet) to point to the appropriate DB
Execute the program
Execute QA process to validate the output
I am currently having difficulty in achieving Point 4 above. All DB-related configurations reside in a database.xml file per program & a program can connect to 1 or more DBs.
Given that developers are free to check-in any DB configurations, I would still like my CI environment to point to a predefined DB to test against. I am unsure on how to dynamically change these configuration files to achieve this.
Please let me know if there are standard methods that others are also using to achieve the same.
TIA
Some approaches:
Properties using Advanced Platforms
Use some web platform like :
zookeeper
http://www.therore.net/java/2015/05/03/distributed-configuration-with-zookeeper-curator-and-spring-cloud-config.html
Spring Cloud
https://www.baeldung.com/spring-cloud-configuration
This is a java spring framework functionality in wich you can create properties file with configurations and configure your applications to read them.
magi-properties-management
This is a java web system in which you can create environments and any key:value in each one. You just need configure your application in any language to read this values.
cyber-properties-management
This is a nodejs application that allows you to store properties files (.properties .yml or .json) and then just consume them as rest endpoint from your applications.
With this approaches , when a change of configurations is required, you just need update the value in the system and restart your application. It is even possible a hot reload in java applications.
Properties from Environment variables
You can export your key:value properties as environment vars before starting the application :
export DATABASE_HOST=10.100.200.300
export LOG_DIR_LOCATION=/logs
And read it after after the application has started:
Java >> System.getEnv("DATABASE_HOST");
node.js >> process.evn.LOG_DIR_LOCATION
php >> getenv('DATABASE_HOST')
Properties from SCM
Create some svn repositoty called development-configurations
Upload your database.xml with development values
In your application, put a database.xml with dummy values : localhost, etc
Create a jenkins job and put the environment as an argument.
In the same job download svn source code of your application.
download svn repository called $environment-configurations. $environment will be your argument
replace the database.xml inside of your application with database.xml of $environment-configurations repository.
Just create another repositories for testing, uat and production. Job must be receive environment as an argument to choose the right database.xml
Properties from Database
Modify your applications to read configurations from some database instead of xml file
Properties from File System
Modify your application to read an external database.xml instead of the database.xml inside of your source code. With this approach you just need put the database.xml in some path of your server and delete it from your application source code.
Note
You can use these approaches not only for backend apps. You can use them for frontends applications:
Devops Variable Substitution for Frontend js applications
Does Jenkins provide a way to generate per-environment (dev/qa/staging/prod) deployment config files based on some templates using substitution variables? Kind of like template task in Ansible?
No, Jenkins does not provide this itself. There might be some plugin, which actually does this, but Jenkins in general just executes commands. These commands could then call Ansible, some other templating engine or even sed to replace tokens in files.
I'm trying to setup something with WebDeploy that will allow configuration of Environment Variables to set the "parameters.xml" parameters in a web deploy package.
From what I've read so far, this should be possible, but I've not had any success yet.
Essentially, it's TeamCity that I'm using to do it, but the concept is beyond a usage in TeamCity.
I'm using the generated cmd that you get from using the /t:Package target.
So my question is, is this possible? or is there another solution for iterating the teamcity variables and updating the SetParameters.xml (rather than manually coding an XML Poke foreach, or using /property syntax on the raw MsDeploy.exe).
You should be able to reference environment variables from the command line when you run the msdeploy command. I don't have experience with TeamCity but I suspect you give it a msdeploy.exe command to run.
This post gives more details - http://evolutionarydeveloper.blogspot.com/2013/05/specifying-environment-variables-at.html
I'm using Jenkins and deploy plugin, with which I'm deploying to tomcat server. How can I pass parameters from properties file to this deploy plugin?
for example I want to pass my property app.server.url to Tomcat URL field.
I tried to pass $app.server.url also %app.server.url%, but that doesn't work.
I am assuming you have a properties file in the format app.server.url=somevalue
You then can inject these properties into the Jenkins environment using EnvInjec Plugin. Use Inject environment variables for your job build step, and just specify your file path (leave the content field blank). After that, you can reference it like this $app.server.url for *nix, or %app.server.url% for Windows. However this only works on the shell level. On the plugin properties level, a lot of plugins are expecting properties only in *nix-style format.
The further problem is that *nix-style variables do not allow dots .. So $app.server.url is not valid on *nix. I do not know if it will work on plugin-level on Windows though.
So after setting up EnvInject plugin, try the following two:
Try using $app.server.url in your deploy plugin.
Else change the property file to something like app_server_url=somevalue, and then try $app_server_url in deploy plugin.