I use Microsoft Team Foundation Server (TFS) for most of my software deployments. TFS allows me to dynamically replace text within specific configuration files during the release process to specific environments (dev, test, prod).
The text it replaces are placeholders called "tokens". For instance, during my automated deployments, TFS will allow us to replace tokens found within configuration files with pre-defined values saved in the build administration for each environment. This way, I don't store any real credentials in source control for any environment. I also don't store any script in source that would hold these sensative credentials. The credentials are dynamically inserted over top the tokens during the release, and the credentials are hosted/saved/configured inside of the release system (not in a script).
For example, I have a configuration file (web.config) that has tokens. A token looks something like this:
MySettingName=${MYSETTINGVALUE}
During the release to DEV, I want the text ${MYSETTINGVALUE} replaced with the word TEN. During the release to PROD, I want that same ${MYSETTINGVALUE} text replaced with the word ORANGE. And I want to store those two values (TEN and ORANGE) in the release administration system, and not in a script.
How do I configure Jenkins to do this same thing?
I have searched up-and-down for this specific answer. While many blogs, articles, documentation exist, none of them speak directly to this issue.
I would prefer NOT to use some additional 3rd party software to do
this.
I would prefer NOT to kick off some manual build and supply these
values each and every time.
I would also prefer NOT to use an Operating System level system
variable (aka evironment variable). In case that server dies, I
would rather not have to remember to setup those OS environment
varialbles on the next server.
Jenkins has a built-in credentials plugin for handling secrets in builds.
See this article on how to use them: https://support.cloudbees.com/hc/en-us/articles/203802500-Injecting-Secrets-into-Jenkins-Build-Jobs
Basically it stores credentials securely and injects them into your jobs as variables which can then be used like any other.
Related
What are the persistence options for fitnesse files? So far it seems like a file system is the only thing supported. There does appear to be an out of date database plugin. Is there anything else that is supported (S3, database, etc.)? Is there a way to control where files are persisted if using the filesystem?
I believe there is very little in that area. The location of the files can be controlled using a command line option. See http://fitnesse.org/FitNesse.FullReferenceGuide.UserGuide.QuickReferenceGuide#FitNesseCommandLINE
-d /path/to/fitnesse/root
How I've used the FitNesse wiki is as a local development tool, with the pages on the file system. Once I'm satisfied with the tests I commit them to version control (e.g. git) so that they become part of the (integration) test pipeline setup (e.g. they are run as part of the CI/CD pipeline of the project).
There is a plugin I believe that will automatically commit any save actions to Git, but I've never used that. Saving each edit action just pollutes version control in my opinion. I only want to see tests after they have been checked/completed, and that tends not to be each save.
Working on a shared wiki environment (where I would expect a non-file system approach would fit in) you run into the same problem, I expect. Developing automated tests is a development task that requires some iterations before it is 'done', and not all attempts reach that 'done' state. So using shared storage for wiki persistence creates 'noise' in the test-set: which are the tests that form the current reference set that should pass and what is work in-progress.
If you are working on a larger project where new features are developed together with their automated tests it becomes even more important to know which test changes belong to which features/changes. Having tests on the file system, in version control, allows you to develop test in sync with code changes in the same branch. This is what I would recommend.
I am looking into how to set up a liferay project with version control and automated deployment. I have a working local development environment in eclipse, but as far as I understand it, setting up a portal in liferay is in part the liferay portal instance running on tomcat and then my custom module projects for customization. I basically want all of that in one git repository which can then be
1: cloned by any developer to set up their local dev environment
2: built and deployed by eg. jenkins into eg. AWS
I have looked at the liferay documentation regarding creating a docker container for the portal, but I don't fully understand how things like portal content would be handled.
I would be very grateful if someone could lead me in the right direction on how a environment like this would be set up.
Code and content are different beasts. Set up a local Liferay instance for every single developer. Share/version the code through whatever version control (you mention git).
This way, every developer can work on their own project, set breakpoints, and create content that doesn't interfere with other developers.
Set up a separate integration test environment, that gets its code exclusively through your CI server, never gets touched manually.
Your production (or preproduction) database will likely have completely different content: Where a developer is quick to create a few "Lorem Ipsum" posts and pages, you don't want them to escape into production. Thus there's no movement of content from development to production. Only code moves that way.
In case you want your developers to work on a production-like environment, you can restore the production content (database) to development machines. Note that this is risky though: The database also contains user accounts, and you might trigger update notification mails from your development machines - something that you want to avoid at all costs. Plus, this way you give developers access to login data (even though it's hashed) which can be abused. And it might even be explicitly forbidden by industry regulations to use production data in development environments.
In general: Every system has its own database (at least their own schema), document store and indexing server. Every developer has their own portal JVM running. The other environments (integration test, load test, authoring, production) are also separate environments. And no, you don't need all of them all the time.
I can't attribute this quote (Milen can - see his comment), but it holds here:
Everybody has a testing environment. Some are lucky to run a completely different production environment.
Be the lucky one. If everyone has their own fully separated environment, nobody is stepping on each other's shoes. And you'll need the integration tests (with the CI output) anyway.
I have reached the point where I have to get my Service Fabric Cluster deployed to Azure :) Besides the the stateful/stateless services I have 2 MVC applications. I currently have a few settings in the web.config files (mostly connection strings).
I plan to configure continuous build / deploy using Visual Studio Online, but have not dogged into to doing that yet.
Where are the recommended place to store the configuration settings. I will need settings for 3 different environments (dev/test/prod).
I found a reference, at some point, to store the settings on the build definition which sounds like a better place to store production credentials than in config files that are being part of the source code for the applications. I need to limit access to values for the production environment and having them in the config files that all developers has access to does not sound like the best way to do this.
Any white papers or best practices regarding this I should be aware of?
You can use de publish profiles and application parameters of the service fabric project to store your settings for each environment.
In my case i have a dev, a homolog and a production environment with different database connection strings, so i created publish profiles named Cloud.Homolog.xml, Cloud.Production.xml and for dev environment i'm still using Local.5Node.xml.
Then, when i want to deploy in some of this environments i choose the correct publish profile.
Here is the documentation for multiple environment management:
Link
I am using Spinnaker to deploy a 3-tier system to QA and then to Production. Configuration files in each of these systems point to others. If I bake in the configuration for QA in the AMI, then how do I change it while promoting to Prod? Is it 1) by having two different sets of AMIs - one for QA and one for Prod, or, 2) by having the AMIs with no configuration and then configure it (somehow) after deployment to change the configuration files?
What is recommended?
You can define custom AWS user data for cluster at deploy time ( under advanced settings of the cluster configuration ). You can then retrieve this user data in your application. This will allow you to change these type of configurations.
At Netflix, we have a series of init scripts that are baked into the base image and provide a mechanism for extending custom startup ( init.d ) scripts via nebula / gradle. This usually sets values like NETFLIX_ENVIRONMENT that are well known and programmed against.
We also use a feature flipping mechanism via https://github.com/Netflix/archaius . This allows us to add properties that are external to the clusters but can be targeted towards them.
When it comes to secured credentials, the approach is outlined in this presentation, but essentially the images reach out to an external service that issues these type of creds. https://speakerdeck.com/bdpayne/key-management-in-aws-how-netflix-secures-sensitive-data-without-its-own-data-center
I am struggling with similar problems myself in our company.
My solution was to create AMIs for specific purposes using a Packer script. This allows me to -
1. Configure the server as much as I can and then store those configurations in an AMI.
2. Easily change these configurations if the need arises.
Then, launching the AMI using an Ansible script, and make all the rest of the configurations on the specific instance.
In my case I chose creating different images for staging and production, but mostly because they differ greatly. If they were more alike I might have chosen using a single AMI for both.
The advantage Ansible gives you here is factoring your configurations, and including written once to both production and staging servers.
How does ASP.NET MVC, if at all, deal with or provide ways to create your application using multiple environments? For example:
Development environment (local machine, probably run via the built-in web server and talking to a local database)
Testing (runs against a preloaded databse with example data, although this part could be skipped and mocks could be used)
Production database on a real server with real data
Ruby on Rails has the concept of environments and "automagically" can deduce if you're in development or production, so you can specify your connection information (connection string) in a config file and the framework dynamically pulls the appropriate one. Is there a similar way of doing things with .NET MVC? If not then how are professional developers using .NET MVC handling different environments?
The only way I can think of is to manually add an "environment" global method (or use an enum, or something like that, maybe this is a use for something like the State pattern?) and store the different connection strings in the web.config file, and then create a base class which all data access classes derive from which provides a way to obtain the connection string for the current environment; this would then have to be set to production when the time comes to put the application live.
Is there another way? Most of the .NET MVC videos and articles I've seen don't even bother with separate environments but only use a development database and don't indicate how you do it in production.
I'd say this is really a question of your company's internal processes. Since every company is a little bit different it's hard to have a "right" generic way to support dev/test/alpha/production and/or other environments.
One way: Create a setup program that supplies the correct connection string based on the environment chosen during the setup process.
Another way: System Admin edits web.config file to supply correct connection string during install.
Yet ANother Way: Connection strings are stored in the system registry.
Even Another Odd Way: You have all your connection strings for all environments in web.config, then a setting in appSettings the tells you which one to use.
Depending on the client, I've done all of these. There are more but these are the more popular.
(One client wanted to store the connecting string in the data base itself. Really.)
You can use alias for your database. You just point these aliases to different servers in the different environments. Stored in the registry under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSSQLServer\Client\Connect if i remember right. Then you use the alias in the connectionstring.
In response to Jason's response:
We use Enterprise Library Environments to configure the different environment paramters and via msbuild invoke the Merge Configuration Tool that generates the different configs for each environment. The deploy process picks the right config file depending on which environment to install.
I was able to solve a similar situation following these steps:
In your Visual Studio, access Build > Configuration Manager
Click in "new"
Choose a name for your configuration, and then copy settings from an existing config. After the configuration creation, it will be available for you to target as build configuration
Create a Web.{env-name-you-chose}.config in your application folder, along with the original Web.config file.
Open your .csproj file with Visual Studio or any text editor
Search for a section that looks like the following and add the highlighted lines, with the config file name you gave previously:
Open your Visual Studio, reload projects if it's required, and now you are able to choose your configuration via CLI or manual publish using Visual Studio.
There is a Publishing Wizard (in Visual Studio) wich let's you change parts of web.config for release build automaticaly. Wich happens to be the feature you are asking about. No magic thou.
What we have done is during our automated build process (Hudson), we alter values in web.config depending on which environment the build is for. Unfortunately there isn't a magical way to do this.
For deployment, which I assume that is what the op was asking about, one creates multiple configurations and in the publish, picks a different configuration. These are called transforms and they operate on the web.config. One would have at least three publish profiles, one for dev, test and prod. One can change more than just the connection string in this way. One can turn on custom errors, turn off debugging and change values of configuration variables. I highly recommend it.
I have a similar question. I have a log table reader. I want it to read log tables in the development, test and production databases. The major difficulty lies in my user account doesn't have permission to look at test and production. It's some silly security thing. The user that I'm impersonating in the application does have permission. I'm struggling trying to tell MVC to build the test and production models using the impersonated user.