I have Jenkins set up, connected with Gitlab, to do some deployment stuff whenever a merge request gets accepted into the development branch. It pulls the code, runs some incantations, increments the version number, and commits+pushes that new version number in. The only issue, that I just discovered, is that updating old, already merged MRs, even in trivial ways such as updating the milestone or title, will trigger the build. Because of this, when I just updated multiple old merge requests to have the milestone for this release, the version number got incremented ten times or so- not ideal.
My Gitlab triggers inside Jenkins are quite closed down (please note I'm not on EE):
My Gitlab integrations settings only fire on MR events (and I'm aware there's no way to say only on merging in, rather than editing):
I also have some custom build scripts that do bits and bobs, but the main part that bumps the version number checks that if [ "$GIT_COMMIT" != "$GIT_PREVIOUS_SUCCESSFUL_COMMIT" ]; then. However, even though this seemed logical at the time, it became apparent recently that old, stale merge requests will still be true for this when compared to the development HEAD.
There are many different aspects to this that could be the cause of the trouble, and it may be a very easy solution, but I'm not sure what that would be quite yet. Any help or suggestions greatly appreciated.
Related
Have a git flow question if anyone can help. I'm not really comfortable with using the git flow approach to things so this may be a very stupid question, but here goes:
The way my company handles things is by naming each feature after a specific issue code that relates to the GitHub "issues" page. So, for example, I finished feature/issue-3118 a short time ago. In that branch, I had to remove 2 columns from the user table and clear out (or change) anything that was trying to access it. This was completed, all tests passed etc so I pushed the branch. I then did a git checkout develop and started a new feature branch for the next task assigned to me (called feature/issue-3201).
The problem I have now is that the current branch is failing a bunch of tests that were fixed in the last branch. Running rake db:migrate:status shows my 2 previous migrations as now being orphaned so methods etc. trying to access them are failing tests.
The issue-3118 branch is still in review so I can not just update my master and be on my way. Rather than going through all the database changes again manually (or waiting for the master to be updated), is there an easy way to get over this issue? If I merge the branches, won't this cause conflict issues when I push the current branch?
(If this does indeed sound like a really stupid question, I am about as junior as junior devs get - started my first web development job on August 1st!)
One option, which is likely the way that I would do it, is to start the new feature branch from your feature/issue-3118 branch, rather than from the head of develop. This is not the standard approach, but it sounds as though your new feature is dependent on the completion of feature/issue-3118.
That way, the new feature, will already have those changes in it, and you can continue work there.
Then, once feature/issue-3118 is merged into develop, you can rebase your new feature branch on develop, and continue work.
This maintains the segregation of the work that needs to happen, but allows you to continue while issue-3118 is being reviewed.
Likely, you should get sign off from other people in your team prior to going down this route.
We are using TFS 2010 with the Basic Branch Plan outlined in the Branching Guide on codeplex
for an internal web application. We have just the 3 basic branches = Dev, Main (QA/Testing), and Release (Production).
Because the app is an internal web application, we only support the single production release.
We basically develop locally and once we complete a task (a bug fix or enhancement), we commit it to Dev. We also generally do a Get Latest from Dev every day when we start work to pull down anything checked in by the other developers. After some period of time (usually a week or two), we'll decide we have enough changes to justify updating the QA site and do a Merge All from Dev to Main and then deploy the merged Main branch to a QA server for testing.
QA will then start testing the site, and after they're satisfied, we'll do a Merge All from Main to Release and deploy the merged Release branch to our production server. Sometimes we even wind up doing multiple Dev-to-Main merges before actually merging everything on up to Release.
Anyway, we've been using this strategy for a couple of months now and until recently everything was looking great. We were able to hotfix Release if we ran into some critical problem in production and then just merge it backwards. All was looking good.
Then we ran into something we didn't know how to deal with. We were given the directive of merging ONLY a single code fix on up from Main to Release (without merging everything else in Main). Now since we didn't know this was coming, when the original changeset was merged from Dev to Main, it was merged along with several other changesets. So when I went to merge from Main to Release, the only option I had was for the entire merged changset. I couldn't "drill-down" into the merged changeset and pick just the one original changeset from Dev that I really wanted.
I wound up manually applying the change like a hotfix in Release just to get it out there. But now I'm trying to understand how you prevent a situation like this.
I've read several articles on merging strategy and everything seems to recommend NOT cherry-picking changesets when you go to merge - to simple merge everything available... which makes sense.. but if you always merge multiple changesets (and they become one changeset in the destination branch), then how do you potentially merge only one of the original changesets on up to production if the need arises?
For example, if merging Dev (C1, C2, C3) to Main (becomes C4) - then how to merge only C1 from 'within' C4 on up to Release?
It makes me think we'd be better off merging every single changeset individually from Dev to Main instead of doing several at once. At least then we could easily just take one on up from Main to Release if the need arises.
Any recommendations/life lessons/etc. on handling branching/merging for this specific scenario would be greatly appreciated.
In your scenario you could have done the following:
Rollback C4 in Main (becomes C5, because rollbacks are changesets themselves, which apply inverse changes)
Merge from Dev to Main again, but this time select only C1 (becomes C6 in Main).
Now rollback changesets C5 and C6 again, so you have all changes in Main like before. (becomes C7 in Main).
After this you have the same code base in Main as before and you can now merge C6 (which has only the changes from C1) from Main to Release.
However, to prevent such trouble in future you should really consider merging every single changeset from dev to main separately.
I would not recommend merging every single change-set from dev to main; That would be a bad idea with much additional risk!
but if you always merge multiple changesets (and they become one
changeset in the destination branch), then how do you potentially
merge only one of the original changesets on up to production if the
need arises?
You don't and should not let the need arise.
This is probably not the easy answer that you are looking for, but there really is no easy answer. Merging every single change-set is creating a massive amount of effort to prepare for something that should not be happening anyway. Indeed the process of merging individual change-sets introduces yet more complexity that will, in the end, bit you in the ass when you can't figure out why your software is not working... "dam, I missed change-set 43 out of 50"...
If the result of a bug:
In your scenario it may have been better if you manually re-applied the "fix" to either a "hotfix" branch off of Release or directly to the Release line.
That is just the cost of having bugs slip through to production and I would spend a little time figuring out why this problem got passed QA and how to prevent it in the future.
If the result of an enhancement:
Did your financial (CFO) guys authorise the reduction in quality in production that is a direct result of shipping untested code? I hope that they did as they effectively own the balance statements upon which that software is listed as an organisational asset!
It is not viable to ship only one feature, built and tested with other features, to production without completing your entire regression cycle again.
Conclusion
I would not recommend merging every single change-set or feature from dev to main; That would be a bad idea with much additional risk that should be hi-lighted to the appropriate people!
I've been looking around online at ways of improving our build time (which is currently ~30-40 minutes, depending on which build agent gets the task), and one common theme I've seen is use CI builds.
I understand the logic behind this, and it makes sense that it would reduce the time each build takes. Our problem, however, is that building on every check-in is a pointless use of our resources, because in our development branch, we only keep the latest successful build. This means that if 2 people check-in in a short space of time, whoever checked-in last will be the one whose build is kept.
It's this reason (along with disk space limitations) that we changed to using Rolling Builds, so that we only built the development branch a maximum of once every 45 minutes (obviously we could manually trigger builds on otp of that).
What I want to know (and haven't been able to find anywhere) is whether there's a way of combining rolling builds AND continuous integration. So keep building only once every 45 minutes, but only get and build files that have changed.
I'm not even sure it's possible, and if not then I'll look into other ways, but this seems like something that should be possible.
Like every commit has a reason and purpose, I think each deploy has a purpose and reason. Source code commits have a comment. But deploying doesn't have any.
How do I record a reason and purpose for each deploy automatically?
I need to keep a record of:
Who deployed to where and what time.
Why deployed? Bug fixes? Feature update? Emergency fix not on iteration plan?
Which git or svn ref was used?
Have anybody felt the need for this kind of system? How do you feel about my approach?
How can I achieve my goal? I'm currently using Capistrano for deployment.
A bounty added. I'd like to hear more stories from different developers who are doing "continuous deployment".
I found two services that do deploy tracking:
Codebase
Hoptoad
Webistrano - https://github.com/peritor/webistrano/wiki - is a web interface to capistrano, that also tracks who's deployed what and when, so that could be worth investigating.
My current project uses a modified version of the apinsein's git-deployment recipe, which (when you tell cap to do a deploy) will tag current HEAD with a Git tag (which gives you all the benefits of normal Git commits).
I've built a web service for this exact problem, http://deploytracking.com, it hooks into capistrano and records the time, user, branch, ref, environment and repo that was involved in the deployment.
Strano - The Github backed Capistrano deployment management UI.
Regarding continuous deployment, I also submitted pull request there, which Introduce automatic deployments for GitHub projects, for now it simply triggers deploy task when somebody push to the master branch.
I don't know if it is still relevant but I would like to come up with a different solution. I am building a new deployment tool that does just what you are looking for.
I do not intend to spam my stuff here but since I am building something that could help you...
Anyway, have a look here https://alessiosantocs.github.io/Captain. I'm gathering feedback so if you have any please let me know.
Update
As suggested, I'm giving an explanation :)
I have also felt this need. I work in a digital startup and we're constantly deploying stuff 5 days a week on different Ruby on Rails application with Capistrano.
What we noticed was that for every single deployment, we should have done several things:
Keep track of which pull requests and commits went online that exact moment
Give some sort of a name to the deploy so we could recognize it
Alert our team members so that everyone could have been on the same page (without asking us of deployment's news)
Keep track of every deployments for future bugs and errors we might find at some point in time (which happened often)
So for this reason we started developing this custom solution that would integrate with Capistrano and our SCM (bitbucket) and keep track of every change we made to our master branch. This is what it does right now.
We are currently tracking deployment environment, repo source, deployment branch and revision. Mainly we manage pull requests, because we found that pull requests, better than commits, did solve an organizational issue in our team (it was difficult to approve other team member's code without a rigid system like PRs)
I would like to explain more about Captain and about our personal dev management strategy with you guys if you want.
Thanks #thirumalaimurugan for asking for clarification!
Update 2
We tried git tagging too. It was good and fun at the beginning but we couldn't manage them very well.
A tag is basically a bookmark to a specific revision. So we're talking about commits. A tag keeps no track of pull requests. It was quite a mess for us.
I don't think they're bad at what you're trying to achieve, but I think there must be some other solutions that could fit exactly your (and our too) problem.
I am curious on how others manage code promotion from DEV to TEST to PROD within an enterprise.
What tools or processes do you use to manage the "red tape", entry/exit criteria side of things?
My current organisation is half stuck between some custom online forms type functionality and paper based dependencies to submit documents, gather approvals and reviews.
All this is left in the project managers hands to track what has been submitted, passed review, approved and advise management if there are any roadblocks that may need approval to be "overlooked" before an application can be promoted to the next environment.
A browser based application would be ideal... so whats out there? please show me that you googlefu is better than mine.
It's hard to find one that's good via google. There is a vast array of tools out there for issue management so I'll mention what we use and what we woudl like to use.
We currently use serena products. They have worked well for us in the past. Team Track is our issue management and handles the life cycle of any issue we work on. Version Manager is our source control and has the feature of implementing promotional groups like DEV TEST And PROD. We use DEV, TSTAGE, TEST, PSTAGE and PROD to signify the movement from one to the other, but it's much the same. The two products integrate nicely so that the source associated with the issues is linked, but we have no build process setup in this environment. It's expensive, but it works well.
We are looking ot move to a more common system using Jira for issue management, Subversion for source control, Fisheye to link the two together and Cruise Control for build management. This is less expensive, totaling a few thousand for an enterprise lisence and provides all the same features but with the added bonus of SVN which is a very nice code version mangager.
I hope that helps.
There are a few different scenarios that I've experienced over the years:
Dev -> Test : There is usually a code freeze date that stops work on new features and gets a test environment the code that has been tagged/labelled/archived that gets built. This then gets copied onto the machines and the tests go fine. This is also usually the least detailed of any push.
Test->Prod : This requires the minor change that production has to go down which can mean that a "gone fishing" page goes up or IIS doesn'thave any sites running and the code is copied over again. There are special cases to this where a load balancer can act as a switch so that the promotion happens and none of the customers experience any down time as the ones on the older server will move once their session ends.
To elaborate on that switch idea, the set up is to have 2 potentially live servers with just one server taking requests that the load balancer just sends all the traffic to one machine that can be switched when the other server has the updated code to go live.
There can also be a staging environment which is between test and production where the process is similar in terms of there is a set date when the promotion happens.
Where I used to work there would be merge days where a developer spent most of a day in Perforce merging code so that it could be promoted from one environment to another.
Now there are a couple of cases where this isn't used:
"Hotfixes" or "Hot patches" would occur where I used to work and in this case the specific files were copied up into the staging and production environments on its own since the code change had to get into Production ASAP since something broke in production or some new thing that had to get done that takes 2 minutes gets done. In this case, the code change getting pushed in had to be reviewed and approved before going out.
Those are the different approaches I've seen used where generally there are schedules and timelines potentially have to be changed or additional resources brought in to make a hard date like if a conference is on a particular weekend that such and such is ready for that.
Of course in a few places there has been the, "Oh, was that broken? Let me see..." and a few minutes later, "No, see it isn't broken for me," where someone changed things without asking permission or anything where a company still has what they call "cowboy programming."
Another point is the scale of the release:
1) Tiny - This is the case where one web page goes up so that user X can do Y.
2) Small - A handful or so of files that isn't really complicated but isn't exactly trivial.
3) Medium - Where going from one environment to another requires changing a bunch of files and usually has scripts to move.
4) Big - Where there are scheduled promotions and various developers are asked for who is taking which shifts when the live push is done. I had this in a case where there was a data migration to do in addition to a release of some new e-commerce sites.
5) Mammoth - Where everything is brand new including how this would be used. I don't think I've ever seen one of this size but I'd imagine Microsoft or Google would have releases of this size.
Somewhere in that spectrum most releases fall and so how much planning and preparation can vary quite a bit and let's not forget that regulatory compliance can be its own pain in getting some things done.