I use TeamCity 10.x (Enterprise), as well as TFS for source control.
I recently committed a changeset which did not break the build, but ended up causing hundreds of unit tests to fail across 10+ test projects. Those tests started failing immediately after my check-in, so naturally I assumed that I was at fault.
The changeset included these changes:
Changing the namespaces of various files (e.g., moving all test stubs to a "Stubs" folder and updating their namespace to end with the ".Stubs" suffix);
Updating the using statements in other files to reference these new namespaces where necessary;
Rewriting some unit tests using the Moq library, replacing TypeMock.
I rolled back the entire changeset, in the hopes that all the unit tests would pass again.
Unfortunately, most of the test projects continue to have a lot of failing tests. Additionally, these failing tests are causing all of the remaining tests to be skipped by the VS Test Engine. This is new behaviour that I hadn't seen before.
Questions:
Why are the tests still failing even though I have rolled back the offending changeset?
What can I do to fix this?
If I haven't provided enough information in this post, please let me know and I shall update.
Please try below items to narrow down the issue:
Check that if you indeed rollback the changeset.
Check how TeamCity get the sources, get the latest changeset or
specific changeset?
Try to clean caches and pervious sources on agent machine（Clean all files before build）, then try it again.
Reference : Clean Checkout and
Clean Sources
Related
I have a TFS 2017 instance running on windows server 2012 R2 with a test box running windows 10.
I am running into a very odd issue. Most of my automated CodedUI tests are running pretty well. However, I have noticed that code changes are not always picked up by TFS when it performs a new build, at least not in the testing code area.
The builds themselves work well, and new code always gets incorporated for those. However, when the latest build triggers a release containing CodedUI tests, those tests do not always grab the latest build.
I have noticed this primarily in my App.config file which contains connection strings that are not being updated. In one case I had three tests that ran apparently successfully, but then they ran again using the values from the old App.config file.
I also have found that changes to the [TestCategory()] attribute are not always picked up either. I use that category to specify which tests I want run in a particular release build. I use variations on the same word for my categories: CodedUI, CodedUIExtended, CodedUIStage. At first I thought the system was doing some sort of StartsWith and picking up the other names, but when I tell it to run CodedUI it is running both the CodedUI and CodedUIStage categories.
[TestCategory("CodedUI"), TestMethod]
public void UI_Login_AdminAuthenticate()
{
...
}
Because the CodedUIStage categories were recently changed and used to be CodedUI that has led me in the direction of suspecting some sort of caching being used in TFS.
Can anyone shed some light on why my category and app.config changes are not being picked up correctly? What is causing this, and could it be happening to the code itself as well when I attempt to deploy a fix/correction?
EDIT:
As suggested, I tried checking the clean option on my TFS build configuration, however it had no affect.
The release, which is triggered immediately upon build does show it is using the newly finished build number.
And looking at the artifacts, the test dll has the current date modified, so it looks like it was just created.
The test.dll.config seems to be the issue. In the artifacts it has a very old Date modified and is not the current version that is checked into TFS. Typically this would feel like a clean issue, however TFS always deletes and re-copies all destination files when doing a build or Release. I have verified that by watching the files being deleted and re-created on the file system during the process.
C:\agent_work\r6\a\ [artifact_name]\bin
EDIT2:
With a little more exploration, the build artifacts are correct. It is when the Release copies those artifacts into the release process that the problem happens.
A week ago I renamed the folder in TFS containing my test project inside the solution. The old folder name is showing up in the artifacts the Release copies into itself. The new folder name is also showing up, which means I now have two dlls, and that is causing problems.
I am not sure where the Release is finding this copy of the old folder. I explicitly deleted it from the Release's copy of the build artifacts and re-ran the build and release and it showed back up.
Thanks to the suggestions from Daniel, I eventually figured out that after creating the artifacts, my build process was then publishing those to a separate place on the file system.
Unfortunately, the Copy and Publish Build Artifacts task does not have a clean feature like the basic Copy Files task does.
As such, whenever files are removed from the build they still exist in that location when the Release process goes to try and grab what it thinks the artifacts are. And so it ends up grabbing extra files.
Manually deleting the old files from that secondary artifact destination location solved the issue.
I'm configuring continuous integration with TFS 2012. I have one problem need solve.
I need to exclude some paths from triggering builds.
For e.g. I have:
$/Project1
$/Project2
And I want that after each check-in of $/Project1 - build has been triggered. And it must build both $/Project1 and $/Project2.
But after checking in $/Project2 I don't want to trigger a build for that Build Definition.
In Source Settings of Build Definition are only functions "Active" and "Cloaked", but it isn't what I need.
Thanks a lot in advance.
P.S. The worth solution is to add the comment ***NO_CI*** on check-in. It will be great if there is some other way.
Based on the comments, this boils down to an X-Y problem. You can't do what you want to do, but the reason you want to do it is because you're trying to solve the wrong problem.
You're running the UI tests at the wrong point in your dev-test cycle. UI tests should not run during a build, they should run after a release. A change to your test project should absolutely result in a build.
Someone is developing UI tests against code that's not yet in source control, which makes no sense. If someone is writing tests against code, the code should be source controlled.
I'm guessing that someone is manually pushing uncommitted code out to a dev server, which is being used by someone else to write tests. Don't do this. Use a real release management solution so that as developers write code, each check-in is automatically deployed to a dev/QA environment. Then the folks writing the UI tests will have something to test against. What's the point in writing tests against code that's in such a state of flux that the developer responsible for it isn't even sure it's worth being source controlled? That just results in spending a lot of time rewriting tests as the code evolves.
Assuming you set everything up properly, every commit of the application code will result in the current set of tests being run against the latest commit. Every commit of the test code will result in the new set of tests being run against the existing application code. The two things (application code and test code) should be coupled, and should always build together.
And one last thing, mostly opinion: UI tests are awful and serve very little utility. They are brittle, slow, and hard to maintain. I have never seen a comprehensive UI test suite actually provide value. UI tests are best served as a small set of post-release smoke tests. Business logic should be primarily unit tested, with a smaller suite of integration tests to back it up.
Introduction
I have a problem with Team Foundation Server Express 2013 on my machine. I have two build definitions on the same controller and agent, both of which run on the same server and the same environment as well.
It should be noted that I already looked at the "similar questions" without any luck. This is clearly not related to the same root cause, and the symptoms are slightly different too.
One of them is a gated check-in build definition, which just compiles everything when commiting to the development branch.
Another is a scheduled build definition, which runs every saturday at 3 AM, building any changes that may have been committed to the main branch since last time.
The gated build definition has a process (which only has minor changes for not running tests and just compiling the code) based on the TfvcTemplate.12.xaml template.
The scheduled build definition process is based on some Azure build definition template that might come from an older version of Visual Studio, possibly based on some Azure template, or maybe the TfvcContinuousDeployment.12.xaml template.
The issue
My gated build definition runs just as expected, without issues. It compiles the full solution, and only passes if the compilation succeeds.
The shceduled build definition however fails compiling (even before it reaches the point where it runs the unit tests). The error I see is as follows.
Obviously this is due to missing fakes assemblies. I tried taking the assemblies and checking them in (which I would rather avoid), only to find that this build definition runs just fine, but not the gated one which ran just fine before.
I thought about just running fakes.exe in the build template to just generate them manually before compiling, but in my initial tests (to see if this theory would even work), it won't even run in the commandline, and outputs some errors and warnings that I don't understand (but are probably not relevant anyway, since I might be running fakes.exe with improper arguments).
Updates
Update #1
It should be noted that I have Visual Studio 2013 Ultimate installed on my build server as well. Both this (and TFS 2013 Express) have Update 3 installed, and the server is fully updated.
I ended up abandoning fakes all together, and implementing Moq instead. Works a lot better, and forces me to abandon shims or moles, which are often considered bad practise anyway.
I have setup a build controller etc and the builds were failing, I have fixed these now and the build failed properly - as in because of an error.
I have fixed the error and checked the code back in but now the code is not being extracted, although sometimes one folder of many is.
I have deleted the code from the build machine and requeued a build but it keeps failing. It complains that it cannot find the solution that I specified as the build solution.
I have checked the check box to build even if nothing has changed.
Have I missed a setting somewhere for extracting the code?
TFS version is 2012 Express
Visual Studio version is 2010 Professional
I had this issue recently with TFS 2012. I think it boils down to this:
In the lastest build definition files, it appears that it performs a Clean task before updating the workspace. This means that if you do something that causes the Clean part of the build to fail, it will never download the new files in order to fix it.
Recently, I was making big changes to my build file and inevitably made a lot of mistakes, I found that if one of these mistakes caused the Clean to break, I had to go onto the Build server and change the file manually to get it working again.
Does this sound like it might be the same issue?
There are several properties in your build definition you can check. I would start with setting the "Clean Workspace" to All to ensure the correct code is being pulled down and built.
There are other checks you can look at as well like the agent set for the build and the "GetVersion" property. Check the below link out. It should be able to help you in more detail.
Define a Build Process that is Based on the Default Template
I recently migrated from VSS to TFS 2010 and I've been absolutely loving it, but there's something I haven't yet been able to get working the way I think it should.
GOALS
I'd like to quickly know when a change to Development breaks a build. If we find out after-the-fact, it's no big deal. Since a lot of check-ins happen throughout the day, we don't want to wait on the build to finish, so it should be asynchronous.
With our Main branch, I'd like to ensure that any time a merge happens into it, we make sure it's not going to break the build. I want immediate feedback on this. The wait time is fine, since we won't be merging into Main often.
CURRENT SETUP
My solution is under a folder called Main. I've made a branch off of that called Development. The workspace I'm working in is tied to the top-level, which includes both Main and Development branches. I tried adjusting my workspace to only point to Development, in case that was the problem. That didn't seem to fix my issue, so I set it back to the way I had it originally -- with both Main and Development.
Within the workspace's Build definitions, I have two definitions defined -- one for the Main branch and another for Development.
The first definition is for building the Main branch. It has a Gated Check-In trigger and "Items to Build" points to the solution file within Main.
The second definition is for building the solution in the Development branch. It has a Rolling Builds trigger and "Items to Build" points to the solution file within Development.
RESULT
Currently, when a change is made in Development and a check-in is performed, the gated check-in to Main is triggered. This not only causes confusion, but it also tends to slow down our overall process.
NEXT STEP
I've looked at some of the TFS guides to branching, general TFS usage, etc. Unfortunately, I haven't yet found a solution to my issue. If you've run across this issue before, I'd appreciate any suggestions you can give.
I'm not 100% sure at the moment, but the build should be triggered by check-ins into the part of the source tree that is covered by the build definition workspace.
So please have a look into the build definition and see what part of the source tree each build definition gets. Your CI build should only get the Dev branch, the gated build should only get the Main branch.