I have a build which hangs in Jenkins. I have deconstructed the build down to a single windows command call, a directory change and a couple of echoes in an effort to isolate the problem. It would appear that the problem lies with a single call to a program executable (now the only call in my build). The build calls the program & then hangs for 30+ minutes (I cancel the build after this time) when it should take less than 1 second. Ordinarily I would be inclined to blame the executable or my misuse thereof, but for the fact that the same call (quite literally copied & pasted) in an ordinary command prompt works perfectly fine. Further muddying the waters is my knowledge of the fact that the build I'm trying implement is working just fine on another Jenkins server I know of, executable and all, and has never had an issue. I'm sorry I can't provide details on the executable in question but it's sensitive information. It may very well be the case that the executable is to blame, but the exact same call in three environments and only one hangs? What do you think?
More info on request.
So there I was, waiting for a reply when I decided to click on one of the "Related" topics on the right hand side. Lo and behold, there it was, a solution to my problem in answer to a different question. In short it goes like this:
Jenkins is a service. Services (on Windows) have a "Log On" account, of which the default appears to be "Local System". This had the effect of forcing (though I don't know why) my program out of quiet mode and thus hanging the build. Changing the "Log On" account to my own seems to have changed the behavioural relationship between Jenkins and the executable so that it now runs smooth and silent.
Related
We have a TFS 2017 build agent executing a Visual Studio Test task to execute our unit tests. This has worked fine for several years, but all of a sudden - without any code changes - the task gets stuck.
All the tests have finished running, we see summary information, and it will sit at what appears to be the place where it would normally publish the results... but then nothing happens. We've waited 12+ hours for it to finish. This step normally takes about 90 minutes.
I've confirmed that the TRX file is being created. It's about 4MB in size. We're running a bit over 3000 unit tests.
I've also tried disabling code coverage and attachments upload inside the test task, but it doesn't appear to make a difference.
Below is a screen cap of the log output when the step is stuck.
Lastly, we have lots of other projects on this server whose tests run / publish fine, as well as TFS Releases for this same build that also run tests (integration/system tests) which work without issue.
UPDATE: We ran this build on a different build server, and it published tests correctly. So this means there is something wrong with this specific build server...
UPDATE 2: So I'm not longer sure what is happening here. The original build server we were having issues on is now working fine with no changes whatsoever. Just started working again. The other build server was working, and then stopped. Same issue. I broke up the 3000+ tests into two steps, roughly 50/50, and that worked a couple of times, but now does not. So this does not appear to be server specific, nor does it appear to be related to the quantity of tests. Debug logging offers nothing useful, as everything seems fine right up until it just stops doing anything after generating the TRX file.
UPDATE 3: Well, it's happening again. I'm not sure how to proceed. I even tried Fiddler on the build box to see if I could catch funky looking traffic, but most of the traffic I'd expect to see I don't. It's like a good chunk of the work isn't being captured (such as source downloads, reporting progress, or test result publishing) by Fiddler. Is it not over HTTP/HTTPS?
This was difficult to figure out due to the quantity of tests we're running, but I was able to narrow it down to a test that launched ping.exe:
[ExpectedException(typeof(TimeoutException))]
[TestMethod]
public void ProcessWillTimeout()
{
const string command = "cmd";
const string args = "/C ping 127.0.0.1 /t";
var externalProcessService = new ExternalProcessService();
externalProcessService.Execute(command, args, TimeSpan.FromMilliseconds(500));
}
For whatever reason, this test was leaving both conhost.exe and ping.exe "orphaned". The fact these processes were not terminating was, for an unknown reason, preventing the tests from publishing their results back to TFS. There is probably something somewhere that waits for the a process to finish and that was never happening.
Indeed, we would see a bunch of conhost.exe and ping.exe processes in both Task Manager and Process Explorer:
You'll notice the tool tip there... "[Error opening process]". I couldn't even use Process Explorer to kill these processes - although Task Manager could. Sure enough, when I killed them, the TFS build task would immediately resume and finish publishing results.
So there is clearly some kind of bug in that ExternalProcessService code we were testing (despite carefully having a finally block that terminated the process), but we are at least able to have our build tests run again without issue.
Suggest you abandon this build and trigger it again. To narrow down if this issue could be reproduced stably.
According to your description, all other builds work properly. And it worked fine for several years. All tests pass, the test report is written, but just the task hangs. Please double check if some other processes might possibly not be properly closing down.
Besides use another build agent to test again. Also try to create a newly build definition with the same settings, trigger that definition, this may do the trick.
Moreover, you could enable verbose logging for trouble shooting. To do this simply adding a build variable named system.debug and setting its value to 'true', this will contain a more detail log info.
I'm building an iOS app that communicates with a server. We have a test / staging server, a production server and each dev has a local instance of the server for development.
I've added some simple logic which configures the address of the server depending on whether we're running a TestFlight build, an App Store build or a debug build (for development). For the development build, the app tries to hit localhost, which is all well and good if we're running on the Simulator, but not so great if we're running on device.
I'm aware of ngrok, which is a possible solution, but since the exposed URL is partially randomly generated (for the free version at least), it's not a great fit. I was thinking that a workable approach for development could be to check the name of the development machine at compile time and insert this value. But I'm not sure how to achieve this, if it's possible at all. I remember doing compile time variable filtering using ant / maven and environment property files back in my Java days, but I'm wondering if there's a fairly straightforward way to achieve this in Xcode.
Can anybody shed any light on this?
So I carried on digging, and went with the following solution. Elements of this have been touched upon in numerous other posts here.
I added a new header file called HostNameMacroHeader.h to my
project.
I added a 'Run Script' phase to my build, before the
'Compile Sources'phase. The script contains the following:
echo "//***AUTOGENERATED FILE***" > ${SRCROOT}/MyAppName/HostNameMacroHeader.h
echo "#define BUILD_HOST_NAME #\"`hostname`\"" >> ${SRCROOT}/MyAppName/HostNameMacroHeader.h
Then in my implementation, where I want to use the server address, I use the generated BUILD_HOST_NAME macro.
It's a somewhat hacky solution, but it does the job for now. Suggestions and cleaner versions are welcome.
I have problems with creating a job in jenkins. At the "Build" step I select Build "execute windows batch command", where I want to start a browser such as opera, google chrome portable or firefox portable, the command never terminates.View screenshot
For IE, firefox and chrome I do not encounter any problems
If someone has a solution. thank you in advance
What do you expect? You have launched an executable that shows a form, albeit in a session that can never have a console session attached.
Jenkins, by design, will sit there until the executable terminates - which generally happens when the form is closed by a user interacting with it. Since there is no GUI, this process can only ever terminate if someone kills it with Task Manager.
What are you actually trying to achieve? There's almost certainly a reasonable way to do it.
I've used TFS Power Tools on a daily basis for several months. I particularly depend on Windows Shell Extension.
For about a week now, two issues have come up:
Issue 1
Every time I enter a directory under TFS control, I have to right-click twice to see the TFS context menu
Issue 2
My pretty TFS Green indicators on files/folders are gone:
I've downloaded and reinstalled the Power Tools, but I still have these two issues.
Any thoughts on how to fix this?
**UPDATE**
Dave gave me the answer via link. Just in case that link goes bad, here is the answer:
Open the registry
Go to : HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\explorer \ShellIconOverlayIdentifiers
Prefix any "Tfs" folder with numbers (“1TfsOverlayAdd” , “2TfsOverlayEdit” etc.). This allows them to take priority over other icon overlays.
Restart the machine or kill explorer.exe and restart it
As far as the delay issue is concerned, I'm not experiencing that this morning; which is odd. I question that the solution for the overlay was also the solution for the right-clicking.
Everything's back to normal now so I am good to go.
The missing green icons can be explained here. It is because Windows has a limit of overlay icons it supports. The details on how to fix it are in that blog post.
How long are you waiting after entering a folder before you perform the first Right Click? It sometimes takes a while for the TfsComProviderSvr.exe to check if the local folder is a workspace and register the shell extension.
When connecting to a remote TFS 2010 installation via my Visual Studio 2010, Process tab of Build Definition freezes. Everything seems to be disabled. Is there any work-around?
UPDATE 1:
By freezing I mean:
I can navigate to all tabs: General, Trigger, Workspace, Build Defaults, Process and Retention Policy. Except for Process tab, they all are working as excpected. When I click Process tab, it is showed, but none of parameters are shown, and the tab itself is in gray. Actually nothing could be clicked. It is like that Visual Studio is waiting for something to be completed or loaded.
UPDATE 2:
I have explored that when I leave Visual Studio alone for a long time about 20 or 30 minutes, the Process tab goes normal. That means all parameters appears and can be edited. That seems downloading or getting something from server causes this very very long delay.
You might want to check out exactly what rights you have on that server with regards to administrating builds.
http://intovsts.net/2009/09/03/more-fine-grained-permissions-in-tfs2010/ and
http://msdn.microsoft.com/en-us/library/ms252587.aspx
I think one of the main ones is "edit build definition"
If you don't have rights, then there won't be a workaround.
UPDATE:
The only things I can really think of at this point is to put a sql monitor on the TFS Database and possibly, watch the network traffic to see how the network interface is being used.
It's possible that something is wrong on the database server causing a huge amount of load. The other possibility is that the network is badly configured and running at 1Mb instead of 100Mb+.
Beyond those, I'd say call MS Support and open a trouble ticket. This is by no means normal and is likely indicative of some underlying hardware or installation issue.
Afsharm,
Happend with me too. This happened when i had the build defination open on one VS instance in edit mode and i opened it for editing on another pc. After a while it errored saying "TFS 20391: Locked for blah blah..." I had to close my visual studio instance on both the machines and restart, it worked there after. These are very occassional cases and very hard to replicate. I would suggest restarting your visual studio session and trying again.
PS - Worth checking if some one else have the process template open for edit as well?
HTH.
Cheers, Tarun