Looking for a little help here. Trying to bootstrap a small side business, and I have never been the DevOps guy. I use the web hosted version Gitlab to store my codebase, but I am unable to use it to act as a repository for docker images that I am creating from that code. The images that I am generating are quite large and exceed the token expiration when I am attempting to push back to the registry from the group gitlab-runner that I have installed on my personal machine. I have an extra machine sitting around, so I installed gitlab-ee and exposed it through a dynamic dns service (NoIP). I then mirrored the repositories that I want to generate images for on my locally hosted gitlab instance. At first, I tried to use a runner that was on the same machine as my gitlab instance, but always failed due to all available memory being consumed and locked up the machine. All in all, gitlab docs pretty much don’t run the runner and instance on the same machine. So, I went back to using the runner I originally used for the web hosted instance, but I am having issues pushing to my local instance. When trying to push to my repository (through the ddns URL), I end up getting a lot of this:
e4fdbd3bf512: Retrying in X seconds
And it eventually times out due to job time limit or token time limit. I am guessing this is due to my connectivity not being great. What I would like to do is have the (installed on a local machine) runner push to the local IP on my network, but I am unsure how to do this with the SSL setup. When trying to login and push in my pipeline, I get the following error:
Error response from daemon: Get "https://xxx.xxx.xxx.xxx:xxxx/v2/": x509: cannot validate certificate for xxx.xxx.xxx.xxx because it doesn't contain any IP SANs
How do I correct this without affecting the https:// SSL that is already setup for when accessing the instance from the DDNS? Appreciate any help you can give me.
I abandoned attempts at getting this to work. Ran through a bunch of scenarios of creating my on CA and trying to create certificates for the IP address and share that with the other machine. Ultimately, gitlab obscures some things with LetsEncrypt. Funny enough it was just a connectivity issue where I was getting timeouts. Ended up hard-lining both machines and getting better throughput. Able to push ~6GB docker images up through the URL.
This was working 1 month back. But recently it is showing following error when I try to either sync or pull from repository.
Error: git.exe pull --progress -v --no-rebase "origin" fatal: unable to access getaddrinfo() thread failed to start
Also I have verified that I am not able to connect github.com. I mean when pinged to github.com it returns no response
I have viewed following threads. Answers / comments given in these threads are not working for me.
*Getting Access Denied while cloning using git clone command
fatal: unable to access <link> getaddrinfo() thread failed to start*
I am using Windows 10 Professional version and Visual Studio 2019
Community Edition.
I am accessing GIT repository from https://xxxx.visaulstudio.com/
Repository URL is HTTPS and not SSH
I have tried following solutions :
Delete credentials from windows credential manager
Uninstall GIT for Windows and installed latest one (2.26.2.1)
Removed GIT folder from location "C:\Program Files (x86)\Microsoft Visual
Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team
Explorer"
Verified Anti Virus (Quick Heal). There is no record or setting for blocking particular domain
5 There is no issue of firewall.
but none of these are working.
I saw following link and same is reported by multiple users. It seems to be issue with compatibility of Git Credential Manager with Azure Devops authentication
https://github.com/microsoft/Git-Credential-Manager-for-Windows/issues
Help is really appreciated.
So I made changes to my rails app on my local machine that were never committed and that I decided to discard. Decided to rollback to most recent commit by deleting my rails app and all its files, and then cloning my most recent commit from my github repository. Unfortunately
git clone https://github.com/my_username/rails_app.git
results in the following error that cannot be found by just googling:
Cloning into 'sample_app2'...
fatal: unable to access 'https://github.com/IPX-/sample_app2.git/': SSLRead() return error -9806
I was recently pushing to the repo earlier today, even set up an SSH key for my this laptop, but all to no avail.
Why is git clone not working for my own repository?
Possible error due to low internet connection. This error occurred when you try to connect to server through ssl and net connection drop immediately. Or some time due to frequent disconnection. YOu can find more about this here http://flummox-engineering.blogspot.in/2014/04/sslread-return-error-9806-server.html
Possible duplicate of SSLRead() error using Git in Mavericks OS X 10.9.2
Anyhow, the answer mentioned here is that you download SSL and install.
When you have already set up SSH then why don't you access your repo over SSH?
You can use user.name#github.com:<your-repo> and it will not ask you for password if you have actually set-up SSH properly!
Enjoy!
I have tfs installed locally on my machine. It used to work fine but for a week or so now, it does not work most of the time.
It is not possible to connect from Visual Studio 2013 now using the team menu item. It is able to connect only once in so many tries.
Git Fetch, Git Pull and Git Push commands from Git Bash take a long time to show the login prompts and sometimes does not even working reporting that it could not connect to localhost:8080
Fetch, Push and Pull from Visual Studio work once in a while.
Connecting with the web always works though sometimes slow.
Git Push with Git Bash and Gui of late gives the error below
POST git-receive-pack (8010 bytes)
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
error: RPC failed; result=7, HTTP code = 0
Pushing to http://localhost:8080/tfs/col1/_git/project1
Everything up-to-date
I've read so many articles and now seem to work. Is there a way to troubleshoot TFS to find out where it is coming from so that it can be corrected.
It turned out this had nothing to do with TFS as generally requests that have to do with local host fails most of the time. It may be an issue with the system when it comes to local host. In any case, I disabled IPv6 and it still didn't work.
What worked however was rawcap. I realized that when I run rawCap to monitor 127.0.0.1, all calls were going through successfully. It appears something it did rectified the issue. Hope it helps someone who also had the same issue with localhost.
Watch for -1 statuses in TFS Activity Log (http://localhost:8080/tfs/_oi). Right click -> Show Detail to get the full exception. Also look for errors in the IIS logs.
I need some advice on how to diagnose a hanging build. It’s only been happening in the last week or two and I have good reason to suspect it’s something that I’ve done recently and not just a coincidence
Setup
TFS 2013
4 machine setup - 2 app tiers (in process of deprecating one of them), 1 sql server, 1 build server running 2 agents.
Build Controller is running on 2nd app tier along with the Job Agent
1st App tier is serving the website (although that machine will soon be shutdown and everything will be passed to the 2nd app tier as the machine is getting old)
Symptoms
All executed builds (doesn’t appear to matter which build process template) never get marked as done, the last step always seems to be the same step “Copy Files to Drop Location”/“Workspace and Copy Files to Drop Location”/”Copy Binaries to drop, Reset the environment” (named differently in each build template)
The files appear to be getting dropped successfully in the build drop folder
Looking at the task manager it appears that all the build processes on the build server are exited (only TFSBuildServiceHost
Builds show their normal steps/logging while executing
Primary app tier has related warnings in the event logs (see warnings below)
Recent Changes
Installed Xamarin Android/iOS on the build server
Installed a few custom built plugins for Job Agent, Message Queue, and Web Services (been using them for years just had them disabled the last few weeks due to a app tier migration)
Installed Tiago’s Task Board Enhancer (again been using this for a long time, just had it disabled recently)
About a month ago we added the 2nd app tier and moved the sql off to another machine
What I’ve Tried
Rebooting both App tiers and build server
Uninstalling Xamarin (although I suspect some parts are still floating around as the Bonjour service appears to still be installed)
Removing the custom plugins
Turned logging diagnostics right up on one of the builds – nothing particularly of interest seems to turn up
Run the Best Practice Analyzer (nothing too unusual shows up)
Multiple build process templates (defaulttemplate, defaulttemplate.11.1, tfvctemplate.12.xaml)
Multiple build definitions
Checked the event logs of both AppTiers and Build server
The Team Foundation service host request monitor has detected the
following condition: Date (UTC): 3/02/2014 12:54:06 a.m. Machine:
CODEBASE Application Domain: /LM/W3SVC/1/ROOT/tfs-1-130357641583538280
Assembly: Microsoft.TeamFoundation.Framework.Server, Version=12.0.0.0,
Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v4.0.30319 Service
Host: 0dc282b5-59a8-4941-b541-a4f7d314cd0f Process Details: Process
Name: w3wp Process Id: 2508 Thread Id: 2504
Detailed Message: A request for service host XXXX has been executing
for 37 seconds, exceeding the warning threshold of 30.
Request details: Request Context Details
Url: /tfs/XXXX/XXXX/_api/_build/stop?__v=4
Method: ApiBuild.stop
Parameters: uri = vstfs:///Build/Build/34064
User Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.102 Safari/537.36
Unique Id: 00000000-0000-0000-0000-000000000000
The Team Foundation service host request monitor has detected the
following condition: Date (UTC): 30/01/2014 11:10:01 p.m. Machine:
CODEBASE Application Domain: /LM/W3SVC/1/ROOT/tfs-1-130355232548668648
Assembly: Microsoft.TeamFoundation.Framework.Server, Version=12.0.0.0,
Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v4.0.30319 Service
Host: 0dc282b5-59a8-4941-b541-a4f7d314cd0f Process Details: Process
Name: w3wp Process Id: 70320 Thread Id: 14540
Detailed Message: A request for service host XXXX has been executing
for 37 seconds, exceeding the warning threshold of 30.
Request details: Request Context Details
Url: /tfs/XXXX/Build/v4.0/BuildService.asmx
Method: StopBuilds
Parameters: uris[0] = vstfs:///Build/Build/34051 uris = Count = 1
User Agent: Team Foundation (devenv.exe, 12.0.21005.1, Premium, SKU:16)
Unique Id: 4d2d3213-fd41-4c4d-8ab0-b87619c96a42
The Team Foundation service host request monitor has detected the
following condition: Date (UTC): 31/01/2014 3:14:17 a.m. Machine:
CODEBASE Application Domain: /LM/W3SVC/1/ROOT/tfs-1-130355232548668648
Assembly: Microsoft.TeamFoundation.Framework.Server, Version=12.0.0.0,
Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v4.0.30319 Service
Host: Process Details: Process Name: w3wp Process Id: 70320
Thread Id: 14540
Detailed Message: There are no active requests for service host XXXX
that exceed the warning threshold of 30.
A quick google suggests upping the timeout in the tfs registry (http://xavierdilipkumar.com/post/2013/07/04/TFS-event-7005-and-7006-warning.aspx) I've tried that and it doesn't appear to change anything.
can you look in the tfs bs logs at
Event Viewer -> Applications and Services Logs -> Microsoft -> Team Foundation Server -> Build-Services -> Operational
these timeouts generally relate to permissions. you should look for TF215106 access denied events. Although the files appear to be there, are they all the current date or are there some with different (older) dates? Also are they any alerts/steps happening when the file drop occurs?
Other than that it could be timing out because one of the dependencies is being used by another service.
You might fire up Sysinternals Process Monitor to see when the processes actually exit and what they were doing (Process Monitor monitors "real-time file system, Registry and process/thread activity").
The best course of action is to call Microsoft Support and open a Service Request. Make sure it gets priority A - your TFS production environment is not working - and be prepared to give them support and access.
The only hint from the log is that call to ApiBuild.stop. It suggests that the build workflow completed, so the code hosting it is calling back to the AT to mark the build completed. As you have no warnings from previous calls, it could be some problem at the database level. You may try activating SQL Tracing but it's not a trivial task, as you should be able to compare the trace with a working one.
Good luck
I'm reluctant to mark this as an answer because I'm not entirely sure why it worked.
Suspecting something was wrong with the build machine I created a new Build Agent on a fresh install - the hanging issue still occurred.
I then added a Build Controller to that machine and noticed that new builds using that controller would complete. This suggested that there was a communication issue between either the BA and the BC, or the BA and the primary AT.
Given that our primary AT had other issues we decided to remove it from the picture, we switched the DNS to point at the second AT and disabled all services on the old primary. Instantly builds started to complete (including the ones that had been stuck for a number of days).
I still don't know which component was broken or why, especially since it worked fine in this configuration for a month prior. I can only assume there was either another change that I am not aware of, or the corruption of the primary AT was causing bigger issues.
We were having the same problem here, the builds were kept open even after successfully passing all workflow stages.
I logged into the build machine and noticed the build controller was "running 6 builds" for some reason, even though there were no builds at all showing in the queue in Visual Studio.
After restarting the controller, the next build worked the first time.
Just wanted to let this one here as a possible answer. I'm not sure yet why the controller had those stuck builds though.
I had this issue when an activity tried to log a huge message in the build log (namely the FxCopCmd activity from the CodePlex TFS Build Extensions project).
The build agent would successfully finish the build but the controller had to chew the huge message into the build log, and it was silently crashing/hanging.
I was able to track the issue down by navigating to C:\Users\[TfsServiceAccount]\AppData\Local\Temp\BuildAgent\[AgentNumber]\Logs\[BuildNumber]\ActivityLog.xml.
The last build message was truncated and by looking at the content, I recognized the FxCop output. In my case, I just set the LogToConsole parameter to False for the FxCop activity in the build process template, and the build completed successfully.
Also appears to happen if the build agent cannot connect to the build controller server on port 9191.
Easily testable with a telnet client.
Appears that my server decided it was on an unknown network and kicked the firewall into overdrive. (The second time I got this issue, not sure if this was the reason I got it the first time but it seems reasonable).