I am using TFS 2017 Update 2 Release processing. I have a functioning deploy process that works within a domain (it runs successfully against 10 different deployment environments)... and now I need to deploy into a different environment, which lives in a different A/D domain.
Unfortunately, the domain trust is one way between the domains - and the destination domain ("Production") does not trust the domain I am installing from ("Dev")
The problem I'm seeing seems to be the infamous "double hop" credential problem.
My TFS app tier can see (and trigger activity on) the release server running TFS vNext Agent 2.117.2 Futher, I can execute inline PowerShell, and locally hosted PowerShell scripts on the release server just fine.
Howerver, as soon as I try to access a PowerShell script not on the release server (be it in the Production domain with the release server, or in the Dev domain) I get an error:
2018-02-13T19:03:32.6611149Z ##[error]. : AuthorizationManager check failed.
At line:1 char:3
+ . '\\unc\path\to\share\TFSScripts\Emit-Variables2. ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : SecurityError: (:) [], PSSecurityException
+ FullyQualifiedErrorId : UnauthorizedAccess
The account running the TFS release service has been confirmed to have access to the script file when running from the desktop of the release server, so access should not be an issue.
Further testing of the issue has identified that if we manually create a PSSession using -Authorization CredSSP and pass a credentials object we can successfully access the off server resources.
However, I can see no way to configure TFS to use CredSSP as the authorization mechanism.
The servers involved are W2K8R2 - so we cant use the constrained delegation functionality that W2K12 introduced. We have also tried SPNs with similar unsuccessful results. Kerberos has been forced to use TCP by setting the max packet size to 0 (thus also preventing fragmented UDP packets and related problems). Our max Kerberos packet size is set to 48000.
In the ultimate end state, The TFS App server, and all the TFS artifacts and release scripts will sit in the "dev" domain on one side of a firewall... and the production release server, and a set of servers to release to will exist in the "production" domain, on the other side of a firewall
CredSSP seems to be the only way to make this work - but I see no way for TFS to be configured for it.
This can't be a unique problem. Can someone provide some insight on how to get around this?
Sorry it's not able to force TFS to use CredSSP when accessing network resources. And on configuration of TFS to use CredSSP as the authorization mechanism
You must manually enable CredSSP in powershell.
Another way take a look at this solution, which may do the trick: TFS2015 Release Management: Deploying to an untrusted domain by having the deployment agent run under a shadow account.
Related
I've been writing an EventHubs message processor that just connects to EventHubs and processes messages on the EventHub. I've been developing in Visual Studio on Windows using .NET 6. Things work as expected on Windows; I can:
Connect to EventHubs
Receive messages
Do the message processing I want
Great. I then wanted to scale my message processor horizontally and decided that I would Dockerize it, and since .NET 6 runs on Linux, I would cross-compile it for Linux and eventually deploy multiple instances of my message processor on Docker Desktop as a next step. I eventually want to stick it on Kubernetes to scale up by an order of magnitude or two.
It was easy to Dockerize my Project in Visual Studio. I simply right-clicked the Project and selected Add -> Docker Support. Visual Studio detected I had Docker Desktop installed and generated all the config files I needed, and added an appropriate build configuration so that I could compile a binary, build a Docker image with it, and automatically deploy it to my local Docker Desktop instance.
.NET 6 also compiled without errors, which was great. However, when my container spins up, I get hit with the following runtime error:
System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: PartialChain
and there is a stack trace (omitted here for brevity) stemming from something in the EventHubs processor library:
<...many layers...> at Azure.Messaging.EventHubs.Primitives.EventProcessor-1.RunProcessingAsync(CancellationToken cancellationToken)
I am correctly passing my EventHubs connection string to my container, but what I surmise is that my container is missing an SSL certificate or has a misconfigured SSL certificate. I suppose Visual Studio has helpfully silently gone ahead and installed a development certificate when I developed my message processor on Windows so that EventHubs connections "just work" in my development environment, but that SSL certificate is not available to my container, since it isn't part of the build output.
I know I probably should be using Azure key vault or whatever secret management service they provide, but how else can I resolve this SSL certificate issue as quickly or painlessly as possible? It would be nice if I can just keep my connection string in my appsettings.json (It's fine. Toy project, only using Azure free credits anyway.)
The easiest way forward would be to register a handler that participates in certificate validation and can, if desired, override normal handling and force acceptance. This, of course, comes with the warning that you're bypassing standard security checks and may be putting your network and host in danger.
You don't mention which client you're using, but each takes a set of options in their constructor. The options for each type have a member named ConnectionOptions which returns an EventHubsConnectionOptions instance that allows you to register a CertificateValidationCallback.
The Event Hubs Influencing SSL certificate validation sample demonstrates how to use it. More information is also available in the .NET documentation for RemoteCertificateValidationCallback.
I've verified that Web Deploy works (using NTLM authorization) when I fire it from Visual Studio on my local machine. Now I want my build server to auto-deploy (if appropriate) every night. I'm using Jenkins on the build server, and I've granted the account access in IIS on the remote machine. My parameters to MSBuild are as follows:
/p:DeployOnBuild=true
/p:Configuration=Debug
/p:Platform=x86
/p:PublishProfile=DEV
/p:AuthType=NTLM
/p:AllowUntrustedCertificate=True
/p:Username=
The DEV publish profile specifies my DEV server which uses a self-signed certificate thus necessitating an untrusted certificate. The NTLM and blank username should use the current user/account to connect.
However, the Jenkins' job's MSBuild step fails with this error
msdeploy error ERROR_USER_UNAUTHORIZED: Web deployment task failed. (Connected to the remote computer ("DEV-SERVER") using the Web Management Service, but could not authorize. Make sure that you are using the correct user name and password, that the site you are connecting to exists, and that the credentials represent a user who has permissions to access the site. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_USER_UNAUTHORIZED.)
When I look at the IIS logs on DEV-SERVER, I see the following:
2016-01-06 23:55:10 159.212.19.186 HEAD /msdeploy.axd site=MySite 8172 - 159.212.19.123 - 401 2 5 0
2016-01-06 23:55:10 159.212.19.186 HEAD /msdeploy.axd site=MySite 8172 CO\BUILD-SERVER$ 159.212.19.123 - 401 2 64 78
I was expecting to see CO\jenkins, the account Jenkins is running under, instead of CO\BUILD-SERVER$. (And what's with the $ on the end?) Am I correct in thinking the wrong account is being used? What do I need to do to get this working?
CO\BUILD-SERVER$ is the machine account of your build server.
If you have a slave running on that machine, is it running as a windows service? If so it's probably running as "System"
Also re Selenium tests, if the tests are running on the build server then the service may need to set to run interactively so that the tests can run against a UI.
We have a solution consisting of several class libraries, and a Web
Application Project. We are using TFS 2012 with Team Build. The solution
compiles correctly on the build server.
I am currently trying to do this via MSBuild Arguments.
/p:DeployOnBuild=True /p:DeployTarget=MsDeployPublish
/p:CreatePackageOnPublish=False /p:MSDeployPublishMethod=RemoteAgent
/p:MsDeployServiceUrl=https://testWebServer:8172/MsDeploy.axd?site=direct /p:AllowUntrustedCertificate=True
/p:DeployIisAppPath="direct"
/p:AuthType=NTLM
The solution builds but does not deploy. I get the following error message:
msdeploy error ERROR_DESTINATION_INVALID: Web deployment task failed.
( Could not connect to the remote computer ("https"). Make sure that
the remote computer name is correct and that you are able to connect
to that computer. Learn more at:
http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_DESTINATION_INVALID.)
[C:\Builds\1\ProjectName\Solution General Build\Sources\Temp
Source\ProjectName\Solution\Project.csproj]
Is there another argument I should be passing to specify the server? I did
not intend for https to be the server name... I have tried omitting the
https:// to no avail, error is the same, so it is getting the value from
somewhere.
I have tried this with the following values for MsDeployServiceUrl:
https://testWebServer:8172/MsDeploy.axd?site=direct
https://testWebServer:8172/MsDeploy.axd
"https://testWebServer:8172/MsDeploy.axd?site=direct"
https://192.168.X.X:8172/MsDeploy.axd?site=direct
"https://192.168.X.X:8172/MsDeploy.axd?site=direct"
testWebServer:8172/MsDeploy.axd?site=direct
Update
Alright, the following is at least connecting:
/p:MsDeployServiceUrl=testWebServer
I have seen numerous posts concerning that particular argument, and almost invariably they are a URL, not just a hostname (the ones that appear to be a hostname I thought were just written that way for brevity).
I am now, however, faced with a new problem. I have made the Build Service Account (domain account) local admin on the webserver, and I am getting msdeploy error ERROR_USER_NOT_ADMIN as well as an Audit failure in the Security log.
Resolution
These are the MSBuild arguments I am currently going with.
/p:DeployOnBuild=True /p:DeployTarget=MsDeployPublish /p:CreatePackageOnPublish=False /p:MSDeployPublishMethod=WMSvc /p:MsDeployServiceUrl="https://SERVER:8172/MsDeploy.axd" /p:AllowUntrustedCertificate=True /p:DeployIisAppPath="siteName"
I am now getting ERROR_USER_UNAUTHORIZED. Apparently I have either not set up the delegation correctly or the IIS Manager User I have created is somehow incorrect. Regardless that will go in a different post if necessary.
What is the Server and IIS version, you are using?
IIS 6 uses Web Deployment Agent Service (MsDepSvc), whereas IIS 7 usually uses Web Management Service (WMSvc) which have different URLs (besides, you have to be an admin on the target server to execute MsDepSvc.
Can you try specifying
/P:MSDeployPublishMethod=WMSvc
Based on this article from Troy Hunt, Web Management Service (WMSvc) is using
.axd
URLs (the one you specify), whereas you are trying to force it use RemoteAgent publish method which seems to be inconsistent.
See this article for complete set of differencies between WMSvc and RemoteAgent publish methods.
I had a similar issue. To resolve the issue I tried the following steps:
As it was a hosted server we had to make sure that the port 8172 was open (obviously).
Creating a new login and set this up in IIS -> Deploy -> Configure -> Configure Web Deploy Publishing on the target server. I made sure that the password didn't have any spaces in to avoid the quotes issue just to be sure.
Actually running a manual deployment from the build server.
Finally specifying an IP address in the MSDeployServicerl:
/p:MsDeployServiceUrl=xxx.xxx.xxx.xxx:8172/msdeploy.axd
None of the web site names worked for me either. None of my parameters had quotes in. Of course if you leave a space in incorrectly in one of your parameters you will get the error:
MSBUILD : error MSB1008: Only one project can be specified
BEFORE: I had a TFS 2010 on a temporary test environment set up with a project and I had web users and everything worked great.
NOW: I've installed it on a permanent environment (same O/S, domain, everything) but any permissions I set no longer seem to have any effect.
It seems only the service account can access any features.
Authentication is NTLM.
Any network users I give access to are either being asked for their credentials to connect to the server and being rejected regardless (they can connect to the default IIS fine) or they get:
500 - Internal server error.
There is a problem with the resource you are looking for, and it cannot be displayed.
Ridiculous, but the problem is that the new install was on the E: not the C: so the local NETWORK SERVICE account (that I use as a service account for TFS) did not have access to the files/folders under \Program Files\Microsoft Team Foundation Server 2010\
Is anyone successfully using MsDeploy for deploying windows services with a preSync runCommand? I've got it working using an Administrator account, but can't for the life of me get it working on a standard user account. Unfortunately I can't use integrated authentication (we're deploying to an external box), and the thought of our Administrator password sitting in plaintext in logs on our build server doesn't exactly make me feel too comfortable. For that matter, neither does any user credentials - but I can't see a way around that.
The command I'm using is this:
"tools/deploy/msdeploy.exe" -verb:sync
-preSync:runCommand="tools\Deploy\PreSyncCommand.cmd",waitInterval=30000
-source:dirPath="C:\BuiltSourcePath"
-dest:computerName=https://server:8172/msdeploy.axd?site=dummysitename,userName=service-deploy,password=service-deploy-pass,authType=basic,dirPath="C:\DeployPath\"
-allowUntrusted
with rules set up in IIS for the dummy site to allow the authentication for the service-deploy windows account, with contentPath and runCommand permissions (for the moment set to C:\ as it's not entirely clear whether this needs to be set to the temporary path that MsDeploy streams to, or the deployment path?). The service-deploy account also has full control of the target directory. I get the following back:
Performing '-preSync'...
Info: Using ID '7a7d34a1-b5d8-49f1-960a-31c9cf825868' for connections to the remote server.
Info: Using ID '4d0b910c-aca4-4640-84bd-3597d22d99d1' for connections to the remote server.
Info: Updating runCommand (C:\TeamCity\buildAgent\work\aec989676b349656\tools\De
ploy\PreSyncCommand.cmd).
Warning: Access is denied.
Warning: The process 'C:\Windows\system32\cmd.exe' (command line '/c "C:\Windows
\ServiceProfiles\LocalService\AppData\Local\Temp\giz2t0kb.0ay.cmd"') exited with
code '0x1'.
This happens even if the contents of PreSyncCommand.cmd is blank. The same command runs fine if I pass in Administrator credentials. I've tried using ProcessMonitor to check if anything's being denied access but can't see any - so I'm guessing it's still a MsDeploy authentication rule. There's nothing in WmSvc.log (debugging is enabled), nor in the event log.
Any ideas? Thanks!
Since you're using Web Deploy via WmSvc, you need to setup appropriate delegation rules on the destination server:
Within IIS Manager, open the "Management Service Delegation" feature. Add a new rule which at least specifies the runCommand provider. In the Run As section, choose Specific User and provide credentials for a local administrator account on that machine. This is the identity under which your runCommand scripts will be executed. Finally, the user which you're specifying for the destination dirPath provider needs to be added to the delegation rule.
That allows you to invoke a deployment using a non-privileged account, and yet have it executed on the target machine under administrative credentials.
More information on IIS feature delegation: http://learn.iis.net/page.aspx/516/configure-the-web-deployment-handler/