Migrating from one service to IBM Cloud for Redis.
I cannot find the correct configuration to connect using TLS. Everything I find on this is related to Heroku. and it ignores verifying the TLS/SSL connection.
I cannot find how to configure our Sidekiq/Redis to connect.
I do have a certificate from the IBM Cloud dashboard and I suspect I have to pass that along somehow.
Configure the Sidekiq.yml like this
:redis:
:url: "rediss://:< PWD >#< DB Name >:< PORT >/0"
:namespace: "app"
:ssl_params:
ca_file: 'path/to/cert'
I keep getting back the error Redis::CommandError - WRONGPASS invalid username-password pair or user is disabled.: however using these same credentials in the migration script I am able to connect to the DB, so the credentials are ok, I think it is not including the certificate correctly and I cannot find the correct way to do this
The sidekiq.yml configuration looks good to me, just make sure this has correct complete path
ca_file: 'path/to/cert'
and change the redis url to
:url: "rediss://< PWD >#< DB Name >:< PORT >/0"
further info you can read from here for TLS secured connection.
I'm not familiar with sidekiq.yml. But I've configured redlin with redis using a python script you can find here: https://github.com/IBM-Cloud/vpc-transit/blob/master/py/test_transit.py. Maybe the configuration is similar.
The relevant code is:
def vpe_redis_test(fip, resource):
"""execute a command in fip to verify postgresql is accessible"""
redis = resource["key"]
credentials = redis["credentials"]
cert_data = credentials["connection.rediss.certificate.certificate_base64"]
cli_arguments = credentials["connection.cli.arguments.0.1"]
command = f"""
#!/bin/bash
set -ex
if [ -x ./redli ]; then
echo redli already installed
else
curl -LO https://github.com/IBM-Cloud/redli/releases/download/v0.5.2/redli_0.5.2_linux_amd64.tar.gz
tar zxvf redli_*_linux_amd64.tar.gz
fi
./redli \
--long \
-u {cli_arguments} \
--certb64={cert_data} << TEST > redis.out
set foo working
I am trying to write .netCore 3.1 API in an Ubuntu Linux container that runs the equivalent of this Curl command.
WORKING LINUX CONTAINER CURL COMMAND:
curl --cacert /etc/root/trustedroot.crt --cert /etc/mutualauth/tls.crt --key /etc/mutualauth/tls.key
--header "SOAPAction:actionName" --data #test.xml https://this.is.the/instance --verbose
Enter PEM pass phrase: *****
<Success...>
We use Windows development laptops so everything starts with Windows.
So far, I have the following HttpClientHandler that my HttpClient is using on a Windows development machine. This code works on Windows with the cert in my local machine and current user personal stores and does not work in Linux:
WORKING WINDOWS HTTPCLIENTHANDLER CODE:
X509Store store = new X509Store(StoreName.My, StoreLocation.CurrentUser);
store.Open(OpenFlags.ReadOnly);
try
{
var cert = store.Certificates.Find(X509FindType.FindByThumbprint, "<<cert thumbprint here>>", true);
var handler = new HttpClientHandler
{
ClientCertificateOptions = ClientCertificateOption.Manual,
SslProtocols = SslProtocols.Tls12,
AllowAutoRedirect = false,
AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip
};
handler.ClientCertificates.Add(cert[0]);
}
catch (Exception e)
{
//Handle errors
}
finally
{
store.Close();
}
The cert I imported was .PFX format so as I understand it, the password went in at the time of import and the code for Windows doesn't need to be concerned with it.
The Curl command mentioned above works from the container. So by that logic, if coded or configured properly, the code should be able to do the same thing. As I see it, the Curl command shown above contains four elements that I need to account for in my HttpClientHandler somehow:
The Trusted Root(CA) Certificate: /etc/root/trustedroot.crt
The TLS Certificate: /etc/mutualauth/tls.crt
The Private Key - /etc/mutualauth/tls.key
The PEM Passphrase
I have been reading into this for a couple of months now and have seen various articles and stack overflow posts but there is a staggering amount of variables and details involved with SSL and I cant find anything that directly addresses this in a way that makes sense to me with my limited understanding.
I also have the option of running a Linux script at the time of deployment to add different/other formats of certs/keys to the stores/filesystem in the container. This is how I get the certs and keys into the container in the first place, so I have some control over what I can make happen here as well:
LINUX CONFIG SCRIPT:
cp /etc/root/trustedroot.crt /usr/share/ca-certificates
cp /etc/mutualauth/tls.crt /usr/share/ca-certificates
cp /etc/mutualauth/tls.key /etc/ssl/private
echo "trustedroot.crt" >> /etc/ca-certificates.conf
echo "tls.crt" >> /etc/ca-certificates.conf
update-ca-certificates
dotnet wsdltest.dll --environment=Production --server.urls http://*:80
I do not believe I can get the binary .PFX file into the container due to security policies and limitations, but I definitely can get its string encoded cert and key formats into the container.
...so if there is a way of using different styles of certs that I can extract from the .PFX or specifying password and cert when the server 'spins up' to make my code not require a password, that would work too - I might just be missing something basic in the Linux config.
Would anyone be so kind as to point me in the proper direction to find out how I can uplift my HttpClientHandler code OR Linux config to be able to make this API call? Any ideas are welcome at this point, this has been a thorn in my side for a long time now... Thank you so much!
This was not the right approach.
The correct approach was an NGINX reverse proxy terminating mutual auth TLS so that Dotnetcore doesn't have to.
Save yourself some time and go NGINX!. :D
I'm searching for the way to get $ go get work with private repository, after many google try.
The first try:
$ go get -v gitlab.com/secmask/awserver-go
Fetching https://gitlab.com/secmask/awserver-go?go-get=1
https fetch failed.
Fetching http://gitlab.com/secmask/awserver-go?go-get=1
Parsing meta tags from http://gitlab.com/secmask/awserver-go?go-get=1 (status code 200)
import "gitlab.com/secmask/awserver-go": parse http://gitlab.com/secmask/awserver-go?go-get=1: no go-import meta tags
package gitlab.com/secmask/awserver-go: unrecognized import path "gitlab.com/secmask/awserver-go
Yep, it did not see the meta tags because I could not know how to provide login information.
The second try:
Follow https://gist.github.com/shurcooL/6927554. Add config to .gitconfig.
[url "ssh://git#gitlab.com/"]
insteadOf = https://gitlab.com/
$ go get -v gitlab.com/secmask/awserver-go --> not work
$ go get -v gitlab.com/secmask/awserver-go.git --> work but I got src/gitlab.com/secmask/awserer-go.git
Yes it work but with .git extension with my project name, I can rename it to original but do it everytime $ go get is not so good, is there an otherway?
You have one thing to configure. The example is based on GitHub but this shouldn't change the process:
$ git config --global url.git#github.com:.insteadOf https://github.com/
$ cat ~/.gitconfig
[url "git#github.com:"]
insteadOf = https://github.com/
$ go get github.com/private/repo
For Go modules to work (with Go 1.11 or newer), you'll also need to set the GOPRIVATE variable, to avoid using the public servers to fetch the code:
export GOPRIVATE=github.com/private/repo
The proper way is to manually put the repository in the right place. Once the repository is there, you can use go get -u to update the package and go install to install it. A package named
github.com/secmask/awserver-go
goes into
$GOPATH/src/github.com/secmask/awserver-go
The commands you type are:
cd $GOPATH/src/github.com/secmask
git clone git#github.com:secmask/awserver-go.git
I had a problem with go get using private repository on gitlab from our company.
I lost a few minutes trying to find a solution. And I did find this one:
You need to get a private token at:
https://gitlab.mycompany.com/profile/account
Configure you git to add extra header with your private token:
$ git config --global http.extraheader "PRIVATE-TOKEN: YOUR_PRIVATE_TOKEN"
Configure your git to convert requests from http to ssh:
$ git config --global url."git#gitlab.mycompany.com:".insteadOf "https://gitlab.mycompany.com/"
Finally you can use your go get normally:
$ go get gitlab.com/company/private_repo
For people using private GitLabs, here's a snippet that may help: https://gist.github.com/MicahParks/1ba2b19c39d1e5fccc3e892837b10e21
Also pasted below:
Problem
The go command line tool needs to be able to fetch dependencies from your private GitLab, but authenticaiton is required.
This assumes your private GitLab is hosted at privategitlab.company.com.
Environment variables
The following environment variables are recommended:
export GO111MODULE=on
export GOPRIVATE=privategitlab.company.com # this is comma delimited if using multiple private repos
The above lines might fit best in your shell startup, like a ~/.bashrc.
Explanation
GO111MODULE=on tells Golang command line tools you are using modules. I have not tested this with projects not using
Golang modules on a private GitLab.
GOPRIVATE=privategitlab.company.com tells Golang command line tools to not use public internet resources for the hostnames
listed (like the public module proxy).
Get a personal access token from your private GitLab
To future proof these instructions, please follow this guide from the GitLab docs.
I know that the read_api scope is required for Golang command line tools to work, and I may suspect read_repository as
well, but have not confirmed this.
Set up the ~/.netrc
In order for the Golang command line tools to authenticate to GitLab, a ~/.netrc file is best to use.
To create the file if it does not exist, run the following commands:
touch ~/.netrc
chmod 600 ~/.netrc
Now edit the contents of the file to match the following:
machine privategitlab.company.com login USERNAME_HERE password TOKEN_HERE
Where USERNAME_HERE is replaced with your GitLab username and TOKEN_HERE is replaced with the access token aquired in the
previous section.
Common mistakes
Do not set up a global git configuration with something along the lines of this:
git config --global url."git#privategitlab.company.com:".insteadOf "https://privategitlab.company.com"
I beleive at the time of writing this, the SSH git is not fully supported by Golang command line tools and this may cause
conflicts with the ~/.netrc.
Bonus: SSH config file
For regular use of the git tool, not the Golang command line tools, it's convient to have a ~/.ssh/config file set up.
In order to do this, run the following commands:
mkdir ~/.ssh
chmod 700 ~/.ssh
touch ~/.ssh/config
chmod 600 ~/.ssh/config
Please note the permissions on the files and directory above are essentail for SSH to work in it's default configuration on
most Linux systems.
Then, edit the ~/.ssh/config file to match the following:
Host privategitlab.company.com
Hostname privategitlab.company.com
User USERNAME_HERE
IdentityFile ~/.ssh/id_rsa
Please note the spacing in the above file matters and will invalidate the file if it is incorrect.
Where USERNAME_HERE is your GitLab username and ~/.ssh/id_rsa is the path to your SSH private key in your file system.
You've already uploaded its public key to GitLab. Here are some instructions.
All of the above did not work for me. Cloning the repo was working correctly but I was still getting an unrecognized import error.
As it stands for Go v1.13, I found in the doc that we should use the GOPRIVATE env variable like so:
$ GOPRIVATE=github.com/ORGANISATION_OR_USER_NAME go get -u github.com/ORGANISATION_OR_USER_NAME/REPO_NAME
Generate a github oauth token here and export your github token as an environment variable:
export GITHUB_TOKEN=123
Set git config to use the basic auth url:
git config --global url."https://$GITHUB_TOKEN:x-oauth-basic#github.com/".insteadOf "https://github.com/"
Now you can go get your private repo.
If you've already got git using SSH, this answer by Ammar Bandukwala is a simple workaround:
$ go get uses git internally. The following one liners will make git and consequently $ go get clone your package via SSH.
Github:
$ git config --global url."git#github.com:".insteadOf "https://github.com/"
BitBucket:
$ git config --global url."git#bitbucket.org:".insteadOf "https://bitbucket.org/"
I came across .netrc and found it relevant to this.
Create a file ~/.netrc with the following content:
machine github.com
login <github username>
password <github password or Personal access tokens >
Done!
Additionally, for latest GO versions, you might need to add this to the environment variables GOPRIVATE=github.com
(I've added it to my .zshrc)
netrc also makes my development environment setup better as my personal github access for HTTPS is been configured now to be used across the machine (just like my SSH configuration).
Generate GitHub personal access tokens: https://github.com/settings/tokens
See this answer for its use with Git on Windows specifically
Ref: netrc man page
If you want to stick with the SSH authentication, then mask the request to use ssh forcefully
git config --global url."git#github.com:".insteadOf "https://github.com/"
More methods for setting up git access: https://gist.github.com/technoweenie/1072829#gistcomment-2979908
That looks like the GitLab issue 5769.
In GitLab, since the repositories always end in .git, I must specify .git at the end of the repository name to make it work, for example:
import "example.org/myuser/mygorepo.git"
And:
$ go get example.org/myuser/mygorepo.git
Looks like GitHub solves this by appending ".git".
It is supposed to be resolved in “Added support for Go's repository retrieval. #5958”, provided the right meta tags are in place.
Although there is still an issue for Go itself: “cmd/go: go get cannot discover meta tag in HTML5 documents”.
After trying multiple solutions my problem still persisted. The final solution after setting up the ~/.netrc and SSH config file, was to add the following line to my ~/.bash_profile
export GOPRIVATE="github.com/[organization]"
I have created a user specific ssh-config, so my user automatically logs in with the correct credentials and key.
First I needed to generate an key-pair
ssh-keygen -t rsa -b 4096 -C "my#email.here"
and saved it to e.g ~/.ssh/id_my_domain. Note that this is also the keypair (private and public) I've connected to my Github account, so mine is stored in~/.ssh/id_github_com.
I have then created (or altered) a file called ~/.ssh/config with an entry:
Host github.com
HostName github.com
User git
IdentityFile ~/.ssh/id_github_com
On another server, the "ssh-url" is admin#domain.com:username/private-repo.git and the entry for this server would have been:
Host domain.com
HostName domain.com
User admin
IdentityFile ~/.ssh/id_domain_com
Just to clarify that you need ensure that the User, Host and HostName is set correctly.
Now I can just browse into the go path and then go get <package>, e.g go get main where the file main/main.go includes the package (from last example above) domain.com:username/private-repo.git.
For me, the solutions offered by others still gave the following error during go get
git#gl.nimi24.com: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
What this solution required
As stated by others:
git config --global url."git#github.com:".insteadOf "https://github.com/"
Removing the passphrase from my ./ssh/id_rsa key which was used for authenticating the connection to the repository. This can be done by entering an empty password when prompted as a response to:
ssh-keygen -p
Why this works
This is not a pretty workaround as it is always better to have a passphrase on your private key, but it was causing issues somewhere inside OpenSSH.
go get uses internally git, which uses openssh to open the connection. OpenSSH takes the certs necessary for authentication from .ssh/id_rsa. When executing git commands from the command line an agent can take care of opening the id_rsa file for you so that you do not have to specify the passphrase every time, but when executed in the belly of go get, this did not work somewhy in my case. OpenSSH wants to prompt you then for a password but since it is not possible due to how it was called, it prints to its debug log:
read_passphrase: can't open /dev/tty: No such device or address
And just fails. If you remove the passphrase from the key file, OpenSSH will get to your key without that prompt and it works
This might be caused by Go fetching modules concurrently and opening multiple SSH connections to Github at the same time (as described in this article). This is somewhat supported by the fact that OpenSSH debug log showed the initial connection to the repository succeed, but later tried it again for some reason and this time opted to ask for a passphrase.
However the solution of using SSH connection multiplexing as put forward in the mentioned article did not work for me. For the record, the author suggested adding the collowing conf to the ssh config file for the affected host:
ControlMaster auto
ControlPersist 3600
ControlPath ~/.ssh/%r#%h:%p
But as stated, for me it did not work, maybe I did it wrong
After setting up GOPRIVATE and git config ...
People may still meeting problems like this when fetching private source:
https fetch: Get "https://private/user/repo?go-get=1": EOF
They can't use private repo without .git extension.
The reason is the go tool has no idea about the VCS protocol of this repo, git or svn or any other, unlike github.com or golang.org them are hardcoded into go's source.
Then the go tool will do a https query before fetching your private repo:
https://private/user/repo?go-get=1
If your private repo has no support for https request, please use replace to tell it directly :
require private/user/repo v1.0.0
...
replace private/user/repo => private.server/user/repo.git v1.0.0
https://golang.org/cmd/go/#hdr-Remote_import_paths
first I tried
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
but it didn't worked for my local.
I tried
ssh -t git#github.com
and it shows my ssh is fine.
finally, I fix the problem to tell the go get to consider all as private and use ssh instead of HTTPS.
adding export GOPRIVATE=*
Make sure you remove your previous gitconfigs, I had the same issue.
Previously I executed gitconfig whose token was expired, when you execute the command next time with new token make sure to delete previous one.
For standalone/final repos, an as a quick fix, why don't just to name the module within the go.mod as a package using your company's domain ... ?
module go.yourcompany.tld/the_repo
go.yourcompany.tld don't even have to exist as a valid (sub)domain...
Also, in the same go.mod you can use replacement block/lines to use private repos previously cloned the same way (within a respective folder cloned also in $GOPATH/src/go.yourcompany.tld) (why do we have to depend too much in GitHub?)
Edit
Needless to say that a private repo usually shall be a private repo, typically a standard git repo, right? With that, why not just to git clone and then go get within the cloned folder?
It's Hard Code In Go Get. Not The Git Reason. So Modify Go Source.
Reason:
repoRootForImportDynamic Will Request: https://....go-get
// RepoRootForImportPath analyzes importPath to determine the
// version control system, and code repository to use.
func RepoRootForImportPath(importPath string, mod ModuleMode, security web.SecurityMode) (*RepoRoot, error) {
rr, err := repoRootFromVCSPaths(importPath, security, vcsPaths)
if err == errUnknownSite {
rr, err = repoRootForImportDynamic(importPath, mod, security)
if err != nil {
err = importErrorf(importPath, "unrecognized import path %q: %v", importPath, err)
}
}
if err != nil {
rr1, err1 := repoRootFromVCSPaths(importPath, security, vcsPathsAfterDynamic)
if err1 == nil {
rr = rr1
err = nil
}
}
So add gitlab domain to vcsPaths will ok.
Download go source code:
vi ./src/cmd/go/internal/vcs/vcs.go
Look for code below:
var vcsPaths = []*vcsPath{
// GitHub
{
pathPrefix: "github.com",
regexp: lazyregexp.New(`^(?P<root>github\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/[A-Za-z0-9_.\-]+)*$`),
vcs: "git",
repo: "https://{root}",
check: noVCSSuffix,
},
add Code As Follow,XXXX Is Your Domain:
// GitLab
{
pathPrefix: "gitlab.xxxx.com",
regexp: lazyregexp.New(`^(?P<root>gitlab.xxxx\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/[A-Za-z0-9_.\-]+)*$`),
vcs: "git",
repo: "https://{root}",
check: noVCSSuffix,
},
compile and replace go.
I am trying to copy my entire Jenkins configuration from RHEL 6.7 to RHEL 6.9 , On doing this everything looks good, but only one jenkins build is failing with below error
Enter pass phrase:
can't connect to `/usr/share/tomcat6/.gnupg/S.gpg-agent': No such file or directory
gpg: skipped "Credit": Bad passphrase
gpg: signing failed: Bad passphrase
Pass phrase check failed
The gpg private key 1.4.5 exists in jenkins configuration. Strange thing is , all other builds are able to sign rpm but only one build is failing
Anyone know how to fix it ?
RPM reads the passphrase uses getpass(3) and sends to gnupg through an additional file descriptor.
This creates two problems that need to be handled by automating signing mechanisms:
1) Some versions of rpm use getpass(3) which will use a tty (to disable echoing) and will require setting up a pseudo tty so that the automated password can be passed to RPM. Make sure you have the pty file system mounted, and expect(1) is one way to setup the pty from which the password can be read. There's another approach using /proc file descriptors that can be attempted on linux. The password is then sent to gnupg using --passphrase-fd.
2) gnupg2 can also handle persistent passwords in a separate agent process which is sometimes tricky to setup and keep running "automatically" because the detection depends on both the user/process id's. Your report seems to have an agent (which means gnupg2 or special gpg1 configuration) even though you mention 1.4.5 (which would seem to use gnupg1).
I see two separate issues in your log that need to be addressed.
can't connect to `/usr/share/tomcat6/.gnupg/S.gpg-agent': No such file
or directory
gpg-agent needs to be running as a daemon on the build host, where it will connect to a socket to listen for requests. Perhaps it is already running, but Jenkins is looking for its socket in the wrong directory because GNUPGHOME is set to some unusual value. Or perhaps gpg-agent isn't running and a new instance needs to be started.
Something like this script can be used to safely attach to an existing gpg-agent or spin up a new instance.
#!/bin/bash
# Decide whether to start gpg-agent daemon.
# Create necessary symbolic link in $GNUPGHOME/.gnupg/S.gpg-agent
SOCKET=S.gpg-agent
PIDOF=`pgrep gpg-agent`
RETVAL=$?
if [ "$RETVAL" -eq 1 ]; then
echo "Starting gpg-agent daemon."
eval `gpg-agent --daemon `
else
echo "Daemon gpg-agent already running."
fi
# Nasty way to find gpg-agent's socket file...
GPG_SOCKET_FILE=`find /tmp/gpg-* -name $SOCKET`
echo "Updating socket file link."
cp -fs $GPG_SOCKET_FILE $GNUPGHOME/.gnupg/S.gpg-agent
You may want to substitute pgrep for pidof, depending on your shell.
If you do end up starting a new agent, you can check to see that your keys have been loaded into it by running gpg --list-keys. If you don't see it listed, you'll need to add it using gpg --import. Follow the Jenkins docs for Using Credentials.
Resolving the gpg-agent issue may resolve your other issue, so check to see if your job is working before doing anything else.
References:
www.linuxquestions.org
gpg: skipped "Credit": Bad passphrase
The GPG key is protected by a passphrase. rpm is asking for this passphrase and expects it to be manually entered. Of course, Jenkins is running things non-interactively, so that's not going to be possible. We need some way to supply the passphrase to rpm so it can forward it along to gpg, or else we need to supply the passphrase to gpg directly via some sort of caching mechanism.
The Expect Method
By wrapping our rpm --addsign call in an expect script, we can use expect to enter the passphrase headlessly. This practice is fairly common. Assuming the following script named rpm_sign.exp:
#!/usr/bin/expect -f
set password [lindex $argv 0]
set files [lrange $argv 1 1]
spawn rpm --define --addsign $files
expect "Enter pass phrase:"
send -- "$password\r"
expect eof
This script can be used in a Jenkins shell step or pipeline as follows:
echo "Signing rpms ..."
sh "./rpm_sign.exp '${GPG_PASSPHRASE}' <list-of-files>"
Please note that, with some modifications, it is possible to specify which GPG identity you want to sign your RPMs with. This is done by passing --define {_gpg_name $YOUR_KEY_ID_HERE} as an argument to rpm inside the wrapper script. Note the TCL syntax. Since we're doing this on Jenkins, which may offer multiple sets of credentials, I assume this is relevant info.
References:
aaronhawley.livejournal.com
lists.fedoraproject.org
Other Methods
There are other solutions out there that may be more appropriate to your configuration. One such solution is to use RpmSignPlugin, which uses expect under the hood. Other solutions can be found in this posting on unix.stackexchange.com.
Over the past few days, I've been working on getting CI working with an external mac mini running OS X Server. However, I have been having many problems with OS X Server 3.2.1 and XCode 6.1b3.
It looks like Apple fixed an issue in Xcode 6.1b3 which didn't put the correct provisioning profiles into Portal.keychain. However, my integrations aren't even running now.
After running a clean OS X build, XCode server won't integrate. I succesfully connected to the server and created a bot. If I visit "SERVER.local" on my development machine, I see the bot that I created.
Everything is set up properly (including the integrate immediately checkbox), however my integrations sit in the "pending" state. I checked the system.log, and nothing seems to be happening.
This could be completely unrelated, but every time I click on a pending integration, I receive this error in system.log:
NSFileCoordinator only handles URLs that use the file: scheme. This one does not:
x-code-xcsbot://XXX
I'm not sure if this is a new problem introduced in OS X server 3.2.1, or if it's just a set up issue. Apparently no one else has had this issue, couldn't find anything on Google/ SO.
This is still happening, but if you just want to nudge the server to wake up and run and aren't prepared to delete your x-code server configuration (provisioningProfiles, credentials and bots are deleted (as I recall)), simply run this terminal command
sudo -u _xcsbuildd /Applications/Xcode.app/Contents/Developer/usr/bin/xcsbuildd
Note that if you have multiple revs of Xcode in your Applications folder you may have named them differently, so the command may be slightly different. In my case, I've got a file named 'Xcode 6.1.1'. So the command would be
sudo -u _xcsbuildd /Applications/Xcode\ 6.1.1.app/Contents/Developer/usr/bin/xcsbuildd
This appears to happen when running XCode beta builds in OS X Server.
Note that this command will ❗️delete all your bots❗️
Run sudo xcrun xcscontrol --reset to reset.
https://devforums.apple.com/message/1051403#1051403
Here's a solution that may fix the problem without requiring to reset Xcode Server.
What's the problem?
First, check whether this answer applies by inspecting the xcsnginx.log log file:
sudo tail /Library/Developer/XcodeServer/Logs/xcsnginx.log
Search for the following line at the end of the log:
nginx: [emerg] SSL_CTX_use_PrivateKey_file("/Library/Developer/XcodeServer/Certificates/xcsnginx.key") failed (SSL: error:0906A068:PEM routines:PEM_do_header:bad password read error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib)
I you don't see that entry in the log, I'm afraid this answer won't help you. If you do see that entry, you may continue.
Why does it happen?
Xcode Server internally runs an Nginx web server (on port 20543) named xcsnginx that acts as a proxy between some services. This server uses a TLS/SSL certificate to ensure communications remain secure. The involved files are the following:
xcsnginx.crt: contains the PEM certificate.
xcsnginx.key: contains the private key for the certificate.
xcsnginx.pass: contains the passphrase for the private key.
As far as I understand, the private key is stored unencrypted, which means the xcsnginx.pass should be empty (and seems to be reset every time Xcode Server starts).
However for some reason, as some point, the private key in xcsnginx.key was exported as an encrypted key. I've no idea how and why this could happen but it did happen on my server so I'll assume it may happen on your server too. The consequence is that xcsnginx cannot load the certificate and fails to launch.
You can verify that xcsnginx is not running by executing:
pgrep xcsnginx || echo "Not running"
How to fix it?
Rather than resetting Xcode Server from scratch, we can:
export the identity again from the xcsnginx.keychain keychain or
restore the previous certificate and key or
create a new certificate and key for xcsnginx.
So let's have a look at each option.
Option 1
Copies of the certificate and private key are stored in the xcsnginx.keychain keychain located in /Library/Developer/XcodeServer/Keychains. This keychain is protected by a passphrase stored in a file named XCSNginxKeychainSharedSecret in the /Library/Developer/XcodeServer/SharedSecrets folder.
If you're familiar with OS X keychains, you may retrieve the certificate and the key from the keychain.
However manipulating keychains using the command-line is a real nightmare so I'll let this as an exercise for the reader (or an editor).
Option 2
The /Library/Developer/XcodeServer/Certificates folder may contains a backup of your certificate and key. Let's find out:
sudo find /Library/Developer/XcodeServer/Certificates -name "*.original"
If you're lucky, you should get the following result:
/Library/Developer/XcodeServer/Certificates/xcsnginx.crt.original
/Library/Developer/XcodeServer/Certificates/xcsnginx.key.original
/Library/Developer/XcodeServer/Certificates/xcsnginx.pass.original
Which mean you can restore the original files:
sudo cp /Library/Developer/XcodeServer/Certificates/xcsnginx.crt.original /Library/Developer/XcodeServer/Certificates/xcsnginx.crt
sudo cp /Library/Developer/XcodeServer/Certificates/xcsnginx.key.original /Library/Developer/XcodeServer/Certificates/xcsnginx.key
sudo cp /Library/Developer/XcodeServer/Certificates/xcsnginx.pass.original /Library/Developer/XcodeServer/Certificates/xcsnginx.pass
Option 3
If you cannot restore the previous certificate and keychains, you may decide to just generate new ones like this:
sudo openssl req -new -x509 -newkey rsa:2048 -nodes -out /Library/Developer/XcodeServer/Certificates/xcsnginx.crt -keyout /Library/Developer/XcodeServer/Certificates/xcsnginx.key -subj /CN=your-server.example.com -days 1000 -batch
where your-server.example.com is replaced with the DNS address of your server. Ideally the certificate should be issued by the Xcode Server Root Certificate Authority but using a single-signed certificate doesn't seem to be a problem (as far as I now / for the moment / your mileage may vary).
Finally
Now we just have to wait until the system starts xcsnginx again. That should happen automatically after a minute or less. You can verify that xcsnginx did start with:
pgrep xcsnginx || echo "Not running"
You can use restart instead of reset from the command line to restart the server. This will terminate the bot you are running that has frozen up.
sudo xcrun xcscontrol --restart
I've run into this several times. I've done the xcrun xcscontrol --reset and it does work, but I'm getting tired of that. As my bots get more complicated with pre/post integration triggers I don't want to set them up again.
What worked for me this time was just going through all mostly recent integrations and deleting any canceled ones. Cancelling a bot run can leave it in an odd state.
After I deleted a bunch of integrations I restarted the machine the server is running on and my integrations start working again!
Hope that helps someone.