Issue on installing a build agent for TFS 2017 - tfs

Question

I have a stand-alone single server TFS 2017 RTM installation on SQL Express. It is not in any domain, but on my laptop. Have brought it up for my own usage (for learning too). Set it up to run on a self-signed certificate for SSL.
Thought of setting up a vNext build agent on the same machine. I remember somewhere in my brain that build machine should be not in the TFS app tier, to reduce surface attack. Well I don't really bother now about surface attacks. But rather I need a working TFS with build setup as well.
While configuring after entering the URL (https URL), I get prompted for authentication type. I typed Negotiate. After entering credentials, I get below error.
TF400813: Resource not available for anonymous access. Client authentication required.
Upon using Integrated for the authentication type, I get below error.
An error occurred while sending the request.
Failed to connect. Try again or ctrl-c to quit
When I use the http site for URL I get below error for both the authentication types.
Connecting to server ...
Found
Failed to connect. Try again or ctrl-c to quit
I am able to ping to my URL.
Below is the error log from _diag folder.
[2016-11-23 19:53:40Z INFO CredentialManager] Creating type Integrated
[2016-11-23 19:53:40Z INFO CredentialManager] Creating credential type: Integrated
[2016-11-23 19:53:40Z INFO IntegratedCredential] GetVssCredentials
[2016-11-23 19:53:40Z INFO ConfigurationManager] cred retrieved
[2016-11-23 19:53:40Z INFO Terminal] WRITE LINE: Connecting to server ...
[2016-11-23 19:53:41Z INFO CommandSettings] Flag 'unattended': 'False'
[2016-11-23 19:53:41Z ERR Terminal] WRITE ERROR (exception):
[2016-11-23 19:53:41Z ERR Terminal] Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Found
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponse(HttpResponseMessage response)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__45.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__42`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.<GetConnectionDataAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.<ConnectAsync>d__41.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Agent.AgentServer.<ConnectAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Agent.Listener.Configuration.ConfigurationManager.<TestConnectAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Agent.Listener.Configuration.ConfigurationManager.<ConfigureAsync>d__7.MoveNext()

Answer 1

Try to disable anonymous authentication and enable windows authentication for your TFS.
Since you set up TFS for your own usage, you can consider using Visual Studio Team Service instead of on-premise TFS. VSTS is free for 5 users.

Answer 2

Try using PAT for the authentication type, and get the personal access token, from the Security page in your profile menu. Using Selected scope - Agent Pools (read, manage)
See http://go.microsoft.com/fwlink/?LinkID=825113
The personal access token is used to register the agent. One final step is suppying the credentials for the Agent Service.

Answer 3

There are a few things that could have gone wrong.
After following Remko's answer and proceeding with "PAT" switch to "Negotiate" method, if you still cant get it working due to "An error occurred while sending the request", it is probably due to certificate issue. In such case, you can simply skip the certificate validation step which happens internally with curl by passing the parameter -sslskipcertvalidaion like below.
./config.sh --sslskipcertvalidation
Here's reference documentation : running agent with self-signed certificate