Connecting to CosmosDB from Docker container - docker

Question

I am trying to learn Docker and am building up a solution slowly as I go. I created an Azure Function App that has one Http endpoint exposed. I created the Docker file to build and run the solution in a linux container(image:mcr.microsoft.com/azure-functions/dotnet:3.0, Debian image). I'm on a Windows machine.
I installed the Azure CosmosDB Emulator on my Windows machine and wanted to connect to it from the function app running in the linux container.
I am passing in the connection string for cosmos as an environment variable.
ARG COSMOS_CONNECTION_STRING="AccountEndpoint=https://host.docker.internal:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw=="
I'm trying to connect to the emulator through the following code:
await new CosmosClient(Environment.GetEnvironmentVariable("AzureCosmosConnectionString", EnvironmentVariableTarget.Process)
.GetContainer("db_name", "container_name")
.UpsertItemAsync<Dto>(dto)
.ConfigureAwait(false);
When I do this, I'm getting the following error(I assume the first few lines are the most relevant, but am including the rest in case I'm wrong):
---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Security.SslStream.ThrowIfExceptional()
at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_1(IAsyncResult iar)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.DocumentClient.HttpRequestMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Microsoft.Azure.Cosmos.GatewayAccountReader.GetDatabaseAccountAsync(Uri serviceEndpoint)
at Microsoft.Azure.Cosmos.Routing.GlobalEndpointManager.GetDatabaseAccountFromAnyLocationsAsync(Uri defaultEndpoint, IList`1 locations, Func`2 getDatabaseAccountFn)
at Microsoft.Azure.Cosmos.GatewayAccountReader.InitializeReaderAsync()
at Microsoft.Azure.Cosmos.CosmosAccountServiceConfiguration.InitializeAsync()
at Microsoft.Azure.Cosmos.DocumentClient.InitializeGatewayConfigurationReaderAsync()
at Microsoft.Azure.Cosmos.DocumentClient.GetInitializationTaskAsync(IStoreClientFactory storeClientFactory)
at Microsoft.Azure.Cosmos.DocumentClient.EnsureValidClientAsync()
at Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler.EnsureValidClientAsync(RequestMessage request)
at Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler.SendAsync(Uri resourceUri, ResourceType resourceType, OperationType operationType, RequestOptions requestOptions, ContainerInternal cosmosContainerble`1 partitionKey, Stream streamPayload, Action`1 requestEnricher, CosmosDiagnosticsContext diagnosticsContext, CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.ContainerCore.ProcessItemStreamAsync(Nullable`1 partitionKey, String itemId, Stream streamPayload, OperationType operationType, ItemRequestOptions requestOptions, CosmosDiagnostiiagnosticsContext, CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.ContainerCore.ExtractPartitionKeyAndProcessItemStreamAsync[T](Nullable`1 partitionKey, String itemId, T item, OperationType operationType, ItemRequestOptions requestOptions, CosmcsContext diagnosticsContext, CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.ContainerCore.UpsertItemAsync[T](T item, Nullable`1 partitionKey, ItemRequestOptions requestOptions, CancellationToken cancellationToken)
When I look at the emulator locally in my browser, it is using a localhost certificate(I assume one created with the dotnet cli).
Attempt 1
Export the localhost cert as a .pfx file and then get the linux container to trust that through the following commands(in my Dockerfile)
ARG CERTIFICATE_PASSWORD="Test|234"
RUN openssl pkcs12 \
-in "/src/localhost.pfx" \
-clcerts \
-nokeys \
-out "/src/localhost.crt" \
-passin pass:${CERTIFICATE_PASSWORD}
RUN cp "/src/localhost.crt" "/usr/local/share/ca-certificates/"
RUN update-ca-certificates
I assume this attempt doesn't work, at least in part, due to the fact that the certificate on the Windows machine is created for localhost, whereas to connect to it from docker, the address needs to be host.docker.internal.
Attempt 2
Add the exported certificate to the Kestrel process running the function app in hopes that it would then honor it by adding the following to my dockerfile
ENV ASPNETCORE_Kestrel__Certificates__Default__Path=/src/localhost.pfx
Attempt 3
Update the CosmosClient instantiation to include options overriding the HttpClientFactory as follows:
CosmosClient = new CosmosClient(
Environment.GetEnvironmentVariable("AzureCosmosConnectionString", EnvironmentVariableTarget.Process),
new CosmosClientOptions
{
HttpClientFactory = () =>
{
using (var httpClientHandler = new HttpClientHandler())
{
httpClientHandler.ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;
return new HttpClient(httpClientHandler);
}
}
});
With every attempt above, I'm still seeing the same error. Not sure what else to try to get this to work...
Just found this issue on GitHub. Seems as though I'm not alone.

Answer 1

Update (2/10/2021)
The SDK now allows overriding SSL validation in an easy fashion (reference https://learn.microsoft.com/azure/cosmos-db/local-emulator?tabs=cli%2Cssl-netstd21#disable-ssl-validation):
CosmosClientOptions cosmosClientOptions = new CosmosClientOptions()
{
HttpClientFactory = () =>
{
HttpMessageHandler httpMessageHandler = new HttpClientHandler()
{
ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator
};
return new HttpClient(httpMessageHandler);
},
ConnectionMode = ConnectionMode.Gateway
};
CosmosClient client = new CosmosClient(endpoint, authKey, cosmosClientOptions);
Older information
Reference:
https://github.com/Azure/azure-cosmos-dotnet-v3/issues/1551#issuecomment-634365001
https://learn.microsoft.com/en-us/azure/cosmos-db/local-emulator#running-on-mac-or-linux
Step 1 - Export the Certificate
Following this guide, but the certificate needs to be a PFX, not CRT. It will ask you to set some password.
Step 2 - Place the cert somewhere you can copy or access from docker
For example, I put it on a folder I can map to when booting Docker along with the code I wanted to run:
Step 3 - Get your machine's IP address
As per https://learn.microsoft.com/en-us/azure/cosmos-db/local-emulator#running-on-mac-or-linux, I used ipconfig and got my Windows IP.
Step 4 - Start a docker image
In my case, I use one with the NET Core 3.1 SDK, the official one from https://learn.microsoft.com/en-us/dotnet/architecture/microservices/net-core-net-framework-containers/official-net-docker-images
I started the container with an interactive shell, and mapping localhost to the IP I got on Step 3. This let's you use localhost in the connection string.
docker run -v /c/DockerSample:/DockerSample --add-host="localhost:192.168.1.15" -it mcr.microsoft.com/dotnet/core/sdk:3.1 /bin/bash
And I'm also mounting the folder where I saved the project and the certificate I want to import. This is not required, but I'm not fluent on Docker to know if there is a better way to pass the certificate.
After the shell starts, I basically run the commands described in the Emulator doc and the certificate gets added.
Step 5
The Docker container should now have the required Cert to connect to localhost and you should not need the HttpClientFactory.
NOTE: Also there is a bug tracking HttpClientFactory not being used everywhere that is the source of your error https://github.com/Azure/azure-cosmos-dotnet-v3/pull/1548

Answer 2

First of all, thank you to Matias Quaranta for all of the useful documentation. I had already pored over most of it before receiving his answer, but really appreciate all the time it took to compile it.
I have been struggling with this for around two weeks now and finally got something that I think will work. This work is largely based on the script found in this GitHub issue. I found that there is a PowerShell module that gets put on your computer when you install the Cosmos DB Emulator, so I tried to leverage those functions as much as possible to do the work.
The script's entry point is the function Start-CosmosDbEmulatorForDocker and it
Makes sure the Emulator is stopped.
Generates a new certificate to be used with the Docker image and replaces the one that was created upon install of the emulator.
Generates a .pfx certificate from the one created for the emulator.
Restarts the emulator once the new certificate is ready.
The password taken by the function is the one used for the .pfx file generated.
azureCosmosDbEmulator.ps1
using namespace System.ServiceProcess
Function Start-CosmosDbEmulatorForDocker(
[Parameter()]
[securestring]
$password
) {
$cosmosDbInstallLocation = Get-CosmosDbInstallLocation
If (!$cosmosDbInstallLocation) {
Install-AzureCosmosDBEmulator
}
Write-Host "Importing Microsoft.Azure.CosmosDB.Emulator powershell module."
Import-Module -Name "$($cosmosDbInstallLocation.InstallLocation)\PSModules\Microsoft.Azure.CosmosDB.Emulator"
Install-CosmosDBDockerCertificate -cosmosDbInstallLocation $cosmosDbInstallLocation.InstallLocation -password $password
Start-CosmosDbEmulator -AllowNetworkAccess -Key "C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw=="
}
Function Get-CosmosDbInstallLocation() {
Get-ChildItem HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall | ForEach-Object { Get-ItemProperty $_.PsPath } | Where-Object { $_.DisplayName -eq "Azure Cosmos DB Emulator" } | Select-Object InstallLocation
}
Function Install-AzureCosmosDBEmulator() {
Write-Host "Installing Azure Cosmos Db Emulator."
$installer = "$PSScriptRoot\cosmosEmulatorInstaller.msi"
curl "https://aka.ms/cosmosdb-emulator" -O $installer
Start-Process -Wait -FilePath msiexec -ArgumentList /i, $installer
Remove-Item $installer
}
Function Install-CosmosDBDockerCertificate(
[Parameter()]
[string]
$cosmosDbInstallLocation,
[Parameter()]
[securestring]
$password
) {
If ((Get-CosmosDbEmulatorStatus) -ne [ServiceControllerStatus]::Stopped) {
Write-Host "Stopping Cosmos DB emulator."
Stop-CosmosDbEmulator
}
$dockerCertificatesPath = Join-Path (Split-Path -Path $PSScriptRoot -Parent) "certificates"
$cosmosDbPfxCertificatePath = "$($dockerCertificatesPath)\cosmosdbemulator.pfx"
Uninstall-Certificate -dockerCertificatePath $cosmosDbPfxCertificatePath
Write-Host "Generating new Cosmos DB certificate to work with Docker."
New-CosmosDbEmulatorCertificate "host.docker.internal"
Start-Sleep -s 5
New-DockerCertificate -dockerCertificatePath $cosmosDbPfxCertificatePath -password $password
Set-Location (Split-Path -Path $PSScriptRoot -Parent)
}
Function Uninstall-Certificate(
[Parameter()]
[string]
$dockerCertificatePath
) {
Write-Host "Removing existing DocumentDbEmulatorCertificate certificate."
if (Test-Path $dockerCertificatePath) {
Remove-Item -Path $dockerCertificatePath
}
}
Function New-DockerCertificate(
[Parameter()]
[string]
$dockerCertificatePath,
[Parameter()]
[securestring]
$password
) {
Write-Host "Generating new pfx version of DocumentDbEmulatorCertificate certificate for use in Docker image."
Get-CosmosDbEmulatorCertificate | Export-PfxCertificate -Filepath $dockerCertificatePath -Password $password
}
I then have a shell script to be run inside the Docker image that will install the .pfx cert into the Docker container. The COSMOS_DB_EMULATOR_PFX_PASSWORD value must match the one used by the PowerShell script.
trust_cosmos_db_emulator_crt.sh
#!/bin/bash
# Save current working directory
PWD=`pwd`
pushd $PWD
# Find and move to the location of this script
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
cd $DIR
if [ -n "$1" ]; then
COSMOS_DB_EMULATOR_PFX=$1
else
COSMOS_DB_EMULATOR_PFX="/certificates/cosmosdbemulator.pfx"
fi
COSMOS_DB_EMULATOR_PFX_PASSWORD="SUPER_SECRET_PASSWORD"
CERT_TO_TRUST="cosmosdbemulator.crt"
# Generate .crt file if pfx exists
if [ -f "$COSMOS_DB_EMULATOR_PFX" ]; then
openssl pkcs12 -in $COSMOS_DB_EMULATOR_PFX -clcerts -nokeys -out cosmosdbemulator.crt -passin pass:$COSMOS_DB_EMULATOR_PFX_PASSWORD;
fi
# # Trust Cert (will end located in /etc/ssl/certs/ based on *.crt name as a *.pem, e.g. /etc/ssl/certs/cosmosdbemulator.pem for cosmosdbemulator.crt)
if [ -f "$CERT_TO_TRUST" ]; then
cp $CERT_TO_TRUST /usr/local/share/ca-certificates/
update-ca-certificates
rm $CERT_TO_TRUST;
fi
# Restore working directory
popd
The project structure I have is as follows:
src/
scripts/
azureCosmosDbEmulator.ps1
trust_cosmos_db_emulator_crt.sh
certificates/
DockerFile
The Dockerfile contains the following lines:
COPY ["/scripts/", "/scripts/"]
COPY ["/certificates/", "/certificates/"]
RUN /scripts/trust_cosmos_db_emulator_crt.sh
With all of that in place, I can build the docker image with docker build -t temp . and then run it with docker run -it -p 80:80 temp and the code running inside of the docker container will talk to my local machine's installed version of the Azure Cosmos DB Emulator.
As this was a HUGE pain in the neck, if you are experiencing this pain as well, vote for better support from Microsoft on this here.