I've added keycloak to my API Gateway and since then its failed to startup, more specifically after adding tokenRelay as default-filter, here's the log
org.springframework.context.ApplicationContextException: Failed to start bean 'eurekaAutoServiceRegistration'; nested exception is reactor.core.Exceptions$ErrorCallbackNotImplemented: java.lang.IllegalArgumentException: Unable to find GatewayFilterFactory with name TokenRelay
I've declared all necessary dependencies
jose
resource server
oauth2 client
starter-security
and my application class has eureka enabled and consuming properties from the config server.
I'm using boot version: 2.3.12.RELEASE, cloud: Hoxton.RELEASE, java 8
I've added in application.yml
...
default-filters:
- TokenRelay
Related
I am trying to connect 2 Keycloak instances. One as an Identity Provider and one as an external broker by following this article.
I am running 2 docker instances for keycloak with the following setup:
Keycloak-External (127.0.0.1:9090)
Keycloak Internal (127.0.0.1:8080)
Created 2 realms:
realm:“keycloak-external-broker” in Keycloak External (127.0.0.1:9090)
realm “keycloak-internal-identity” in Keycloak Internal (127.0.0.1:8080)
Registered Identity Provider on external broker with following setup
Configured Broker as Client on internal identity provider
Created a demo user from the user's section in Keycloak Internal (127.0.0.1:8080)
Accessed External Login url and click on sigin
http://127.0.0.1:9090/realms/keycloak-external-broker/account/
Following page shows up:
Clicking Employee login redirects me to internal identity provider as
But when i try to login with user created in internal keycloak.it gives me following error.
I'm putting out logs as well. I tried troubleshooting with various options as suggested in various blogs but still facing the issue. Appreciate any leads, Is there anything that i'm missing?
2023-02-15 09:11:40,900 WARN [org.keycloak.events] (executor-thread-179) type=IDENTITY_PROVIDER_LOGIN_ERROR, realmId=bf5ad1d9-c4c9-4f20-8272-e415c04ec67c, clientId=account-console, userId=null, ipAddress=172.17.0.1, error=identity_provider_login_failure, code_id=970a50e2-50af-4f38-80e1-8d4dd900c825, authSessionParentId=970a50e2-50af-4f38-80e1-8d4dd900c825, authSessionTabId=SPngHd5ApfM
2023-02-15 10:05:28,991 ERROR [org.keycloak.broker.oidc.AbstractOAuth2IdentityProvider] (executor-thread-190) Failed to make identity provider oauth callback: org.keycloak.broker.provider.IdentityBrokerException: No access_token from server. error='Realm does not exist', error_description='null', error_uri='null'
at org.keycloak.broker.oidc.OIDCIdentityProvider.verifyAccessToken(OIDCIdentityProvider.java:574)
at org.keycloak.broker.oidc.OIDCIdentityProvider.getFederatedIdentity(OIDCIdentityProvider.java:384)
at org.keycloak.broker.oidc.AbstractOAuth2IdentityProvider$Endpoint.authResponse(AbstractOAuth2IdentityProvider.java:502)
at jdk.internal.reflect.GeneratedMethodAccessor669.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:170)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130)
at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:660)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:524)
at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:474)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:476)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:434)
at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:192)
at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:152)
at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:183)
at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:141)
at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:32)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:492)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:261)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:161)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:164)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:247)
at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:73)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:151)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.handle(VertxRequestHandler.java:82)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.handle(VertxRequestHandler.java:42)
at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1284)
at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:173)
at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:140)
at io.quarkus.vertx.http.runtime.StaticResourcesRecorder$2.handle(StaticResourcesRecorder.java:84)
at io.quarkus.vertx.http.runtime.StaticResourcesRecorder$2.handle(StaticResourcesRecorder.java:71)
at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1284)
at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:173)
at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:140)
at io.quarkus.vertx.http.runtime.VertxHttpRecorder$6.handle(VertxHttpRecorder.java:430)
at io.quarkus.vertx.http.runtime.VertxHttpRecorder$6.handle(VertxHttpRecorder.java:408)
at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1284)
at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:173)
at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:140)
at org.keycloak.quarkus.runtime.integration.web.QuarkusRequestFilter.lambda$createBlockingHandler$0(QuarkusRequestFilter.java:82)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:576)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-02-15 10:05:29,004 WARN [org.keycloak.events] (executor-thread-190) type=IDENTITY_PROVIDER_LOGIN_ERROR, realmId=bf5ad1d9-c4c9-4f20-8272-e415c04ec67c, clientId=account-console, userId=null, ipAddress=172.17.0.1, error=identity_provider_login_failure, code_id=9ab52933-adf7-4987-97af-320797a534cb, authSessionParentId=9ab52933-adf7-4987-97af-320797a534cb, authSessionTabId=o8n1TLDNldc
I am enabling SSL in my mulesoft application.
I have added a keyStore in my application in http request connector.
Everything works fine. Deployed to cloudhub by skipping Munits and tested. Everything works fine. Munit are running fine in local machine.
But when I run MUnit tests in jenkins. It throws below error.
Caused by:
org.mule.runtime.api.lifecycle.InitialisationException:Unable to
initialise TLS configuration
Caused by:org.mule.runtime.api.lifecycle.CreateException: Failed to
load KeyStore:
/mnt/jenkins/workspace/mule-jobs/application-prc-nonprod/target/munitworkingdir-5223661232046316/container/apps/application-prc/server-keystore-nonprod.jks
Caused by: java.io.IOException: Invalid keystore format
I have enabled this TLS for mulesoft system API and I am mocking this request connector. Not sure why it wants this jks file from jenkins location as shown in the error.
Any lead would be really helpful.
Http connection configuration
<http:request-config name="Mule_AWS_Request_configuration" doc:name="HTTP Request configuration" doc:id="067166d7-313b-432a-b653-88617b728c5a" responseTimeout="${muleApi.aws.response.timeout}">
<http:request-connection protocol="HTTPS" host="${muleApi.aws.host} " connectionIdleTimeout="${muleApi.aws.idletTimeout}">
<reconnection >
<reconnect frequency="${muleApi.aws.retryFrequency}" count="${muleApi.aws.reconnectionAttempts}" />
</reconnection>
<tls:context>
<tls:trust-store insecure="true"/>
<tls:key-store type="jks" path="${tls.keyStore.path}" keyPassword="${secure::tls.keyStore.keyPassword}" password="${secure::tls.keyStore.password}" />
</tls:context>
</http:request-connection>
</http:request-config>
Http request
<http:request method="POST" doc:name="POST ZIP / aws system api call" doc:id="965033c6-4623-45cd-9326-920534684ace" config-ref="Mule_AWS_Request_configuration" path="${muleApi.aws.path}" responseTimeout="600000" requestStreamingMode="ALWAYS">
<non-repeatable-stream />
<http:headers><![CDATA[#[output application/java
---
{
"client_secret" : p('secure::muleApi.aws.client_secret'),
"Content-Type" : "application/zip",
"client_id" : p('secure::muleApi.aws.client_id')
}]]]></http:headers>
<http:query-params><![CDATA[#[output application/java
---
{
"bucketName" : p('muleApi.aws.params.bucketName'),
"key" : vars.key,
"extractZipContent": vars.extractZipContent default "true",
"extractContentType": vars.extractContentType default "csv"
}]]]></http:query-params>
<http:response-validator>
<http:success-status-code-validator values="200..499" />
</http:response-validator>
</http:request>
Location of keyStore.
path: src/main/resource/server-keystore-nonprod.jks
I am just mocking the request connector as shown below.
[![enter image description here][3]][3]
Mule Version is 4.4.0
MUnit version is 2.3.5
Providing the logs till i see this error below.
INFO 2022-12-07 09:27:52,417 [munit.01] org.mule.munit.runner.remote.api.server.RunnerServer: Waiting for client connection
INFO 2022-12-07 09:27:52,418 [munit.01] org.mule.munit.runner.remote.api.server.RunnerServer: Client connection received from localhost - true
WARN 2022-12-07 09:27:52,859 [munit.01] org.mule.runtime.config.internal.dsl.spring.ComponentConfigurationBuilder: Parameter ignore from extension munit has a defaultValue configured in the componentBuildingDefinition but not in the extensionModel.
INFO 2022-12-07 09:27:53,029 [munit.01] org.mule.runtime.api.message.AbstractMuleMessageBuilderFactory: Loaded MuleMessageBuilderFactory implementation 'org.mule.runtime.core.internal.message.DefaultMessageBuilderFactory' from classloader 'java.net.URLClassLoader#3bbf841e'
WARN 2022-12-07 09:27:53,056 [munit.01] org.mule.runtime.core.internal.security.tls.TlsProperties: File tls-default.conf not found, using default configuration.
INFO 2022-12-07 09:27:53,057 [munit.01] org.mule.runtime.api.tls.AbstractTlsContextFactoryBuilderFactory: Loaded TlsContextFactoryBuilderFactory implementation 'org.mule.runtime.module.tls.api.DefaultTlsContextFactoryBuilderFactory' from classloader 'java.net.URLClassLoader#3bbf841e'
WARN 2022-12-07 09:27:53,097 [munit.01] org.mule.runtime.module.tls.internal.DefaultTlsContextFactory: TLS context trust store set as insecure. No certificate validations will be performed, rendering connections vulnerable to attacks. Use at own risk.
INFO 2022-12-07 09:27:57,038 [munit.01] org.mule.runtime.core.privileged.lifecycle.AbstractLifecycleManager: Initialising Bean: org.mule.runtime.module.extension.internal.runtime.config.ConfigurationProviderToolingAdapter-HTTP-requestConfig-implicit
INFO 2022-12-07 09:27:57,054 [munit.01] org.mule.runtime.core.privileged.lifecycle.AbstractLifecycleManager: Starting Bean: org.mule.runtime.module.extension.internal.runtime.config.ConfigurationProviderToolingAdapter-HTTP-requestConfig-implicit
INFO 2022-12-07 09:27:58,454 [munit.01] org.mule.runtime.core.privileged.lifecycle.AbstractLifecycleManager: Initialising Bean: org.mule.runtime.module.extension.internal.runtime.config.DynamicConfigurationProvider-JSON_Logger_Config
INFO 2022-12-07 09:27:58,458 [munit.01] org.mule.runtime.core.privileged.lifecycle.AbstractLifecycleManager: Initialising Bean: org.mule.runtime.module.extension.internal.runtime.config.ConfigurationProviderToolingAdapter-Mule_AWS_Request_configuration
org.mule.runtime.api.exception.MuleRuntimeException: org.mule.runtime.api.lifecycle.InitialisationException: Unable to initialise TLS configuration
Caused by: org.mule.runtime.api.lifecycle.InitialisationException: Unable to initialise TLS configuration
Caused by: org.mule.runtime.api.lifecycle.CreateException: Failed to load KeyStore: /mnt/jenkins/workspace/mule-jobs-nonprod/biz-prc-marketing-bulk-api-nonprod/target/munitworkingdir-5223661232046316/container/apps/biz-prc-marketing-bulk-api/server-keystore-nonprod.jks
Caused by: java.io.IOException: Invalid keystore format
I am setting up Apache Atlas with Cassandra as backend and solr as indexer. Below are my configuration setting for Atlas-cassandra
atlas.graph.storage.backend=cql
atlas.graph.storage.hostname=cassandra-dc-service
atlas.graph.storage.cassandra.keyspace=JanusGraph
atlas.graph.storage.clustername=cstone-cassandra
atlas.graph.storage.port=9042
atlas.EntityAuditRepository.impl=org.apache.atlas.repository.audit.CassandraBasedAuditRepository
atlas.EntityAuditRepository.keyspace=atlas_audit
atlas.EntityAuditRepository.replicationFactor=1
atlas.graph.storage.username=cassandra-superuser
atlas.graph.storage.password=xxxxxx
I took help from (Apache atlas 2.1 configuration connection cassandra backend [help]) to resolve cassandra connection issue for creating keyspace. But Atlas Service is not coming up with below exception:
2022-02-17 22:03:02,742 ERROR - [main:] ~ Context initialization failed (ContextLoader:350)
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'services': Invocation of init method failed; nested excep
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotat
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutow
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:16
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
Caused by: com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra-dc-service/
at com.datastax.driver.core.AuthProvider$1.newAuthenticator(AuthProvider.java:40)
at com.datastax.driver.core.Connection$5.apply(Connection.java:257)
at com.datastax.driver.core.Connection$5.apply(Connection.java:242)
at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:210)
at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:199)
at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:111)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1024)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
I see open Jira issue for the same.
If anyone already resolved this issue kindly suggest the workaround
Officially till Atlas 2.2.0 not support for authenticated Cassandra. But while building code we can update two files which are given in below PR and build Atlas. Which works as expected
https://github.com/apache/atlas/pull/67
I'm running iotedge on kubernetes.
The K8S cluster is a local cluster setup largely using the "Kubernetes the hard way" method, with some modifications.
I did manage to get things working on one installation. However, I'm now getting this on another installation. The initial installation works fine, but after shutting down a machine to simulate a hardware failure, the pod gets recreated, but starts to show this error again. This error happens EVEN if the node shutdown is NOT the one iotedged is running on.
Environment
3 Nodes running Ubuntu 20.04 LTS
Two networks on each node, one for the internet, one for an internal network. K8S is setup using the internal, static IP address
HAProxy/Keepalived for HA without a load balancer, running on a Virtual IP address
Multus CNI for attaching pods to additional networks
CoreDNS
Troubleshooting
Confirmed that CoreDNS seems to be functioning fine, and is able to resolve internal and external addresses
Remaining nodes are able to ping pods on other nodes
Deleting the iotedged pod and allowing k8s to recreate it works, but then edgeAgent an edgeHub have errors until I delete/recreate them as well
Re-run the entire k8s installation. Initial installation works fine, but simulating machine failure continues to be problematic.
Kubernetes Versions:
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:25:06Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
edgeiotd error:
<6>2021-07-09T22:00:02Z [INFO] - Starting Azure IoT Edge Security Daemon - Kubernetes mode
<6>2021-07-09T22:00:02Z [INFO] - Version - 1.1.3
<6>2021-07-09T22:00:02Z [INFO] - Using config file: /etc/iotedged/config.yaml
<6>2021-07-09T22:00:02Z [INFO] - Configuring /var/lib/iotedge as the home directory.
<6>2021-07-09T22:00:02Z [INFO] - Configuring certificates...
<6>2021-07-09T22:00:02Z [INFO] - Transparent gateway certificates not found, operating in quick start mode...
<6>2021-07-09T22:00:02Z [INFO] - Finished configuring provisioning environment variables and certificates.
<6>2021-07-09T22:00:02Z [INFO] - Initializing hsm...
<6>2021-07-09T22:00:02Z [INFO] - Finished initializing hsm.
<6>2021-07-09T22:00:02Z [INFO] - Provisioning edge device...
<6>2021-07-09T22:00:02Z [INFO] - Starting provisioning edge device via manual mode using a device connection string...
<6>2021-07-09T22:00:02Z [INFO] - Manually provisioning device "********" in hub "********.azure-devices.net"
<6>2021-07-09T22:00:02Z [INFO] - Finished provisioning edge device.
<6>2021-07-09T22:00:02Z [INFO] - Initializing the module runtime...
<6>2021-07-09T22:00:02Z [INFO] - Attempting to use config from /home/edgeletuser/.kube/config file.
<6>2021-07-09T22:00:02Z [INFO] - Using in-cluster config
<3>2021-07-09T22:00:34Z [ERR!] - The daemon could not start up successfully: Could not initialize module runtime
<3>2021-07-09T22:00:34Z [ERR!] - caused by: Could not initialize kubernetes module runtime
<3>2021-07-09T22:00:34Z [ERR!] - caused by: HTTP response error: SelfSubjectAccessReviewCreate
<3>2021-07-09T22:00:34Z [ERR!] - caused by: Hyper HTTP error
<3>2021-07-09T22:00:34Z [ERR!] - caused by: error trying to connect: Connection timed out (os error 110)
<6>2021-07-09T22:00:02Z [INFO] (/project/hsm-sys/azure-iot-hsm-c/src/hsm_log.c:log_init:41) Initialized logging
edgeHub Logs after recreating iotedged:
2021-08-18 19:05:40 Starting Edge Hub
2021-08-18 19:05:40.481 +00:00 Edge Hub Main()
<7> 2021-08-18 19:05:40.609 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Making a Http call to http://localhost:35001/ to CreateServerCertificateAsync
<7> 2021-08-18 19:05:40.912 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Error when getting an Http response from http://localhost:35001/ for CreateServerCertificateAsync
HTTP Response:
{"message":"Module not found"}
Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.GeneratedCode.IoTEdgedException`1[Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.GeneratedCode.ErrorResponse]: Not Found
at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.GeneratedCode.HttpWorkloadClient.CreateServerCertificateAsync(String api_version, String name, String genid, ServerCertificateRequest request, CancellationToken cancellationToken) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/generatedCode/HttpWorkloadClient.cs:line 624
at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter[T](Task`1 task, TimeSpan timeout) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 126
at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 59
Unhandled exception. System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Module not found)
---> Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException- Message:Error calling CreateServerCertificateAsync: Module not found, StatusCode:404, at: at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 109
at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 77
at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35
at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 260
at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration, ILogger logger) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 54
at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 54
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task`1.get_Result()
at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 33
Are you still blocked? what troubleshooting steps you have tried so far? Did you check the Common issues and resolutions for Azure IoT Edge? As per the error messages Transparent gateway certificates not found, operating in quick start mode and The daemon could not start up successfully: Could not initialize module runtime looks like the setup is not configured properly. Try restarting the server and check the transparent gateway setup. Please refer the transparent gateway setup and check if you have missed anything.
We are trying to establish a grpcs (TLS) connection between a docker container running API server (based on Node.js) and another docker container running peer0 from Fabric network.
All containers are orchestated by docker swarm, and both containers happen to be running on the same Linux host.
The error log thrown by API container is the following:
2021-01-07T18:27:38.110Z - error: [Remote.js]: Error: Failed to
connect before the deadline URL:grpcs://10.0.1.2:9051 Query has
completed, checking results error from query = { Error: Failed to
connect before the deadline URL:grpcs://10.0.1.2:9051
at checkState (/usr/src/app/node_modules/grpc/src/client.js:833:16) connectFailed:
true } sampleEvent ERROR : Error: 14 UNAVAILABLE: Connect Failed E0107
18:27:53.602719124 16 ssl_transport_security.cc:1229] Handshake
failed with fatal error SSL_ERROR_SSL: error:14090086:SSL
routines:ssl3_get_server_certificate:certificate verify failed.
And the error log thrown from peer0 is:
2021-01-07 18:50:22.224 UTC [core.comm] ServerHandshake -> ERRO 043 TLS handshake failed with error EOF server=PeerServer remoteaddress=10.0.1.4:46212
IP addresses layout
IP address for API container is 10.0.1.94
IP address for peer0 container is 10.0.1.3
virtual IP address for docker service peer0 is 10.0.1.2
IP address for docker swarm load balancer endpoint is 10.0.1.4
Any suggestion of where to further troubleshoot? At this point is not clear if the problem is with the docker swarm internal networking, or an issue with ssl certificates in either side of the network.
UPDATE Feb 2 2021
The original TLS handshake error was fixed by upgrading the javascript used in NodeSDK. Among other things we started using the addToWallet.js script contained in the commercial-paper example
After being able to stablish TLS succesfully between Node.js API and peer0, we get a new access denied error when making a simple query to chaincode_example02
Facts:
We are running the query with 2 Admin users
One Admin is first-network original Admin#org1.example.com, with credentials generated by cryptogen tool
The other Admin is Admin#buyer.dlt.com whose credentials were created with openssl and a self signed in-company CA
From CLI, both Admin are good and are allowed to run peer commands interchangeably
From Node.js app, only Admin#org1.example.com is allowed to run queries. The message printed to console.log is:
Transaction has been evaluated, result is: 100
When running queries with Admin#buyer.dlt.com we get the following error logs:
Error logs from peer0#buyer.dlt.com
2021-02-02T04:08:45.291086617Z ^[[36m2021-02-02 04:08:45.290 UTC [protoutils] checkSignatureFromCreator -> DEBU 6e637^[[0m creator is &{BuyerMSP 8b7cc2ee996be4f7e5dbb1a4f64db67afd2ff8a2f41276c9bd7f33a2447dd9df}
2021-02-02T04:08:45.291094817Z ^[[36m2021-02-02 04:08:45.290 UTC [protoutils] checkSignatureFromCreator -> DEBU 6e638^[[0m creator is valid
2021-02-02T04:08:45.291100418Z ^[[36m2021-02-02 04:08:45.290 UTC [msp.identity] 2021-02-02T04:08:45.303821799Z ^[[33m2021-02-02 04:08:45.303 UTC [protoutils] ValidateProposalMessage -> WARN 6e63b^[[0m channel [mychannel]: creator's signature over the proposal is not valid: The signature is invalid
2021-02-02T04:08:45.303891604Z ^[[36m2021-02-02 04:08:45.303 UTC [endorser] func1 -> DEBU 6e63c^[[0m Exit: request from 10.0.1.84:52696
2021-02-02T04:08:45.303902005Z ^[[34m2021-02-02 04:08:45.303 UTC [comm.grpc.server] 1 -> INFO 6e63d^[[0m unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=10.0.1.84:52696 error="access denied: channel [mychannel] creator org [BuyerMSP]" grpc.code=Unknown grpc.call_duration=13.783655ms
Error log on console.log from script query.js:
2021-02-02T04:08:45.305Z - error: [Channel.js]: Error: 2 UNKNOWN: access denied: channel [mychannel] creator org [BuyerMSP]
2021-02-02T04:08:45.307Z - error: [Network]: _initializeInternalChannel: Unable to initialize channel. Attempted to contact 1 Peers. Last error was Error: 2 UNKNOWN: access denied: channel [mychannel] creator org [BuyerMSP]
Failed to evaluate transaction: Error: Unable to initialize channel. Attempted to contact 1 Peers. Last error was Error: 2 UNKNOWN: access denied: channel [mychannel] creator org [BuyerMSP]
In the end, this issue turned out to be two issues, in a 'russian doll like' style.
1. First issue: TLS Handshake error
This was fixed by upgrading the SDK library to the latest release
2. Second issue: Node SDK query triggers error "The signature is invalid".
The reason turned out to be that the CLI (written on Go) is using the Go crypto support which allows it to generate a signature from a hash without any knowledge of the curve used for the key. Instead, the SDK libraries used by the Node implementation require a specific curve to be specified by the code generating the signature, separately from the private key itself.
Bottom line, private keys used within Node SDK should be P-256.
As an alternative, as suggested by hyperledger dev team:
If you really must use a curve other than P-256 then you might be able
to use one of the following approaches:
-Use the off-line signing approach included in the documentation but specify an alternative curve instead of 'p256'. The supported curves
for the elliptic package documented here:
https://github.com/indutny/elliptic
-Set your own CryptoSuite implementation on the Client that underpins the Gateway object, with your own CryptoSuite.sign() implementation:
https://hyperledger.github.io/fabric-sdk-node/release-2.2/CryptoSuite.html#sign