WSo2 APIM getting error 401 trying to generate keys - oauth

Question

I am getting the following error when I try generate keys for a created application in the WSO2 API Manager, not sure why or how do I resolve this. I have WSO2 AM running on seperate instances than the gateways and KM node, everything else seems fine so far just errors when I try go through the key generation wizard or generate them manually:
TID: [-1234] [api/am/store] [2021-06-14 08:21:59,124] ERROR {org.wso2.carbon.apimgt.impl.AbstractKeyManager} - Cannot create OAuth application : wso2useradmin_DefaultApplication_PRODUCTION org.wso2.carbon.apim
gt.impl.kmclient.KeyManagerClientException: Received status code: 401 Reason:
at org.wso2.carbon.apimgt.impl.kmclient.KMClientErrorDecoder.decode_aroundBody0(KMClientErrorDecoder.java:29)
at org.wso2.carbon.apimgt.impl.kmclient.KMClientErrorDecoder.decode(KMClientErrorDecoder.java:27)
at feign.AsyncResponseHandler.handleResponse(AsyncResponseHandler.java:96)
at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:138)
at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:89)
at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:100)
at com.sun.proxy.$Proxy480.createApplication(Unknown Source)
at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.createApplication_aroundBody0(AMDefaultKeyManagerImpl.java:135)
at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.createApplication(AMDefaultKeyManagerImpl.java:97)
at org.wso2.carbon.apimgt.impl.workflow.AbstractApplicationRegistrationWorkflowExecutor.dogenerateKeysForApplication_aroundBody8(AbstractApplicationRegistrationWorkflowExecutor.java:154)
at org.wso2.carbon.apimgt.impl.workflow.AbstractApplicationRegistrationWorkflowExecutor.dogenerateKeysForApplication(AbstractApplicationRegistrationWorkflowExecutor.java:125)
at org.wso2.carbon.apimgt.impl.workflow.AbstractApplicationRegistrationWorkflowExecutor.generateKeysForApplication_aroundBody6(AbstractApplicationRegistrationWorkflowExecutor.java:121)
at org.wso2.carbon.apimgt.impl.workflow.AbstractApplicationRegistrationWorkflowExecutor.generateKeysForApplication(AbstractApplicationRegistrationWorkflowExecutor.java:118)
at org.wso2.carbon.apimgt.impl.workflow.ApplicationRegistrationSimpleWorkflowExecutor.complete_aroundBody2(ApplicationRegistrationSimpleWorkflowExecutor.java:78)
at org.wso2.carbon.apimgt.impl.workflow.ApplicationRegistrationSimpleWorkflowExecutor.complete(ApplicationRegistrationSimpleWorkflowExecutor.java:66)
at org.wso2.carbon.apimgt.impl.workflow.ApplicationRegistrationSimpleWorkflowExecutor.execute_aroundBody0(ApplicationRegistrationSimpleWorkflowExecutor.java:54)
at org.wso2.carbon.apimgt.impl.workflow.ApplicationRegistrationSimpleWorkflowExecutor.execute(ApplicationRegistrationSimpleWorkflowExecutor.java:47)
at org.wso2.carbon.apimgt.impl.APIConsumerImpl.requestApprovalForApplicationRegistration_aroundBody146(APIConsumerImpl.java:4262)
at org.wso2.carbon.apimgt.impl.APIConsumerImpl.requestApprovalForApplicationRegistration(APIConsumerImpl.java:4094)
at org.wso2.carbon.apimgt.rest.api.store.v1.impl.ApplicationsApiServiceImpl.applicationsApplicationIdGenerateKeysPost(ApplicationsApiServiceImpl.java:545)
at org.wso2.carbon.apimgt.rest.api.store.v1.ApplicationsApi.applicationsApplicationIdGenerateKeysPost(ApplicationsApi.java:126)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:193)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:103)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:216)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:301)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:220)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:276)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:86)
at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:110)
at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:102)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:49)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:145)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.wso2.carbon.tomcat.ext.valves.RequestCorrelationIdValve.invoke(RequestCorrelationIdValve.java:124)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)```

Answer 1

As per the shared trace, the API Manager was not able to authenticate with the (401) Key Manager component to register a client.
TID: [-1234] [api/am/store] [2021-06-14 08:21:59,124] ERROR {org.wso2.carbon.apimgt.impl.AbstractKeyManager} - Cannot create OAuth application : wso2useradmin_DefaultApplication_PRODUCTION org.wso2.carbon.apim
gt.impl.kmclient.KeyManagerClientException: Received status code: 401 Reason:
at org.wso2.carbon.apimgt.impl.kmclient.KMClientErrorDecoder.decode_aroundBody0(KMClientErrorDecoder.java:29)
at org.wso2.carbon.apimgt.impl.kmclient.KMClientErrorDecoder.decode(KMClientErrorDecoder.java:27)
at feign.AsyncResponseHandler.handleResponse(AsyncResponseHandler.java:96)
at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:138)
at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:89)
at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:100)
at com.sun.proxy.$Proxy480.createApplication(Unknown Source)
at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.createApplication_aroundBody0(AMDefaultKeyManagerImpl.java:135)
at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.createApplication(AMDefaultKeyManagerImpl.java:97)
at org.wso2.carbon.apimgt.impl.workflow.AbstractApplicationRegistrationWorkflowExecutor.dogenerateKeysForApplication_aroundBody8(AbstractApplicationRegistrationWorkflowExecutor.java:154)
This can be due to misconfigured username and password in the TOML configurations. If you have changed the super admin credentials, please do verify the following configurations are updated with the correct set of credentials in your environment
[apim.key_manager]
username = "superadmin-username"
password = "superadmin-password"
Either update the above-mentioned configurations with the correct set of credentials or comment on both username and password configurations to pick the super admin credentials on their own with the default mappings (since you are having the Key Manager in the same node as the Devportal).
Please try out the suggested approach and let us know