I am trying to deploy k8s deployment(simple pod) to EKS.
I am using Jenkins 2.319.1 and all plugins up to date and using Kubernetes cd plugin.
I have an error that user system:anonymous cannot patch resource "pods" in API group in the namespace default. I am not sure what is the reason? I seared a lot, but didn't solve yet. Could someone help me
Starting Kubernetes deployment
Loading configuration: /var/lib/jenkins/workspace/eks/nginx.yaml
ERROR: ERROR: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://0615cdb22445853e01165ad9c054a48f.gr7.us-east-1.eks.amazonaws.com/api/v1/namespaces/default/pods. Message: Forbidden! User arn:aws:eks:us-east-1:705746146520:cluster/my-eks-last doesn't have permission. pods is forbidden: User "system:anonymous" cannot create resource "pods" in API group "" in the namespace "default".
hudson.remoting.ProxyException: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://0615cdb22445853e01165ad9c054a48f.gr7.us-east-1.eks.amazonaws.com/api/v1/namespaces/default/pods. Message: Forbidden! User arn:aws:eks:us-east-1:705746146520:cluster/my-eks-last doesn't have permission. pods is forbidden: User "system:anonymous" cannot create resource "pods" in API group "" in the namespace "default".
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:472)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:409)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:381)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:227)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:780)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:349)
at com.microsoft.jenkins.kubernetes.KubernetesClientWrapper$PodUpdater.createResource(KubernetesClientWrapper.java:806)
at com.microsoft.jenkins.kubernetes.KubernetesClientWrapper$PodUpdater.createResource(KubernetesClientWrapper.java:775)
at com.microsoft.jenkins.kubernetes.KubernetesClientWrapper$ResourceUpdater.createOrApply(KubernetesClientWrapper.java:377)
at com.microsoft.jenkins.kubernetes.KubernetesClientWrapper.apply(KubernetesClientWrapper.java:183)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.doCall(DeploymentCommand.java:168)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:122)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:105)
at hudson.FilePath.act(FilePath.java:1259)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:67)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:46)
at com.microsoft.jenkins.azurecommons.command.CommandService.runCommand(CommandService.java:88)
at com.microsoft.jenkins.azurecommons.command.CommandService.execute(CommandService.java:96)
at com.microsoft.jenkins.azurecommons.command.CommandService.executeCommands(CommandService.java:75)
at com.microsoft.jenkins.azurecommons.command.BaseCommandContext.executeCommands(BaseCommandContext.java:77)
at com.microsoft.jenkins.kubernetes.KubernetesDeploy.perform(KubernetesDeploy.java:42)
at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:54)
at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:35)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
...POST at: https://0615cdb22445853e01165ad9c054a48f.gr7.us-east-1.eks.amazonaws.com/api/v1/namespaces/default/pods...
You didn't setup the plugin credential correctly, as a result your job is making direct call to the cluster endpoint WITHOUT authenticate with the cluster. Typically, you need to run your job using an IAM user who has access to the cluster, and you set this up using secret file (actually it's the kubeconfig file).
Related
I have problem uploading the findings of minIO securecodebox outputs to OWASP DefectDojo.
Screenshot of Error
https://drive.google.com/file/d/1PqVOazjr7r_1oMPf6SQsh8_iPFgnqkjC/view?usp=sharing
I try following these steps
https://github.com/DefectDojo/django-DefectDojo/blob/dev/readme-docs/KUBERNETES.md
then
https://docs.securecodebox.io/docs/hooks/defectdojo/
This is the link for the scanners
https://github.com/secureCodeBox/secureCodeBox/tree/main/scanners
The Error:
2022-03-07 07:23:54 INFO DefectDojoPersistenceProvider:35 - Downloading Scan Result ence provider
2022-03-07 07:23:56 INFO DefectDojoPersistenceProvider:39 - Uploading Findings to DefectDojo at: http://defectdojo.default.minikube.local:8080/ tDojo at: http://defectdojo.default.minikube.local:8080/
Exception in thread "main" org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://defectdojo.default.minikube.locarror on GET request for "http://defectdojo.default.minikube.local:8080/api/v2/users/": defectdojo.default.minikube.local; nested exception is java.net.UnknownHostException: defectdojo.default.minikube.local
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:785)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:751) rnalSearch(GenericDefectDojoService.java:151)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:621) ch(GenericDefectDojoService.java:167)
at io.securecodebox.persistence.defectdojo.service.GenericDefectDojoService.intechUnique(GenericDefectDojoService.java:187)rnalSearch(GenericDefectDojoService.java:151) ionedEngagementsStrategy.java:82)
at io.securecodebox.persistence.defectdojo.service.GenericDefectDojoService.search(GenericDefectDojoService.java:167)
at io.securecodebox.persistence.defectdojo.service.GenericDefectDojoService.searchUnique(GenericDefectDojoService.java:187)
at io.securecodebox.persistence.strategies.VersionedEngagementsStrategy.run(VersionedEngagementsStrategy.java:82)
at io.securecodebox.persistence.DefectDojoPersistenceProvider.main(DefectDojoPersistenceProvider.java:42)
Caused by: java.net.UnknownHostException: defectdojo.default.minikube.local
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:229)
at java.base/java.net.Socket.connect(Socket.java:609)
at java.base/java.net.Socket.connect(Socket.java:558)
at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
at java.base/sun.net.www.http.HttpClient.(HttpClient.java:242)
at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341)
at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1253)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015)
at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:76)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:776)
... 7 more
Thank you for the reponse!
there is a dedicated DefectDojo Hook which will do it for you.
You just need to install in on a cluster with some basic configuration.
Installing the DefectDojo persistenceProvider hook will add a ReadAndWrite Hook to your namespace.
kubectl create secret generic defectdojo-credentials --from-literal="username=admin" --from-literal="apikey=08b7..."
helm upgrade --install dd secureCodeBox/persistence-defectdojo
--set="defectdojo.url=https://defectdojo-django.default.svc"
The hook will automatically import the scan results into an engagement in DefectDojo. If the engagement doesn't exist the hook will create the engagement (CI/CD engagement) and all objects required for it (product & product type). The hook will then pull the imported information from DefectDojo and use them to replace the findings inside secureCodeBox.
More https://docs.securecodebox.io/docs/hooks/defectdojo
I have installed ZAP plugin on Jenkins and downloaded ZAP-Proxy WAR file in Jenkins Server(ubuntu)
and configured as jenkins job but its giving error when I execute the Job
4051 [ZAP-ProxyThread-15] WARN org.zaproxy.zap.extension.api.API - Bad request to API endpoint [/xml/spider/action/scanAsUser/] from [127.0.0.1]:
The provided url is not in the required context (url_not_in_context) : url
at org.zaproxy.zap.extension.spider.SpiderAPI.scanURL(SpiderAPI.java:508)
at org.zaproxy.zap.extension.spider.SpiderAPI.handleApiAction(SpiderAPI.java:283)
at org.zaproxy.zap.extension.api.API.handleApiRequest(API.java:506)
at org.parosproxy.paros.core.proxy.ProxyThread.processHttp(ProxyThread.java:499)
at org.parosproxy.paros.core.proxy.ProxyThread.run(ProxyThread.java:335)
at java.lang.Thread.run(Thread.java:748)
ERROR: org.zaproxy.clientapi.core.ClientApiException: The provided url is not in the required context
There is a property named Include in Context in job.
Found under: Build > Session Properties > Include in Context
If you have your starting point like
http://<IP/Host>:port/context1/context2/
under attack mode.
Keep Include in Context configured as
http://<IP/Host>:port/.*
so as to consider that URL as valid context with anything changing after /.
We are trying to enable SSL connection between Apache Ranger and Kafka cluster. After creating keystore and truststore for both Kafka and Ranger, we are unable to connect Kafka to Ranger and we are getting the following error message:
[2020-06-25 20:47:40,013] ERROR Unable to get the Credential Provider from the Configuration (org.apache.ranger.authorization.hadoop.utils.RangerCredentialProvider)
java.lang.IllegalArgumentException: The value of property hadoop.security.credential.provider.path must not be null
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1115)
at org.apache.ranger.authorization.hadoop.utils.RangerCredentialProvider.getCredentialProviders(RangerCredentialProvider.java:68)
at org.apache.ranger.authorization.hadoop.utils.RangerCredentialProvider.getCredentialString(RangerCredentialProvider.java:46)
at org.apache.ranger.plugin.util.RangerRESTClient.getCredential(RangerRESTClient.java:386)
at org.apache.ranger.plugin.util.RangerRESTClient.getKeyManagers(RangerRESTClient.java:272)
at org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:188)
at org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:176)
at org.apache.ranger.plugin.util.RangerRESTClient.getResource(RangerRESTClient.java:156)
at org.apache.ranger.admin.client.RangerAdminRESTClient.createWebResource(RangerAdminRESTClient.java:275)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:126)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
[2020-06-25 20:47:40,013] ERROR PolicyRefresher(serviceName=KafkaTest): failed to refresh policies. Will continue to use last known version of policies (51) (org.apache.ranger.plugin.util.PolicyRefresher)
java.lang.IllegalArgumentException: TrustManager is not specified
at org.apache.commons.lang.Validate.notNull(Validate.java:192)
at org.apache.ranger.plugin.util.RangerRESTClient.getSSLContext(RangerRESTClient.java:369)
at org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:190)
at org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:176)
at org.apache.ranger.plugin.util.RangerRESTClient.getResource(RangerRESTClient.java:156)
at org.apache.ranger.admin.client.RangerAdminRESTClient.createWebResource(RangerAdminRESTClient.java:275)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:126)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
In Kafka-Ranger plugin, I have configured below properties in install.properites
COMPONENT_INSTALL_DIR_NAME=/home/ec2-user/kafka
POLICY_MGR_URL=https://public-dns-of-ec2:6182
REPOSITORY_NAME=KafkaTest
SSL_KEYSTORE_FILE_PATH=/etc/hadoop/conf/<keystore>.jks
SSL_KEYSTORE_PASSWORD=<password
>
SSL_TRUSTSTORE_FILE_PATH=/etc/hadoop/conf/<truststore>.jks
SSL_TRUSTSTORE_PASSWORD=<password>
Note: We are not using Ambari
Make sure following properties are set:-
xasecure.policymgr.clientssl.keystore.credential.file=jceks://file/{{credential_file}}
xasecure.policymgr.clientssl.truststore.credential.file=jceks://file/{{credential_file}}
xasecure.policymgr.clientssl.truststore=/path/to/truststore
I was hoping someone here could help me out. We are currently evaluating JFrog's Artifactory - Container Registry running as a Docker service and for the life of me I cannot get this thing to work properly with our Active Directory instance. I had it working fine in version 6 but with the release of version 7, decided to start new with the new version.
So I have artifactory-jcr:7.3.2 up and running in our swarm. Go into administration -> security -> LDAP and create a new LDAP settings profile with the following fields:
LDAP URL: ldap://mydc.company.net:389/DC=company,DC=net
User DN Pattern: blank
Email Attribute: mail
Search Filter: (sAMAccountName={0})
Search Base: OU=Company Users
Search Sub-Tree: checked
Manager DN: CN=_svcAccount,OU=Service Accounts,OU=Company Users,DC=company,DC=net
Manager Password: Correct Password
The Manager DN is correct and the password has been verified and tested. I can log in with the service account from any machine and successfully query the directory using ADExplorer and issue a query for my account using only my sAMAccountName which returns my user object. So I know the service account's password is correct, permissions for it are correct and it can successfully issue queries.
But when trying to test an account from the LDAP settings profile page, I get a generic error message popup stating "Error connecting to the LDAP server:"
For the log, I am looking at the /var/opt/artifactory/artifactory-service.log file.
Here's the entry immediately following a failed 'test account' attempt:
2020-04-03T17:16:46.714Z [jfrt ] [ERROR] [7faa71d56a50ef2b] [o.a.s.l.AbstractLdapService:67] [http-nio-8081-exec-4] - Error connecting to the LDAP server:
org.springframework.security.authentication.AuthenticationServiceException: User myuseraccount failed to authenticate
at org.artifactory.security.ldap.ArtifactoryBindAuthenticator.authenticate(ArtifactoryBindAuthenticator.java:166)
at org.artifactory.security.ldap.LdapServiceImpl.testLdapConnection(LdapServiceImpl.java:77)
at org.artifactory.security.SecurityServiceImpl.testLdapConnection(SecurityServiceImpl.java:3193)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:205)
at com.sun.proxy.$Proxy156.testLdapConnection(Unknown Source)
at org.artifactory.ui.rest.service.admin.security.ldap.ldapsettings.TestLdapSettingsService.testLdapConnection(TestLdapSettingsService.java:76)
at org.artifactory.ui.rest.service.admin.security.ldap.ldapsettings.TestLdapSettingsService.execute(TestLdapSettingsService.java:63)
at org.artifactory.rest.common.service.ServiceExecutor.process(ServiceExecutor.java:38)
at org.artifactory.rest.common.resource.BaseResource.runService(BaseResource.java:92)
at org.artifactory.ui.rest.resource.admin.security.ldap.LdapSettingResource.testLdapSetting(LdapSettingResource.java:90)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
2020-04-03T17:16:46.732Z [jfrt ] [ERROR] [7faa71d56a50ef2b] [o.a.s.l.AbstractLdapService:68] [http-nio-8081-exec-4] - Error connecting to the LDAP server:
2020-04-03T17:17:57.524Z [jfrt ] [WARN ] [81a5689d90762c9 ] [o.a.s.l.LdapServiceImpl:179 ] [http-nio-8081-exec-8] - Unexpected exception in LDAP query:for user myuseraccount vid LDAP: [LDAP: error code 49 - 80090308: LdapErr: DSID-0C090446, comment: AcceptSecurityContext error, data 52e, v2580]; nested exception is javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308: LdapErr: DSID-0C090446, comment: AcceptSecurityContext error, data 52e, v2580]
2020-04-03T17:17:57.547Z [jfrt ] [INFO ] [81a5689d90762c9 ] [o.a.s.l.LdapServiceImpl:129 ] [http-nio-8081-exec-8] - Couldn't find user named "myuseraccount" in ADsettings
From the login ui, I try to use my sam account name only I get a message above the login form stating: "Username or password is incorrect"
Here's the log entry that's generated at the time:
2020-04-03T17:05:12.060Z [jfrt ] [WARN ] [77c816e57e51530 ] [o.a.s.l.LdapServiceImpl:179 ] [http-nio-8081-exec-8] - Unexpected exception in LDAP query:for user admin vid LDAP: [LDAP: error code 49 - 80090308: LdapErr: DSID-0C090446, comment: AcceptSecurityContext error, data 52e, v2580]; nested exception is javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308: LdapErr: DSID-0C090446, comment: AcceptSecurityContext error, data 52e, v2580]
I am only using the sam account name for the login, not the user principal name. I am also leaving off the netbios domain name portion of the login. If I try to add in the full SAM Account name to include the domain, "companyname\myuseraccount" I get a Status 500 error page.
Can anyone tell me what I'm doing wrong here?
Thanks for any help!
Shortly after posting this question, I decided to shell into the running jcr container and copy and install the necessary rpm files and to get openldap working. Then used ldapsearch from the container to query our domain controller using the settings I had provided the artifactory UI. And viola! The issue was the bind DN. I thought the the Manager DN form field was supposed to be the full distinguished name of the binding user account used to query the directory but ldapsearch was returning object not found errors.
I changed the binding account to the service account's SAM account name ("_svcAccount") and got a result back. I've since gone back into the artifactory settings and update the Manager DN to be "_svcAccount" and everything is working.
Jfrog should change the description of the Manager DN field. A distinguished name consists of the full LDAP path to the object. Which doesn't work, at least not in my particular situation. Other Java based products we use like SonarQube, use the classic full distinguished name for the bind account. Jfrog Container Registry apparently does not.
-Update -- I ended up having to use the NetBIOS domain as part of the Manager DN account to get it to authenticate. So, instead of "_svcAccount" as the Manager DN, I had to use "mycompany\_svcAccount" as the Manager DN. However, Active Directory users do not use the NetBIOS domain when logging into the Container Registry, just the SAM account name. (i.e. "myAccount" vs. "mycompany\myAccount")
Version : spring-cloud-dataflow-server-yarn-1.2.2.RELEASE
Issue : All OOTB / Custom Task Apps seem to be NOT working with Yarn Deployer (I Specifically tested with timestamp-task-1.3.0.RELEASE and a hello world Custom Task built per the reference doc).
We have a Yarn cluster where all the streams that we have deployed are running fine which rules out any issue with hadoop/yarn cluster. The moment we try to deploy a task, the task exits with code 0 with below message logged in Yarn Container/AppMaster stdout
2018-09-19 18:04:20.782 DEBUG 22625 --- [ask-scheduler-2] o.s.yarn.am.allocate.AbstractAllocator : completed container: container_1536919363436_0805_01_000002 with status=ContainerStatus: [ContainerId: container_1536919363436_0805_01_000002, State: COMPLETE, Diagnostics: Exception from container-launch.
Container id: container_1536919363436_0805_01_000002
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Container exited with a non-zero exit code 1
, ExitStatus: 1, ]
Full AppMaster Log can be found here and corresponding servers.yml can be found here
Any help is Appreciated.
I am answering my own question -- our yarn server had log aggregation enabled and hence container logs weren't displayed immediately and I had to grep through the aggregated logs to find out why custom tasks weren't launching. Once we (temporarily) disabled log aggregation in yarn, custom task's Container.stdout and Container.stderror were visible under log directory configured in yarn-site.xml