ECS Agent starting Docker container doesn't complete. Manual start successful - docker

Question

I followed the ECS Getting Started tutorial but the ECS Agent isn't getting the container started. When I start the image manually on the same instance it starts fine.
The image is a Spring Boot web application with a single endpoint on / that returns the string "Hello world!!". The container runs fine locally, and also runs fine on a CentOS EC2 instance I've created. The endpoint is available publicly when I run the docker image on the CentOS EC2 instance.
The ECS Instance has security groups created by the wizard and has port 80 open. I added port 22 for SSH access.
When I SSH into the ECS instance to see the Docker logs for my container, and it looks like it's hanging during the entrypoint execution.
Here are the Docker logs for the hanging instance:
[ec2-user#ip-10-0-0-156 ~]$ docker logs --tail 100 107d4cf04dd8
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.4.2.RELEASE)
2016-11-25 17:36:22.505 INFO 1 --- [ main] ecstest.Application : Starting Application on 107d4cf04dd8 with PID 1 (/ecstest-1.0-SNAPSHOT.jar started by root in /)
2016-11-25 17:36:22.546 INFO 1 --- [ main] ecstest.Application : No active profile set, falling back to default profiles: default
2016-11-25 17:36:23.059 INFO 1 --- [ main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#6d21714c: startup date [Fri Nov 25 17:36:23 UTC 2016]; root of context hierarchy
2016-11-25 17:36:30.972 INFO 1 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat initialized with port(s): 8080 (http)
2016-11-25 17:36:31.014 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service Tomcat
2016-11-25 17:36:31.016 INFO 1 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet Engine: Apache Tomcat/8.5.6
2016-11-25 17:36:31.464 INFO 1 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2016-11-25 17:36:31.464 INFO 1 --- [ost-startStop-1] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 8458 ms
At first it seems like an application error in my container image, but when I stop the docker process and run the same image manually, the output is as expected and I can reach my endpoint from outside the instance as expected.
[ec2-user#ip-10-0-1-124 ~]$ docker stop -t 1 4d2401d7db93 && docker run -p 80:8080 -d ############.dkr.ecr.us-west-2.amazonaws.com/ecstest
4d2401d7db93
db8cffa89995401d9314d7d70e954f09c7fde972a5e6a423615827d8c47b9d10
[ec2-user#ip-10-0-1-124 ~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
db8cffa89995 ############.dkr.ecr.us-west-2.amazonaws.com/ecstest "java -jar ecstest-1." 10 seconds ago Up 9 seconds 0.0.0.0:80->8080/tcp small_gates
85bd18480c99 amazon/amazon-ecs-agent:latest "/agent" 11 minutes ago Up 11 minutes ecs-agent
[ec2-user#ip-10-0-1-124 ~]$ docker logs --tail 1000 db8cffa89995
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.4.2.RELEASE)
2016-11-25 18:06:57.960 INFO 1 --- [ main] ecstest.Application : Starting Application on db8cffa89995 with PID 1 (/ecstest-1.0-SNAPSHOT.jar started by root in /)
2016-11-25 18:06:58.004 INFO 1 --- [ main] ecstest.Application : No active profile set, falling back to default profiles: default
2016-11-25 18:06:58.578 INFO 1 --- [ main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#6d21714c: startup date [Fri Nov 25 18:06:58 UTC 2016]; root of context hierarchy
2016-11-25 18:07:05.784 INFO 1 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat initialized with port(s): 8080 (http)
2016-11-25 18:07:05.866 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service Tomcat
2016-11-25 18:07:05.876 INFO 1 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet Engine: Apache Tomcat/8.5.6
2016-11-25 18:07:06.283 INFO 1 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2016-11-25 18:07:06.283 INFO 1 --- [ost-startStop-1] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 7753 ms
2016-11-25 18:07:07.026 INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.ServletRegistrationBean : Mapping servlet: 'dispatcherServlet' to [/]
2016-11-25 18:07:07.031 INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'characterEncodingFilter' to: [/*]
2016-11-25 18:07:07.031 INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2016-11-25 18:07:07.032 INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'httpPutFormContentFilter' to: [/*]
2016-11-25 18:07:07.033 INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'requestContextFilter' to: [/*]
2016-11-25 18:07:08.432 INFO 1 --- [ main] s.w.s.m.m.a.RequestMappingHandlerAdapter : Looking for #ControllerAdvice: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#6d21714c: startup date [Fri Nov 25 18:06:58 UTC 2016]; root of context hierarchy
2016-11-25 18:07:08.786 INFO 1 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[],methods=[GET]}" onto public java.lang.String ecstest.Application.get()
2016-11-25 18:07:08.800 INFO 1 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2016-11-25 18:07:08.801 INFO 1 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2016-11-25 18:07:09.036 INFO 1 --- [ main] o.s.w.s.handler.SimpleUrlHandlerMapping : Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2016-11-25 18:07:09.036 INFO 1 --- [ main] o.s.w.s.handler.SimpleUrlHandlerMapping : Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2016-11-25 18:07:09.204 INFO 1 --- [ main] o.s.w.s.handler.SimpleUrlHandlerMapping : Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2016-11-25 18:07:09.893 INFO 1 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup
2016-11-25 18:07:10.201 INFO 1 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8080 (http)
2016-11-25 18:07:10.216 INFO 1 --- [ main] ecstest.Application : Started Application in 14.385 seconds (JVM running for 16.522)
Any ideas why the ECS Agent isn't getting my application started?
Task Definition JSON
{
"attributes": null,
"requiresAttributes": [
{
"value": null,
"name": "com.amazonaws.ecs.capability.ecr-auth",
"targetId": null,
"targetType": null
}
],
"taskDefinitionArn": "arn:aws:ecs:us-west-2:############:task-definition/DcTaskDefinition:4",
"networkMode": "bridge",
"status": "ACTIVE",
"revision": 4,
"taskRoleArn": null,
"containerDefinitions": [
{
"volumesFrom": [],
"memory": 128,
"extraHosts": null,
"dnsServers": null,
"disableNetworking": null,
"dnsSearchDomains": null,
"portMappings": [
{
"hostPort": 80,
"containerPort": 8080,
"protocol": "tcp"
}
],
"hostname": null,
"essential": true,
"entryPoint": null,
"mountPoints": [],
"name": "DcContainer",
"ulimits": null,
"dockerSecurityOptions": null,
"environment": [],
"links": null,
"workingDirectory": null,
"readonlyRootFilesystem": null,
"image": "############.dkr.ecr.us-west-2.amazonaws.com/ecstest:latest",
"command": null,
"user": null,
"dockerLabels": null,
"logConfiguration": null,
"cpu": 0,
"privileged": null,
"memoryReservation": null
}
],
"placementConstraints": [],
"volumes": [],
"family": "DcTaskDefinition"
}

Answer 1

The memory key in the Task Definition JSON imposes a hard memory limit on the container. When a container tries to exceed that limit, the Docker daemon is supposed to kill it.
I'm not sure whether this can cause your container to get "stuck", however that's the only important difference I see between how your container runs when ECS runs it and how it runs from the command line.
So, I would try to set the memory value to at least 300, or use the memoryReservation key instead which imposes a soft memory limit.
More information on the difference between hard and soft memory limits can be found in the official ECS documentation.