ECS fargate deployment got stuck and failed to deploy ECS Service - docker

I am new to ECS and I am trying to deploy it with Cloudformation.
I was making the following cloudformation template by looking at the documentation and some examples I found from blogs and some articles.
However, for some reason, it got stuck in updating one of the resources and eventually timed out.
I am not sure why it gets stuck and fails.
Can someone spot the mistake I am making?
For now, my goal is to deploy and see the app on the internet. I am not really looking for the advanced configuration.
I also pass the ecr url to this upon deployment aws cli.
Thank you in advance.
AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Description: >
ECS Service
Parameters:
Environment:
Type: String
Default: alpha
AllowedValues:
- alpha
- beta
- production
ECRDockerUri:
Type: String
Default: <url for ecr repo>
ContainerPort:
Type: Number
Default: 8080
Resources:
LogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Sub "${Environment}-fake-user-api-logGroup"
RetentionInDays: 30
ECSCluster:
Type: 'AWS::ECS::Cluster'
Properties:
ClusterName: !Sub "${Environment}-MyFargateCluster"
ExecutionRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub "${Environment}-${AWS::AccountId}-ExecutionRole"
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: ecs-tasks.amazonaws.com
Action: 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
ECSService:
Type: AWS::ECS::Service
Properties:
ServiceName: !Sub "${Environment}-${AWS::AccountId}-ECSService"
Cluster: !Ref ECSCluster
TaskDefinition: !Ref TaskDefinition
DesiredCount: 1
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
TaskRoleArn: !Ref ExecutionRole
ContainerDefinitions:
- Name: !Sub "${Environment}-${AWS::AccountId}-Container"
Image: !Ref ECRDockerUri
Memory: 1024
Essential: true
DisableNetworking: false
Privileged: true
ReadonlyRootFilesystem: true
Environment:
- Name: SPRING_PROFILES_ACTIVE
Value: !Ref Environment
PortMappings:
- ContainerPort: !Ref ContainerPort
HostPort: !Ref ContainerPort
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref LogGroup
awslogs-region: ca-central-1

I went through your CFN stack and found some things missing. I noticed that your Cluster name is ENV-MyFargateCluster so I am assuming your goal is to create a fargate service. To run a fargate service, you need to provide the networking configurations and notify that you want to create a Fargate service by specifying the Launch Type. Plus Fargate tasks cannot be Privileged.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions.html#cfn-ecs-taskdefinition-containerdefinition-privileged
Below is my snippet of the code :
AWSTemplateFormatVersion: 2010-09-09
Transform: 'AWS::Serverless-2016-10-31'
Description: |
ECS Service
Parameters:
Environment:
Type: String
Default: alpha
AllowedValues:
- alpha
- beta
- production
ECRDockerUri:
Type: String
Default: 'image'
ContainerPort:
Type: Number
Default: 80
Resources:
LogGroup:
Type: 'AWS::Logs::LogGroup'
Properties:
LogGroupName: !Sub '${Environment}-fake-user-api-logGroup'
RetentionInDays: 30
ECSCluster:
Type: 'AWS::ECS::Cluster'
Properties:
ClusterName: !Sub '${Environment}-MyFargateCluster'
ExecutionRole:
Type: 'AWS::IAM::Role'
Properties:
RoleName: !Sub '${Environment}-${AWS::AccountId}-ExecutionRole'
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: ecs-tasks.amazonaws.com
Action: 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
ECSService:
Type: 'AWS::ECS::Service'
Properties:
ServiceName: !Sub '${Environment}-${AWS::AccountId}-ECSService'
LaunchType: FARGATE
Cluster: !Ref ECSCluster
TaskDefinition: !Ref TaskDefinition
DesiredCount: 1
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- sg-XXXXXXXXXX
Subnets:
- subnet-XXXXXXXXXX
TaskDefinition:
Type: 'AWS::ECS::TaskDefinition'
Properties:
RequiresCompatibilities:
- FARGATE
TaskRoleArn: !Ref ExecutionRole
ExecutionRoleArn: !Ref ExecutionRole
ContainerDefinitions:
- Name: !Sub '${Environment}-${AWS::AccountId}-Container'
Image: !Ref ECRDockerUri
Memory: 1024
Essential: true
DisableNetworking: false
Privileged: false
ReadonlyRootFilesystem: true
Environment:
- Name: SPRING_PROFILES_ACTIVE
Value: !Ref Environment
PortMappings:
- ContainerPort: !Ref ContainerPort
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref LogGroup
awslogs-region: ca-central-1
awslogs-stream-prefix: test
Cpu: '1024'
Memory: '2048'
NetworkMode: awsvpc

Related

Chaincode Build Failed in Hyperledger Fabric on Kubernetes

Deploying Hyperledger Fabric v2.0 in Kubernetes
I am Trying to deploy a sample chaincode in a Private Kubernetes Cluster which is running in Azure Cloud. After creating the nodes and then running the Install chaincode operation is getting failed and throwing the below error. I am only using a single Kubernetes cluster.
Error:
chaincode install failed with status: 500 - failed to invoke backing implementation of 'InstallChaincode': could not build chaincode: docker build failed: docker image inspection failed: cannot connect to Docker endpoint
command terminated with exit code 1
Below is the peer configuration template for Deployment, Service & ConfigMap
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: ${PEER}
name: ${PEER}
namespace: ${ORG}
spec:
replicas: 1
selector:
matchLabels:
app: ${PEER}
strategy: {}
template:
metadata:
labels:
app: ${PEER}
spec:
containers:
- name: couchdb
image: blockchainpractice.azurecr.io/hyperledger/fabric-couchdb
env:
- name: COUCHDB_USER
value: couchdb
- name: COUCHDB_PASSWORD
value: couchdb
ports:
- containerPort: 5984
- name: fabric-peer
image: blockchainpractice.azurecr.io/hyperledger/fabric-peer:2.0
resources: {}
envFrom:
- configMapRef:
name: ${PEER}
volumeMounts:
- name: dockersocket
mountPath: "/host/var/run/docker.sock"
- name: ${PEER}
mountPath: "/etc/hyperledger/fabric-peer"
- name: client-root-tlscas
mountPath: "/etc/hyperledger/fabric-peer/client-root-tlscas"
volumes:
- name: dockersocket
hostPath:
path: "/var/run/docker.sock"
- name: ${PEER}
secret:
secretName: ${PEER}
items:
- key: key.pem
path: msp/keystore/key.pem
- key: cert.pem
path: msp/signcerts/cert.pem
- key: tlsca-cert.pem
path: msp/tlsca/tlsca-cert.pem
- key: ca-cert.pem
path: msp/cacerts/ca-cert.pem
- key: config.yaml
path: msp/config.yaml
- key: tls.crt
path: tls/tls.crt
- key: tls.key
path: tls/tls.key
- key: orderer-tlsca-cert.pem
path: orderer-tlsca-cert.pem
- key: core.yaml
path: core.yaml
- name: client-root-tlscas
secret:
secretName: client-root-tlscas
---
apiVersion: v1
kind: ConfigMap
metadata:
creationTimestamp: null
name: ${PEER}
namespace: ${ORG}
data:
CORE_PEER_ADDRESSAUTODETECT: "true"
CORE_PEER_ID: ${PEER}
CORE_PEER_LISTENADDRESS: 0.0.0.0:7051
CORE_PEER_PROFILE_ENABLED: "true"
CORE_PEER_LOCALMSPID: ${ORG_MSP}
CORE_PEER_MSPCONFIGPATH: /etc/hyperledger/fabric-peer/msp
# Gossip
CORE_PEER_GOSSIP_BOOTSTRAP: peer0.${ORG}:7051
CORE_PEER_GOSSIP_EXTERNALENDPOINT: "${PEER}.${ORG}:7051"
CORE_PEER_GOSSIP_ORGLEADER: "false"
CORE_PEER_GOSSIP_USELEADERELECTION: "true"
# TLS
CORE_PEER_TLS_ENABLED: "true"
CORE_PEER_TLS_CERT_FILE: "/etc/hyperledger/fabric-peer/tls/tls.crt"
CORE_PEER_TLS_KEY_FILE: "/etc/hyperledger/fabric-peer/tls/tls.key"
CORE_PEER_TLS_ROOTCERT_FILE: "/etc/hyperledger/fabric-peer/msp/tlsca/tlsca-cert.pem"
CORE_PEER_TLS_CLIENTAUTHREQUIRED: "false"
ORDERER_TLS_ROOTCERT_FILE: "/etc/hyperledger/fabric-peer/orderer-tlsca-cert.pem"
CORE_PEER_TLS_CLIENTROOTCAS_FILES: "/etc/hyperledger/fabric-peer/client-root-tlscas/tlsca.${ORG}-cert.pem"
CORE_PEER_TLS_CLIENTCERT_FILE: "/etc/hyperledger/fabric-peer/tls/tls.crt"
CORE_PEER_TLS_CLIENTKEY_FILE: "/etc/hyperledger/fabric-peer/tls/tls.key"
# Docker
CORE_PEER_NETWORKID: ${ORG}-fabnet
CORE_VM_ENDPOINT: unix:///host/var/run/docker.sock
CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE: "bridge"
# CouchDB
CORE_LEDGER_STATE_STATEDATABASE: CouchDB
CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS: 0.0.0.0:5984
CORE_LEDGER_STATE_COUCHDBCONFIG_USERNAME: couchdb
CORE_LEDGER_STATE_COUCHDBCONFIG_PASSWORD: couchdb
# Logging
CORE_LOGGING_PEER: "info"
CORE_LOGGING_CAUTHDSL: "info"
CORE_LOGGING_GOSSIP: "info"
CORE_LOGGING_LEDGER: "info"
CORE_LOGGING_MSP: "info"
CORE_LOGGING_POLICIES: "debug"
CORE_LOGGING_GRPC: "info"
GODEBUG: "netdns=go"
---
apiVersion: v1
kind: Service
metadata:
name: ${PEER}
namespace: ${ORG}
spec:
selector:
app: ${PEER}
ports:
- name: request
port: 7051
targetPort: 7051
- name: event
port: 7053
targetPort: 7053
type: LoadBalancer
Can anyone help me out. Thanks in advance
I'd suggest that it would be good to look at the K8S test network deployment in fabric-samples (https://github.com/hyperledger/fabric-samples/tree/main/test-network-k8s)
Note that the classic way the peer creates chaincode is create a new docker container via the docker daemon. This really doesn't sit well with K8S. So the chaincode-as-a-service approach is strongly recommended.

ec2 instance timeout only when ecs cluster tasks are using bridge network interface

I had an ecs cluster running with ec2: I had a service running a nginx task and i had an ec2 autoscaling group, with an ALB in front of then. The task network interface was awsvcp. it worked fine but, as i need to allow dynamic port mapping (for running more than one task per ec2 instance), i changed my settings (now the task uses a bridge interface network and allow dynamic port mapping (host port = 0)). But, since i did that changes, my alb receive 504 (timeout) when try to communicate with the ec2 instances. And i can't even ssh inside the ec2 instance anymore (timeout too). Why this small setting (changing the network interface for dynamic port mapping) messed up my cluster? I suspect is something related to the ec2 instances settings, because i can't even ssh on then anymore. Bellow i pasted the keys settings on my cloudformation template:
LoadBalancer:
Condition: CreateMainResources
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
Scheme: internet-facing
Subnets:
- !Ref PublicSubnet1
- !Ref PublicSubnet2
Type: application
SecurityGroups:
- !Ref ECSSecurityGroup
Listener80:
Condition: CreateMainResources
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
LoadBalancerArn: !Ref LoadBalancer
Port: !Ref ListeningOnPort
Protocol: HTTP
DefaultActions:
- TargetGroupArn: !Ref MyTargetGroup
Type: forward
MyTargetGroup:
Condition: CreateMainResources
DependsOn: LoadBalancer
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Matcher:
HttpCode: 200-499 # 200-499 ou 200,204
Port: !Ref ListeningOnPort
Protocol: HTTP
TargetType: instance # ip
VpcId: !Ref VPC
EC2LaunchTemplate:
Type: AWS::EC2::LaunchTemplate
Condition: CreateEC2Resources
Properties:
LaunchTemplateData:
ImageId: !Ref Ec2ImageId
InstanceType: !Ref InstanceType
IamInstanceProfile:
Arn: !GetAtt EC2InstanceProfile.Arn
Monitoring:
Enabled: true
KeyName: !Ref Key
NetworkInterfaces:
- AssociatePublicIpAddress: true
DeviceIndex: '0'
Groups:
- !GetAtt EC2SecurityGroup.GroupId
SubnetId: !Ref PublicSubnet1
UserData:
Fn::Base64: !Sub
- |
#!/bin/bash
echo ECS_CLUSTER=${cluster_name} >> /etc/ecs/ecs.config
- cluster_name: !Sub ${AWS::StackName}-cluster
EC2SecurityGroup:
Type: "AWS::EC2::SecurityGroup"
Properties:
VpcId: !Ref VPC
SecurityGroupIngress:
- CidrIp: 0.0.0.0/0
FromPort: !Ref ListeningOnPort
IpProtocol: "tcp"
ToPort: !Ref ListeningOnPort
- IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: !Ref SSHUserIP
NginxWebServerTaskDefinition:
Condition: CreateECSResources
Type: AWS::ECS::TaskDefinition
Properties:
ContainerDefinitions:
- Name: !Ref TaskContainerName
Image: !Ref ContainerDefinitionImage
Essential: true
Privileged: false
PortMappings:
- ContainerPort: !Ref ListeningOnPort
HostPort: 0 # !Ref ListeningOnPort
Protocol: tcp
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref LogGroup
awslogs-region: us-east-1
awslogs-stream-prefix: nginx
Cpu: !Ref TaskDefinitionCpu
Memory: !Ref TaskDefinitionMemory
ExecutionRoleArn: !Ref ExecutionRole
Family: !Sub ${AWS::StackName}-nginx-task
NetworkMode: bridge # awsvpc
RequiresCompatibilities:
- EC2
TaskRoleArn: !Ref TaskRole
ECSSecurityGroup:
Condition: CreateMainResources
Type: AWS::EC2::SecurityGroup
Properties:
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: !Ref ListeningOnPort
ToPort: !Ref ListeningOnPort
CidrIp: 0.0.0.0/0
VpcId: !Ref VPC
Service:
Condition: CreateECSResources
DependsOn:
- Listener80
- EC2AutoScalingGroup
Type: AWS::ECS::Service
Properties:
Cluster: !Ref Cluster
CapacityProviderStrategy:
- CapacityProvider: !Ref MainCapacityProvider
Weight: !Ref Weight
TaskDefinition: !Ref NginxWebServerTaskDefinition
SchedulingStrategy: REPLICA
DeploymentConfiguration:
MaximumPercent: 200
MinimumHealthyPercent: 100
DeploymentController:
Type: ECS
PlacementStrategies:
- Type: binpack
Field: memory
DesiredCount: !Ref TaskDefinitionInstantiations
LoadBalancers:
- ContainerName: !Ref TaskContainerName
ContainerPort: !Ref ListeningOnPort
TargetGroupArn: !Ref MyTargetGroup
# NetworkConfiguration: # awsvpc only
# AwsvpcConfiguration:
# Subnets:
# - !Ref PublicSubnet1
# - !Ref PublicSubnet2
# SecurityGroups:
# - !Ref ECSSecurityGroup
The issue was that my ec2 instance must listening on all possible ephemeral host port to it works (because of the dynamic mapping setting), otherwise those port was unreachable and the timeout would be trigger. So, i needed to change my security group settings:
EC2SecurityGroup:
Type: "AWS::EC2::SecurityGroup"
Properties:
VpcId: !Ref VPC
SecurityGroupIngress:
- IpProtocol: tcp
# range de portas efemeras do alb 32768-65535
FromPort: !If [DynamicPortMapping, 32768, !Ref ListeningOnPort]
ToPort: !If [DynamicPortMapping, 65535, !Ref ListeningOnPort ]
SourceSecurityGroupId: !Ref ECSSecurityGroup
- IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: !Ref SSHUserIP
references:
https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_PortMapping.html
https://aws.amazon.com/premiumsupport/knowledge-center/dynamic-port-mapping-ecs/
https://www.youtube.com/watch?v=cmRZleI18Yg ( 4:52 - 5:12 is the key moment)

Jenkins on Kubernetes - working directory not accessible using workspaceVolume dynamicPVC

I'm running Jenkins on EKS cluster with k8s plugin and i'd like to write a declarative pipeline in which I specify the pod template in each stage. So a basic example would be the following, in which in the first stage a file is created and in the second one is printed :
pipeline{
agent none
stages {
stage('First sample') {
agent {
kubernetes {
label 'mvn-pod'
yaml """
spec:
containers:
- name: maven
image: maven:3.3.9-jdk-8-alpine
"""
}
}
steps {
container('maven'){
sh "echo 'hello' > test.txt"
}
}
}
stage('Second sample') {
agent {
kubernetes {
label 'bysbox-pod'
yaml """
spec:
containers:
- name: busybox
image: busybox
"""
}
}
steps {
container('busybox'){
sh "cat test.txt"
}
}
}
}
}
This clearly doesn't work since the two pods don't have any kind of shared memory. Reading this doc I realized I can use workspaceVolume dynamicPVC () in the yaml declaration of the pod so that the plugin creates and manages a persistentVolumeClaim in which hopefully i can write the data I need to share between stages.
Now, with workspaceVolume dynamicPVC (...) both pv and pvc are successfully created but the pod goes on error and terminates. In particular, the pods provisioned is the following :
apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/psp: eks.privileged
runUrl: job/test-libraries/job/sample-k8s/12/
creationTimestamp: "2020-08-07T08:57:09Z"
deletionGracePeriodSeconds: 30
deletionTimestamp: "2020-08-07T08:58:09Z"
labels:
jenkins: slave
jenkins/label: bibibu
name: bibibu-ggb5h-bg68p
namespace: jenkins-slaves
resourceVersion: "29184450"
selfLink: /api/v1/namespaces/jenkins-slaves/pods/bibibu-ggb5h-bg68p
uid: 1c1e78a5-fcc7-4c86-84b1-8dee43cf3f98
spec:
containers:
- image: maven:3.3.9-jdk-8-alpine
imagePullPolicy: IfNotPresent
name: maven
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
tty: true
volumeMounts:
- mountPath: /home/jenkins/agent
name: workspace-volume
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-5bt8c
readOnly: true
- env:
- name: JENKINS_SECRET
value: ...
- name: JENKINS_AGENT_NAME
value: bibibu-ggb5h-bg68p
- name: JENKINS_NAME
value: bibibu-ggb5h-bg68p
- name: JENKINS_AGENT_WORKDIR
value: /home/jenkins/agent
- name: JENKINS_URL
value: ...
image: jenkins/inbound-agent:4.3-4
imagePullPolicy: IfNotPresent
name: jnlp
resources:
requests:
cpu: 100m
memory: 256Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /home/jenkins/agent
name: workspace-volume
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-5bt8c
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: ...
nodeSelector:
kubernetes.io/os: linux
priority: 0
restartPolicy: Never
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: workspace-volume
persistentVolumeClaim:
claimName: pvc-bibibu-ggb5h-bg68p
- name: default-token-5bt8c
secret:
defaultMode: 420
secretName: default-token-5bt8c
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2020-08-07T08:57:16Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2020-08-07T08:57:16Z"
message: 'containers with unready status: [jnlp]'
reason: ContainersNotReady
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2020-08-07T08:57:16Z"
message: 'containers with unready status: [jnlp]'
reason: ContainersNotReady
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2020-08-07T08:57:16Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://9ed5052e9755ee4f974704fa4b74f2d89702283a4437e60a9945cf4ec7d6da68
image: jenkins/inbound-agent:4.3-4
imageID: docker-pullable://jenkins/inbound-agent#sha256:62f48a12d41e02e557ee9f7e4ffa82c77925b817ec791c8da5f431213abc2828
lastState: {}
name: jnlp
ready: false
restartCount: 0
state:
terminated:
containerID: docker://9ed5052e9755ee4f974704fa4b74f2d89702283a4437e60a9945cf4ec7d6da68
exitCode: 1
finishedAt: "2020-08-07T08:57:35Z"
reason: Error
startedAt: "2020-08-07T08:57:35Z"
- containerID: docker://96f747a132ee98f7bf2488bd3cde247380aea5dd6f84bdcd7e6551dbf7c08943
image: maven:3.3.9-jdk-8-alpine
imageID: docker-pullable://maven#sha256:3ab854089af4b40cf3f1a12c96a6c84afe07063677073451c2190cdcec30391b
lastState: {}
name: maven
ready: true
restartCount: 0
state:
running:
startedAt: "2020-08-07T08:57:35Z"
hostIP: 10.108.171.224
phase: Running
podIP: 10.108.171.158
qosClass: Burstable
startTime: "2020-08-07T08:57:16Z"
Retrieving logs from jnlp container on the pod with kubectl logs name-of-the-pod -c jnlp -n jenkins-slaves led me towards this error :
Exception in thread "main" java.io.IOException: The specified working directory should be fully accessible to the remoting executable (RWX): /home/jenkins/agent
at org.jenkinsci.remoting.engine.WorkDirManager.verifyDirectory(WorkDirManager.java:249)
at org.jenkinsci.remoting.engine.WorkDirManager.initializeWorkDir(WorkDirManager.java:201)
at hudson.remoting.Engine.startEngine(Engine.java:288)
at hudson.remoting.Engine.startEngine(Engine.java:264)
at hudson.remoting.jnlp.Main.main(Main.java:284)
at hudson.remoting.jnlp.Main._main(Main.java:279)
at hudson.remoting.jnlp.Main.main(Main.java:231)
I also tried to specify the accessModes as parameter of dynamicPVC, but the error is the same.
What am I doing wrong?
Thanks
The docker image being used is configured to run as a non-root user jenkins. By default PVCs will be created only allowing root-user access.
This can be configured using the security context, e.g.
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
(The jenkins user in that image is ID 1000)

How to deploy elasticsearch in kubernetes established by AWS EKS [duplicate]

I am running my kubernetes cluster on AWS EKS which runs kubernetes 1.10.
I am following this guide to deploy elasticsearch in my Cluster
elasticsearch Kubernetes
The first time I deployed it everything worked fine. Now, When I redeploy it gives me the following error.
ERROR: [2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2018-08-24T18:07:28,448][INFO ][o.e.n.Node ] [es-master-6987757898-5pzz9] stopping ...
[2018-08-24T18:07:28,534][INFO ][o.e.n.Node ] [es-master-6987757898-5pzz9] stopped
[2018-08-24T18:07:28,534][INFO ][o.e.n.Node ] [es-master-6987757898-5pzz9] closing ...
[2018-08-24T18:07:28,555][INFO ][o.e.n.Node ] [es-master-6987757898-5pzz9] closed
Here is my deployment file.
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: es-master
labels:
component: elasticsearch
role: master
spec:
replicas: 3
template:
metadata:
labels:
component: elasticsearch
role: master
spec:
initContainers:
- name: init-sysctl
image: busybox:1.27.2
command:
- sysctl
- -w
- vm.max_map_count=262144
securityContext:
privileged: true
containers:
- name: es-master
image: quay.io/pires/docker-elasticsearch-kubernetes:6.3.2
env:
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: CLUSTER_NAME
value: myesdb
- name: NUMBER_OF_MASTERS
value: "2"
- name: NODE_MASTER
value: "true"
- name: NODE_INGEST
value: "false"
- name: NODE_DATA
value: "false"
- name: HTTP_ENABLE
value: "false"
- name: ES_JAVA_OPTS
value: -Xms512m -Xmx512m
- name: NETWORK_HOST
value: "0.0.0.0"
- name: PROCESSORS
valueFrom:
resourceFieldRef:
resource: limits.cpu
resources:
requests:
cpu: 0.25
limits:
cpu: 1
ports:
- containerPort: 9300
name: transport
livenessProbe:
tcpSocket:
port: transport
initialDelaySeconds: 20
periodSeconds: 10
volumeMounts:
- name: storage
mountPath: /data
volumes:
- emptyDir:
medium: ""
name: "storage"
I have seen a lot of posts talking about increasing the value but I am not sure how to do it. Any help would be appreciated.
Just want to append to this issue:
If you create EKS cluster by eksctl then you can append to NodeGroup creation yaml:
preBootstrapCommand:
- "sed -i -e 's/1024:4096/65536:65536/g' /etc/sysconfig/docker"
- "systemctl restart docker"
This will solve the problem for newly created cluster by fixing docker daemon config.
Update default-ulimit parameter in the file '/etc/docker/daemon.json'
"default-ulimits": {
"nofile": {
"Name": "nofile",
"Soft": 65536,
"Hard": 65536
}
}
and restart docker daemon.
This is the only thing that worked for me using EKS setting up an EFK stack. Add this to your nodegroup creation YAML file under nodeGroups:. Then create your nodegroup and apply your ES pods on it.
preBootstrapCommands:
- "sysctl -w vm.max_map_count=262144"
- "systemctl restart docker"

Kubernetes: creating pod with specific environments

I tried to create a pod with a particular environment for uwsgi configuration , but it was this message :
failed to load "phptime.yml": JSON: I can not unpack the number in the value of the string type Go
when I tried to run this command :
kubectl create -f phptime.yml
I found that trouble in environments that has names like this:
UWSGI_HTTP-MODIFIER1
or
UWSGI_PHP-SAPI-NAME
or
UWSGI_MASTER-AS-ROOT
but with environments that has a next names all ok:
UWSGI_HTTP
or
UWSGI_INCLUDE
A lot of our containers took configuration from environments and I need include all of my conf environments. This is my rc conf:
containers:
- name: phptime
image: ownregistry/phpweb:0.5
env:
- name: UWSGI_UID
value: go
- name: UWSGI_GID
value: go
- name: UWSGI_INCLUDE
value: /var/lib/go-agent/pipelines/test/test-dev0/.uwsgi_dev.ini
- name: UWSGI_PHP-SAPI-NAME
value: apache
- name: UWSGI_HTTP
value: :8086
- name: UWSGI_HTTP-MODIFIER1
value: 14
- name: UWSGI_PIDFILE
value: '/tmp/uwsgi.pid'
- name: UWSGI_MASTER-FIFO
value: '/tmp/fifo0'
- name: UWSGI_MASTER-AS-ROOT
value: 'true'
- name: UWSGI_MASTER
value: 'true'
ports:
- containerPort: 8086
resources:
limits:
cpu: 500m
memory: 200Mi
requests:
cpu: 500m
memory: 200Mi
volumeMounts:
- mountPath: /var/lib/go-agent/pipelines/test/test-dev0/
name: site
readOnly: true
volumes:
- hostPath:
path: /home/user/www/
name: site
Is this kubernetes issue or it`s my? How to solve this? Thanks!
You must quote all of the values that you want to set as environment variables that the yaml parser might interpret as a non-string type.
For example, in influxdb-grafana-controller.yaml the values true and false are quoted because they could be interpreted as booleans. The same constraint applies to purely numerical values.

Resources