Using a docker container for testing in Atlassian Bitbucket pipeline - docker

I'm using a dynamodb docker container to run some tests in an Atlassian Bitbucket pipeline. These steps work locally with the same exact docker run command, but for some reason I cannot connect to the db container after it starts while running in the pipeline:
image: python:3.6
pipelines:
default:
- step:
caches:
- docker
script:
- docker run -d -p 8000:8000 --name dynamodb --entrypoint java amazon/dynamodb-local -jar DynamoDBLocal.jar -sharedDb -inMemory
- curl http://localhost:8000
services:
- docker
The curl command returns:
curl http://localhost:8000 % Total % Received % Xferd Average Speed
Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0
0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:-
-:-- 0curl: (56) Recv failure: Connection reset by peer
I've tried with both localhost and dynamodb as the host names with the same result. I've also posted this on the Atlassian Community, but got no answers.

You should not start amazon/dynamodb-local manually, you should use services instead:
definitions:
services:
dynamodb-local:
image: amazon/dynamodb-local
memory: 2048
pipelines:
default:
- step:
image: python:3.6
size: 2x
services:
- dynamodb-local
script:
- export DYNAMODB_LOCAL_URL=http://localhost:8000
- export AWS_DEFAULT_REGION=us-east-1
- export AWS_ACCESS_KEY_ID=''
- export AWS_SECRET_ACCESS_KEY=''
- aws --endpoint-url ${DYNAMODB_LOCAL_URL} dynamodb delete-table --table-name test || true
- aws --endpoint-url ${DYNAMODB_LOCAL_URL} dynamodb create-table --cli-input-json file://test.table.json
- python -m unittest test_module.TestClass
You'll probably need to double the size of container and memory as DynamoDB is pretty heavvweight (but it may work on defaults as well).

Related

An error occurred (InvalidArgument) when calling the ListBuckets operation: S3 API Requests must be made to API port

We are using Minio for local testing of S3 AND we have created docker-compose file with Minio and our app dependency is as follows:
Docker-Compose File:
version: "2.1"
services:
minio:
image: minio/minio
container_name: minio
ports:
- 9001:9001
volumes:
- minio_storage:/data
environment:
MINIO_ACCESS_KEY: minio
MINIO_SECRET_KEY: minio123
MINIO_REGION: us-east-1
command: server /data --console-address ":9001"
mem_limit: 512m
populate-minio-data:
container_name: "minio-data"
image: minio/mc
volumes:
- ./hello.txt:/tmp/hello.txt
entrypoint: >
/bin/sh -c "
/usr/bin/mc config host rm local;
/usr/bin/mc config host add --quiet --api s3v4 local http://minio:9001 minio minio123;
/usr/bin/mc mb --quiet local/somebucketname1/;
/usr/bin/mc policy set public local/somebucketname1;
/usr/bin/mc cp /tmp/hello.txt local/somebucketname1/hello.txt;
"
depends_on:
- minio
archive-api-app:
image: openjdk:11
container_name: "archive-api-app"
ports:
- 8091:6001
volumes:
- /home/apcuser/dev/projects/ea-archive-service-v2/projects/application/archive-api:/app
command: [ 'java', '-jar', '/app/build/libs/archive-api-1.0.0.jar' ]
env_file:
- ./vars/default.env
volumes:
minio_storage:
And In java code, I have configured MINIO URL as S3 Endpoint as follows:
#Bean
public AmazonS3 getS3Client() {
return AmazonS3ClientBuilder.standard()
.withClientConfiguration(new ClientConfiguration().withMaxConnections(maxConnections)
.withConnectionTimeout(connectionTimeout).withMaxErrorRetry(maxRetry))
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("http://minio:9001", "us-east-1"))
.build();
}
Once, I run docker-compose file,I am able to see minio-ui in my local Linux machine as follows:
But I am not seeing any data in Minio, instead, I am seeing the below error while uploading data in minio:
Attaching to minio-data
minio-data | Removed `local` successfully.
minio-data | Added `local` successfully.
minio-data | mc: <ERROR> Unable to make bucket `local/somebucketname1/`. S3 API Requests must be made to API port.
minio-data | mc: <ERROR> Unable to set policy `public` for `local/somebucketname1`. S3 API Requests must be made to API port.
minio-data | `/tmp/hello.txt` -> `local/somebucketname1/hello.txt`
minio-data | mc: <ERROR> Failed to copy `/tmp/hello.txt`. S3 API Requests must be made to API port.
minio-data | Total: 0 B, Transferred: 0 B, Speed: 0 B/s
Even same error I am seeing When I am trying to list MINIO data from my local linux host machine:
export AWS_ACCESS_KEY_ID=minio
export AWS_SECRET_ACCESS_KEY=minio123
export AWS_REGION=us-east-1
aws --endpoint-url http://127.0.0.1:9001 s3 ls
**An error occurred (InvalidArgument) when calling the ListBuckets operation: S3 API Requests must be made to API port.**
Can anyone help here, please?
The error message indicates that you need to use the API port instead of console port while using mc.
/usr/bin/mc config host add --quiet --api s3v4 local http://minio:9001 minio minio123;
You need to use port 9000 instead of 9001.

Browser request hanging when running the latest Docker ColdFusion image

I installed the Docker release of ColdFusion with the following command:
docker pull eaps-docker-coldfusion.bintray.io/cf/coldfusion:latest
I then created a compose file:
version: "3.7"
services:
coldfusion:
image: eaps-docker-coldfusion.bintray.io/cf/coldfusion:latest
ports:
- "8500:8500"
networks:
coldfusion:
hostname: coldfusion
volumes:
- "~/dev/docker/projects/coldfusion/volumes/app:/app"
- "~/dev/docker/projects/coldfusion/volumes/logs:/opt/coldfusion/cfusion/logs"
environment:
acceptEULA: "YES"
password: "ColdFusion123"
enableSecureProfile: "false"
HOST_USER_ID: ${CURRENT_UID}
HOST_GROUP_ID: ${CURRENT_GID}
deploy:
replicas: 1
restart_policy:
condition: any
delay: 5s
max_attempts: 3
window: 10s
healthcheck:
test: curl --fail http://localhost:8500 || exit 1
interval: 1m
timeout: 3s
retries: 3
networks:
coldfusion:
name: coldfusion
common:
external: true
name: common
and started it with the command:
docker stack deploy --compose-file docker-compose-dev.yml coldfusion
The log shows:
stephane#stephane-pc:~/dev/docker/projects/coldfusion$ docker service logs -f coldfusion_coldfusion
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | Updating webroot to /app
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | Configuring virtual directories
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | Updating password
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | Skipping language updation
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | Serial Key: Not Provided
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | Previous Serial Key: Not Provided
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | Starting ColdFusion
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | Starting ColdFusion 2018 server ...
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | The ColdFusion 2018 server is starting up and will be available shortly.
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | ======================================================================
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | ColdFusion 2018 server has been started.
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | ColdFusion 2018 will write logs to /opt/coldfusion/cfusion/bin/../logs/coldfusion-out.log
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | ======================================================================
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | [000] Checking server startup status...
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | % Total % Received % Xferd Average Speed Time Time Time Current
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | External Addons: Disabled
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | External Session Storage: Disabled
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | Skipping setup script invocation
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | Secure Profile: Disabled
coldfusion_coldfusion.1.rlhixv3jctvm#stephane-pc | Cleaning up setup directories
But it hangs when typing the http://localhost:8500/ request in the browser.
The log remains empty:
tail -f volumes/logs/coldfusion-out.log
I created an index.cfm page in the /app directory:
hi
<cfset firstName = "World">
Hello <cfoutput>#firstName#</cfoutput>!
This CFML tutorial was designed for
<cfif firstName eq "World">
you!
<cfelse>
the world to see.
</cfif>
UPDATE: A 200 response comes back fine when using 127.0.0.1 instead of localhost
Opening the firewall ports does not change anything to the issue:
stephane#stephane-pc:~$ sudo ufw allow from 127.0.0.0 to any port 8500;
Rules updated
stephane#stephane-pc:~$ sudo ufw allow from any to any port 8500;
Rules updated
Rules updated (v6)
My host /etc/hosts file contains the line:
127.0.0.1 localhost
The nmap command responds:
stephane#stephane-pc:~/dev/docker/projects/coldfusion$ nmap -p 8500 localhost
Starting Nmap 7.80 ( https://nmap.org ) at 2020-09-08 12:09 CEST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00038s latency).
PORT STATE SERVICE
8500/tcp open fmtp
Nmap done: 1 IP address (1 host up) scanned in 0.22 seconds
Your docker compose file shows
hostname: coldfusion
so shouldn't it be available at http://coldfusion:8500?
If it's docker compose v3, it should be
services:
dns:
hostname: 'your-domain'

How to POST a file using Curl from Jenkins to a Go server across a Docker net bridge?

As a part of a Jenkins freestyle project I am attempting to execute the following Post Step shell command:
curl -X POST -i -F "SimpleTest=#target/surefire-reports/TEST-junitfaq.SimpleTest.xml" 127.0.0.1:9090/
This request is targeting the following, simple, Go server:
package main
import (
"fmt"
"github.com/gorilla/mux"
"log"
"net/http"
)
func main() {
router := mux.NewRouter()
router.
Path("/").
Methods("POST").
HandlerFunc(UploadFile)
fmt.Println("Starting...")
log.Fatal(http.ListenAndServe(":9090", router))
}
func UploadFile(w http.ResponseWriter, r *http.Request) {
(w).Header().Set("Access-Control-Allow-Origin", "*")
err := r.ParseForm()
if err != nil {
w.Write([]byte(err.Error()))
return
}
file, _, err := r.FormFile("SimpleTest")
defer file.Close()
if err != nil {
w.Write([]byte(err.Error()))
return
}
bytes := make([]byte, 100)
n, err := file.Read(bytes)
if err != nil {
w.Write([]byte(err.Error()))
return
}
fmt.Printf("%d bytes: %s\n", n, string(bytes[:n]))
}
Both Jenkins and the Go server are running in Docker containers created using the following Docker-compose file:
version: '3'
services:
foobar:
container_name: foobar
image: foobar:latest
hostname: foobar
build:
context: .
dockerfile: Dockerfile
networks:
- foobar-net
ports:
- 9090:9090
labels:
kompose.service.type: LoadBalancer
jenkins:
container_name: jenkins
image: jenkinsci/blueocean
restart: always
hostname: foobar
ports:
- 7070:8080
networks:
- foobar-net
depends_on:
- foobar
links:
- foobar
volumes:
- $PWD/data/jenkins:/var/jenkins_home
networks:
foobar-net:
driver: bridge
When I execute this job, the Curl command fails and I am returned this result:
curl -X POST -i -F 'SimpleTest=#target/surefire-reports/TEST-junitfaq.SimpleTest.xml' 127.0.0.1:9090/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to 127.0.0.1 port 9090: Connection refused
Build step 'Execute shell' marked build as failure
What could be the issue that's causing my connection to be refused?
In the Jenkins container, curl to 127.0.0.1:9090 will attempt to connect to a service inside the same container, but it looks like you're running this service in another container. You'll probably want to curl to foobar:9090

docker-compose can't connect to adjacent service via service name

I have this docker-compose.yml that basically builds my project for e2e test. It's composed of a postgres db, a backend Node app, a frontend Node app, and a spec app which runs the e2e test using cypress.
version: '3'
services:
database:
image: 'postgres'
backend:
build: ./backend
command: /bin/bash -c "sleep 3; yarn backpack dev"
depends_on:
- database
frontend:
build: ./frontend
command: /bin/bash -c "sleep 15; yarn nuxt"
depends_on:
- backend
spec:
build:
context: ./frontend
dockerfile: Dockerfile.e2e
command: /bin/bash -c "sleep 30; yarn cypress run"
depends_on:
- frontend
- backend
The Dockerfiles are just simple Dockerfiles that based off node:8 which copies the project files and run yarn install. In the spec Dockerfile, I pass http://frontend:3000 as FRONTEND_URL.
But this setup fails at the spec command when my cypress runner can't connect to frontend with error:
spec_1 | > Error: connect ECONNREFUSED 172.20.0.4:3000
As you can see, it resolves the hostname frontend to the IP correctly, but it's not able to connect. I'm scratching my head over why can't I connect to the frontend with the service name. If I switch the command on spec to do sleep 30; ping frontend, it's successfully pinging the container. I've tried deleting and let docker-compose recreate the network, I've tried specifying expose and links to the services respectively. All to no success.
I've set up a sample repo here if you wanna try replicating the issue:
https://github.com/afifsohaili/demo-dockercompose-network
Any help is greatly appreciated! Thank you!
Your application is listening on loopback:
$ docker run --rm --net container:demo-dockercompose-network_frontend_1 nicolaka/netshoot ss -lnt
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 127.0.0.11:35233 *:*
LISTEN 0 128 127.0.0.1:3000 *:*
From outside of the container, you cannot connect to ports that are only listening on loopback (127.0.0.1). You need to reconfigure your application to listen on all interfaces (0.0.0.0).
For your app, in the package.json, you can add (according to the nuxt faq):
"config": {
"nuxt": {
"host": "0.0.0.0",
"port": "3000"
}
},
Then you should see:
$ docker run --rm --net container:demo-dockercompose-network_frontend_1 nicolaka/netshoot ss -lnt
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:3000 *:*
LISTEN 0 128 127.0.0.11:39195 *:*
And instead of an unreachable error, you'll now get a 500:
...
frontend_1 | response: undefined,
frontend_1 | statusCode: 500,
frontend_1 | name: 'NuxtServerError' }
...
spec_1 | The response we received from your web server was:
spec_1 |
spec_1 | > 500: Server Error

Docker + webdevops: "This site can’t be reached"

On a current project I use Docker. I must clarify that I am pretty inexperienced at it.
My project is a PHP/Symfony project. Until then, I used nginx:alpine and phpdocker/php-fpm to have my project running on my dev environment. However, I found these unfit to my case as my production actually uses Apache.
I found another project I'm on uses the webdevops Docker images without trouble. I want to replace the two containers listed above with a single one, the webdevops/php-apache-dev:alpine docker image.
Although the configuration between the two projects seems almost identical, my dev environment does not seem to work properly: I end up with this:
This site can’t be reached - 172.18.0.7 refused to connect.
(I also use Traefik, but the routed URI does not work any better. The error message is slightly different though: Bad Gateway).
I find myself unable to debug this. I don't even know where to look.
Below is the docker-compose.yml configuration I want to use:
version: '3.2'
services:
app:
image: webdevops/php-apache-dev:alpine
container_name: my-app
working_dir: /app
env_file: .env
environment:
WEB_DOCUMENT_ROOT: /public
WEB_DOCUMENT_INDEX: index.php
LOG_STDOUT: ./var/log/app.stdout.log
LOG_STDERR: ./var/log/app.stderr.log
# #todo list of unwanted PHP modules, cf. https://dockerfile.readthedocs.io/en/latest/content/DockerImages/dockerfiles/php-apache-dev.html#php-modules
# PHP_DISMOD:
php.error_reporting: E_ALL
PHP_DISPLAY_ERRORS: 1
PHP_POST_MAX_SIZE: 80M
PHP_UPLOAD_MAX_FILESIZE: 200M
PHP_MEMORY_LIMIT: 521M
PHP_MAX_EXECUTION_TIME: 300
PHP_DATE_TIMEZONE: Europe/Paris
volumes:
- .:/app
# - ./docker/apache2/conf.d:/opt/docker/etc/httpd/conf.d
- ~/.ssh:/home/application/.ssh:ro
- ~/.composer:/home/application/.composer
depends_on:
- elasticsearch
- database
The other containers work just as well as they did before. This one is the only one that fails.
When calling docker-compose up no error is thrown. All the logs I could find within the container remain silent. As far I as can tell, Traefik does not seem to be the problem. Here is the result of docker ps:
[/var/www/html/citizen-game]$ docker ps *[master]
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6e9639e7a84d webdevops/php-apache-dev:alpine "/entrypoint supervi…" 4 hours ago Up 4 hours 80/tcp, 443/tcp, 9000/tcp my-app-app
be1b90fdf768 docker.elastic.co/elasticsearch/elasticsearch:6.2.4 "/usr/local/bin/dock…" 4 hours ago Up 4 hours (healthy) 9200/tcp, 9300/tcp my-app-elasticsearch
76fb8743a12f phpmyadmin/phpmyadmin "/run.sh supervisord…" 4 hours ago Up 4 hours 80/tcp, 9000/tcp my-app-phpmyadmin
dd41b4afe267 mysql:5.7 "docker-entrypoint.s…" 4 hours ago Up 4 hours (healthy) 3306/tcp, 33060/tcp my-app-database
91893783bcb1 rabbitmq:3.7-management "docker-entrypoint.s…" 4 hours ago Up 4 hours 4369/tcp, 5671/tcp, 0.0.0.0:5672->5672/tcp, 15671/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp my-app-rabbitmq
63f551884bbf traefik:maroilles "/traefik --web --do…" 4 hours ago Up 4 hours 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:8080->8080/tcp
My question is, I guess: how can I debug this? Am I missing something trivial?
Edit
Here is (part of) the content of the docker-compose.override.yml file:
version: '3.2'
services:
app:
volumes:
- ~/.ssh:/home/application/.ssh
- ~/.composer:/home/application/.composer
labels:
- "traefik.backend=my-app"
- "traefik.frontend.rule=Host:my-app.docker"
- "traefik.docker.network=proxy"
networks:
- internal
- proxy
environment:
PHP_DEBUGGER: xdebug
#XDEBUG_REMOTE_HOST: <your host IP address>
XDEBUG_REMOTE_PORT: 9000
XDEBUG_REMOTE_AUTOSTART: 1
XDEBUG_REMOTE_CONNECT_BACK: 1
XDEBUG_PROFILER_ENABLE: 1
XDEBUG_PROFILER_ENABLE_TRIGGER: 1000
traefik:
image: traefik
container_name: citizen-game-traefik
command: --web --docker --docker.domain=docker --logLevel=DEBUG
ports:
- "80:80"
- "443:443"
- "8080:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
restart: always
networks:
- internal
- proxy
rabbitmq:
networks:
- internal
- proxy
networks:
proxy:
external:
name: traefik
internal:
EDIT 2:
#Mostafa
I ran the following:
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' my-app-app
Result is:
172.18.0.7172.19.0.5
Trying these directly from the browser fails "This site can't be reached". I suppose it was to be expected.
I ran the following from inside the container:
bash-4.4# supervisorctl status apache:apached
apache:apached RUNNING pid 13575, uptime 0:00:00
As suggested, I used ss -plant | grep 80. This does not work from within the container. Here is the result when called outside of it:
[/var/www/html/my-app]$ ss -plant | grep 80
LISTEN 0 80 127.0.0.1:3306 0.0.0.0:*
ESTAB 0 0 192.168.1.88:39360 198.252.206.25:443 users:(("chromium-browse",pid=4203,fd=80))
SYN-SENT 0 1 192.168.1.88:50680 192.241.181.178:443 users:(("chromium-browse",pid=4203,fd=41))
LISTEN 0 128 *:80 *:*
LISTEN 0 128 *:8080 *:*
I'm not sure it tells much though. I tried to install ss from inside the container with apk but:
bash-4.4# apk add ss
ERROR: unsatisfiable constraints:
ss (missing):
required by: world[ss]
EDIT 3:
Here is the result of calling netstat:
bash-4.4# netstat -plant
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 229/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.11:32843 0.0.0.0:* LISTEN -
tcp 0 0 :::22 :::* LISTEN 229/sshd
tcp 0 0 :::9000 :::* LISTEN 225/php-fpm.conf)
bash-4.4# netstat -plant | grep httpd
(nothing)
I'm not sure how much this helps though, since my other project, that works, yields the same result n bash-4.4# netstat -plant | grep httpd. Without the grep, it includes much more lines, though.
As the output that you have posted described the exposed ports 80,443,9000 for the container from this image webdevops/php-apache-dev:alpine
Then you can access the container using its IP directly from the browser. So first you need to ensure from the following:
Check if 172.18.0.7 is the actual IP of my-app-app container, use the following command to check the IP of your running container
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' my-app-app
Or just docker inspect my-app-app to get all info about the container
Check the logs for my-app-app and you may need to enter the container itself and check if apache is actually running by executing the following supervisorctl command which will tell you about the status of apache service
$ supervisorctl status apache:apached
apache:apached RUNNING pid 72, uptime 0:07:43
If apache is running correctly then you should be able to browse the content using the container IP, in my case it gives me something like this as I don't have an actual application
Regarding your issue with traefik which is Bad Gateway that's because traefik itself cannot reach your backend service which is the my-app-app container in our case. you need to ensure that both traefik and my-app-app are within the same network or at least they can ping each other's IPs
Update:
Instead of ss it turns out the image contains netstat command, in order to check what port is used by apache you can do the following from inside the container:
# netstat -plant | grep httpd
tcp 0 0 :::80 :::* LISTEN 98/httpd
tcp 0 0 :::443 :::* LISTEN 98/httpd

Resources