I'm using Ansible to deploy a Ruby on Rails application using Puma as a web server. As part of the deployment, the Puma configuration binds to the IP address of the server on port 8080:
bind "tcp://{{ ip_address }}:8080"
This is then used in the nginx vhost config to access the app:
upstream {{ app_name }} {
server {{ ip_address }}:8080;
}
All of this is working fine. However, I now want to deploy multiple copies of the app (staging, production) onto the same server and obviously having several bindings on 8080 is causing issues so I need to use different ports.
The most simple solution would be to include the port in a group var and then just drop it in when the app is deployed. However, this would require background knowledge of the apps already running on the server and it kind of feels like the deployment should be able to "discover" the port to use.
Instead, I was considering doing some kind of iteration through ports, starting at 8080, and then checking each until one is not being used. netstat -anp | grep 8080 gives a return code 0 if the port is being used so perhaps I could use that command to test (though I'm not sure of how to do the looping bit).
Has anyone come up against this problem before? Is there a more graceful solution that I'm overlooking?
I'd define list of allowed ports and compare it to available ports.
Something like this:
- hosts: myserver
vars:
allowed_ports:
- 80
- 8200
tasks:
- name: Gather occupied tcp v4 ports
shell: netstat -nlt4 | grep -oP '(?<=0.0.0.0:)(\d+)'
register: used_ports
- name: Set bind_port as first available port
set_fact:
bind_port: "{{ allowed_ports | difference(used_ports.stdout_lines | map('int') | list) | first | default(0) }}"
failed_when: bind_port | int == 0
- name: Show bind port
debug: var=bind_port
You may want to tune 0.0.0.0 in the regexp if you need to check ports on specific interface.
Related
I am using the opentelemetry-ruby otlp exporter for auto instrumentation:
https://github.com/open-telemetry/opentelemetry-ruby/tree/main/exporter/otlp
The otel collector was installed as a daemonset:
https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-collector
I am trying to get the OpenTelemetry collector to collect traces from the Rails application. Both are running in the same cluster, but in different namespaces.
We have enabled auto-instrumentation in the app, but the rails logs are currently showing these errors:
E, [2022-04-05T22:37:47.838197 #6] ERROR -- : OpenTelemetry error: Unable to export 499 spans
I set the following env variables within the app:
OTEL_LOG_LEVEL=debug
OTEL_EXPORTER_OTLP_ENDPOINT=http://0.0.0.0:4318
I can't confirm that the application can communicate with the collector pods on this port.
Curling this address from the rails/ruby app returns "Connection Refused". However I am able to curl http://<OTEL_POD_IP>:4318 which returns 404 page not found.
From inside a pod:
# curl http://localhost:4318/
curl: (7) Failed to connect to localhost port 4318: Connection refused
# curl http://10.1.0.66:4318/
404 page not found
This helm chart created a daemonset but there is no service running. Is there some setting I need to enable to get this to work?
I confirmed that otel-collector is running on every node in the cluster and the daemonset has HostPort set to 4318.
The problem is with this setting:
OTEL_EXPORTER_OTLP_ENDPOINT=http://0.0.0.0:4318
Imagine your pod as a stripped out host itself. Localhost or 0.0.0.0 of your pod, and you don't have a collector deployed in your pod.
You need to use the address from your collector. I've checked the examples available at the shared repo and for agent-and-standalone and standalone-only you also have a k8s resource of type Service.
With that you can use the full service name (with namespace) to configure your environment variable.
Also, the Environment variable now is called OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, so you will need something like this:
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=<service-name>.<namespace>.svc.cluster.local:<service-port>
The correct solution is to use the Kubernetes Downward API to fetch the node IP address, which will allow you to export the traces directly to the daemonset pod within the same node:
containers:
- name: my-app
image: my-image
env:
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://$(HOST_IP):4318
Note that using the deployment's service as the endpoint (<service-name>.<namespace>.svc.cluster.local) is incorrect, as it effectively bypasses the daemonset and sends the traces directly to the deployment, which makes the daemonset useless.
I am moving a Docker Image from Docker to a K8s Deployment. I have auto-scale rules on so it starts 5 but can go to 12. The Docker image on K8s starts perfectly with a k8s service in front to cluster the Deployment.
Now each container has its own JVM which has a Prometheus app retrieving its stats. In Docker, this is no problem because the port that serves Prometheus info is dynamically created with a starting port of 8000, so the docker-compose.yml grows the port by 1 based on how many images are started.
The problem is that I can't find how to do this in a K8s [deployment].yml file. Because Deployment pods are dynamic, I would have thought there would be some way to set a starting HOST port to be incremented based on how many containers are started.
Maybe I am looking at this the wrong way so any clarification would be helpful meanwhile will keep searching the Google for any info on such a thing.
Well after reading and reading and reading so much I came to the conclusion that K8s is not responsible to open ports for a Docker Image or provide ingress to your app on some weird port, it's not its responsibility. K8s Deployment just deploys the Pods you requested. You can set the Ports option on a DEPLOYMENT -> SPEC -> CONTAINERS -> PORTS which just like Docker is only informational. But this allows you to JSONPath query for all PODS(containers) with a Prometheus port available. This will allow you to rebuild the "targets" value in Prometheus.yaml file. Now having those targets makes them available to Grafana to create a dashboard.
That's it, pretty easy. I was complicating something because I did not understand it. I am including a script I QUICKLY wrote to get something going USE AT YOUR OWN RISK.
By the way, I use Pod and Container interchangeably.
#!/usr/bin/env bash
#set -x
_MyappPrometheusPort=8055
_finalIpsPortArray=()
_prometheusyamlFile=prometheus.yml
cd /docker/images/prometheus
#######################################################################################################################################################
#One container on the K8s System is weave and it holds the subnet we need to validate against.
#weave-net-lwzrk 2/2 Running 8 (7d3h ago) 9d 192.168.2.16 accl-ffm-srv-006 <none> <none>
_weavenet=$(kubectl get pod -n kube-system -o wide | grep weave | cut -d ' ' -f1 )
echo "_weavenet: $_weavenet"
#The default subnet is the one that lets us know the conntainer is part of kubernetes network.
# Range: 10.32.0.0/12
# DefaultSubnet: 10.32.0.0/12
_subnet=$( kubectl exec -n kube-system $_weavenet -c weave -- /home/weave/weave --local status | sed -En "s/^(.*)(DefaultSubnet:\s)(.*)?/\3/p" )
echo "_subnet: $_subnet"
_cidr2=$( echo "$_subnet" | cut -d '/' -f2 )
echo "_cidr2: /$_cidr2"
#######################################################################################################################################################
#This is an array of the currently monitored containers that prometheus was sstarted with.
#We will remove any containers form the array that fit the K8s Weavenet subnet with the myapp prometheus port.
_targetLineFound_array=($( egrep '^\s{1,20}-\s{0,5}targets\s{0,5}:\s{0,5}\[.*\]' $_prometheusyamlFile | sed -En "s/(.*-\stargets:\s\[)(.*)(\]).*/\2/p" | tr "," "\n"))
for index in "${_targetLineFound_array[#]}"
do
_ip="${index//\'/$''}"
_ipTocheck=$( echo $_ip | cut -d ':' -f1 )
_portTocheck=$( echo $_ip | cut -d ':' -f2 )
#We need to check if the IP is within the subnet mask attained from K8s.
#The port must also be the prometheus port in case some other port is used also for Prometheus.
#This means the IP should be removed since we will put the list of IPs from
#K8s currently in production by Deployment/AutoScale rules.
#Network: 10.32.0.0/12
_isIpWithinSubnet=$( ipcalc $_ipTocheck/$_cidr2 | sed -En "s/^(.*)(Network:\s+)([0-9]{1}[0-9]?[0-9]?\.[0-9]{1}[0-9]?[0-9]?\.[0-9]{1}[0-9]?[0-9]?\.[0-9]{1}[0-9]?[0-9]?)(\/[0-9]{1}[0-9]{1}.*)?/\3/p" )
if [[ "$_isIpWithinSubnet/$_cidr2" == "$_subnet" && "$_portTocheck" == "$_MyappPrometheusPort" ]]; then
echo "IP managed by K8s will be deleted: _isIpWithinSubnet: ($_ip) $_isIpWithinSubnet"
else
_finalIpsPortArray+=("$_ip")
fi
done
#######################################################################################################################################################
#This is an array of the current running myapp App containers with a prometheus port that is available.
#From this list we will add them to the prometheus file to be available for Grafana monitoring.
readarray -t _currentK8sIpsArr < <( kubectl get pods --all-namespaces --chunk-size=0 -o json | jq '.items[] | select(.spec.containers[].ports != null) | select(.spec.containers[].ports[].containerPort == '$_MyappPrometheusPort' ) | .status.podIP' )
for index in "${!_currentK8sIpsArr[#]}"
do
_addIPToMonitoring=${_currentK8sIpsArr[index]//\"/$''}
echo "IP Managed by K8s as myapp app with prometheus currently running will be added to monitoring: $_addIPToMonitoring"
_finalIpsPortArray+=("$_addIPToMonitoring:$_MyappPrometheusPort")
done
######################################################################################################################################################
#we need to recreate this string and sed it into the file
#- targets: ['192.168.2.13:3201', '192.168.2.13:3202', '10.32.0.7:8055', '10.32.0.8:8055']
_finalPrometheusTargetString="- targets: ["
i=0
# Iterate the loop to read and print each array element
for index in "${!_finalIpsPortArray[#]}"
do
((i=i+1))
_finalPrometheusTargetString="$_finalPrometheusTargetString '${_finalIpsPortArray[index]}'"
if [[ $i != ${#_finalIpsPortArray[#]} ]]; then
_finalPrometheusTargetString="$_finalPrometheusTargetString,"
fi
done
_finalPrometheusTargetString="$_finalPrometheusTargetString]"
echo "$_finalPrometheusTargetString"
sed -i -E "s/(.*)-\stargets:\s\[.*\]/\1$_finalPrometheusTargetString/" ./$_prometheusyamlFile
docker-compose down
sleep 4
docker-compose up -d
echo "All changes were made. Exiting"
exit 0
Ideally, you should be using the Average of JVM across all the replicas. There is no meaning to create a different deployment with a different port if you are running the single same Docker image across all the replicas.
i think keeping a single deployment with resource requirements set to deployment would be the best practice.
You can get the JVM average of all the running replicas
sum(jvm_memory_max_bytes{area="heap", app="app-name",job="my-job"}) / sum(kube_pod_status_phase{phase="Running"})
as you are running the same Docker image across all replicas and K8s service by default will be managing the Load Balancing, average utilization would be an option to monitor.
Still, if you want to filter and get different values you can create different deployments (Not at all good way) or use the stateful sets.
You can also filter the data by hostname (POD name) in Prometheus, so will get the each replica usage.
I have followed the instructions, I think, and have come up with the following configuration:
version: '3.9'
services:
flask:
image: ops:imgA
ports:
- 5000:5000
volumes:
- /opt/models:/opt/models
entrypoint: demo flask
streamlit:
image: ops:imgB
ports:
- 8501:8501
entrypoint: streamlit run --server.port 8501 demo -- stream --flask-hostname flask
The --flask-hostname flask sets the host name used in an http connect, i.e.: http://flask:5000. I can set it to anything.
The basic problem here is that I can spin up one of these images, install tmux, and run everything within a single image.
But, when I split it across multiple images and use docker-compose up (which seems better than tmux), the containers can't seem to connect to each other.
I have rattled around the documentation on docker's website, but I've moved on to the troubleshooting stage. This seems to be something that should "just work" (since there are few questions along these lines). I have total control of the box I am using, and can open or close whatever ports needed.
Mainly, I am trying to figure out how to allow, with 100% default settings nothing complicated, these two services (flask and streamlit) to speak to each other.
There must be 1 or 2 settings that I need to change, and that is it.
Any ideas?
Update
I can access all of the services externally, so I am going to open up external connections between the services (using the external IP) as a "just work" quick fix, but obviously getting the composition to work internally would be the best option.
I have also confirmed that the docker-compose and docker versions are up to date.
Update-2: changed from flask#127.0.0.1 to flask#0.0.0.0
Flask output:
flask_1 | * Serving Flask app "flask" (lazy loading)
flask_1 | * Environment: production
flask_1 | WARNING: This is a development server. Do not use it in a production deployment.
flask_1 | Use a production WSGI server instead.
flask_1 | * Debug mode: on
flask_1 | INFO:werkzeug: * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
flask_1 | 2020-12-19 02:22:16.449 INFO werkzeug: * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
flask_1 | INFO:werkzeug: * Restarting with inotify reloader
flask_1 | 2020-12-19 02:22:16.465 INFO werkzeug: * Restarting with inotify reloader
flask_1 | WARNING:werkzeug: * Debugger is active!
flask_1 | 2020-12-19 02:22:22.003 WARNING werkzeug: * Debugger is active!
Streamlit:
streamlit_1 |
streamlit_1 | You can now view your Streamlit app in your browser.
streamlit_1 |
streamlit_1 | Network URL: http://172.18.0.3:8501
streamlit_1 | External URL: http://71.199.156.142:8501
streamlit_1 |
streamlit_1 | 2020-12-19 02:22:11.389 Generating new fontManager, this may take some time...
And the streamlit error message:
ConnectionError:
HTTPConnectionPool(host='flask', port=5000):
Max retries exceeded with url: /foo/bar
(Caused by NewConnectionError(
'<urllib3.connection.HTTPConnection object at 0x7fb860501d90>:
Failed to establish a new connection:
[Errno 111] Connection refused'
)
)
Update-3: Hitting refresh fixed it.
The server process must be listening on the special "all interfaces" address 0.0.0.0. Many development-type servers by default listen on "localhost only" 127.0.0.1, but in Docker each container has its own private notion of localhost. If you use tmux or docker exec to run multiple processes inside a container, they have the same localhost and can connect to each other, but if the client and server are running in different containers, the request doesn't arrive on the server's localhost interface, and if the server is listening on "localhost only" it won't receive it.
Your setup is otherwise correct, with only the docker-compose.yml you include in the question. Some other common problems:
You must connect to the port the server process is listening on inside the container. If you remap it externally with ports:, that's ignored, and you'd connect to the second ports: number. Correspondingly, ports: aren't required. (expose: also isn't required and doesn't do anything at all.)
The client may need to wait for the server to start up. If the client depends_on: [flask] the host name will usually resolve (unless the server dies immediately) but if it takes a while to start up you will still get "connection refused" errors. See Docker Compose wait for container X before starting Y.
Neither container may use network_mode: host. This disables Docker's networking features entirely.
If you manually declare networks:, both containers need to be on the same network. You do not need to explicitly create a network for inter-container communication to work: Compose provides a default network for you, which is used if nothing else is declared.
Use the Compose service names as host names. You don't need to explicitly specify container_name: or links:.
I'm using Ansible to deploy my containers to production. During development, I'd like to have Ansible deploy all containers on my localhost instead of the production servers.
Production servers run on Ubuntu, localhost is OS X with boot2docker.
I have 2 inventory files, production and localhost. My dir structure looks like this:
.
|--inventories
| |localhost
| |production
|
|--group_vars
| |all
| |localhost
| |production
|
|--roles
| |--web
| |--tasks
|main.yml
|
|web.yml
web.yml just defines the host group and assigns the web role.
/roles/web/tasks/main.yml looks like this:
- name: Run web container
docker:
name: web
image: some/image
state: reloaded
...
tls_hostname: boot2docker
volumes:
- "/data/db:/data/db"
env:
...
tags:
- ...
I need to set tls_hostname conditionally, only if the localhost inventory was used; likewise, I want to set the volume only if the production inventory file was used.
Very new to Ansible - it seems like I'm not approaching this to right way there's an easier way to do this? I want to avoid creating completely separate tasks to deploy locally; I just need a way to define volume and tls_hostname conditionally (and leave it at default setting otherwise)
As of Ansible 1.8, you can omit variables and module parameters using the default filter with the special omit value. For example,
- name: Run web container
docker:
name: web
image: some/image
state: reloaded
...
tls_hostname: "{{ hostname | default('some default value') }}"
volumes: "{{ volumes | default(omit) }}"
env:
...
tags:
- ...
I see you do have group_vars files for localhost and production. Are you familiar with that concept? Because I think this is what you're looking for.
The variables defined in the group_vars section will be applied if the host belongs to the respective group.
I need to set tls_hostname conditionally, only if the localhost inventory was used; likewise, I want to set the volume only if the production inventory file was used.
So that sounds like you want to define tls_hostname in ./group_vars/localhost and volume in ./group_vars/production.
(and leave it at default setting otherwise)
Default values can be stored in several places. If you have a role you can store in <role>/defaults/main.yml. group_vars/all also is possible. As well you can set a default value in your yml's.
- name: Run web container
docker:
name: web
image: some/image
state: reloaded
...
tls_hostname: "{{ hostname | default('some default value') }}"
volumes: "{{ volumes | default(['/data/db:/data/db']) }}"
env:
...
tags:
- ...
If hostname or volumes are not defined Ansible will fall back to the defined default value.
Hello I am having an issue with configuring both TCP and UDP to the same port via SaltStack and the dockerio module. In this case I am trying to configure DNS for both TCP and UDP port 53.
The environment is the same for both master and minion, the details are:
Ubuntu 15.04
lxc-docker 1.6.2 package (from deb https://get.docker.com/ubuntu docker main)
salt-common 2015.5.0+ds-1utopic1 package
salt-minion 2015.5.0+ds-1utopic1 package (salt-minion 2015.5.0 (Lithium))
salt-master 2015.5.0+ds-1utopic1 package (salt-master 2015.5.0 (Lithium))
docker-py 1.2.2
From the Dockerfile for the container:
EXPOSE 53
EXPOSE 53/udp
The relevant config for SaltStack from the state.sls file for this container (inside docker.running):
{% set hostport1 = '53' %}
{% set hostport1_proto = 'udp' %}
{% set hostport2 = '53' %}
{% set hostport2_proto = 'tcp' %}
...
- ports:
"{{ hostport1 }}/{{ hostport1_proto }}":
HostIp: ""
HostPort: "{{ hostport1 }}"
"{{ hostport2 }}/{{ hostport2_proto }}":
HostIp: ""
HostPort: "{{ hostport2 }}"
...
The container starts up and runs successfully. The issue is only the TCP port is ever mapped, the UDP port remains unmapped. It does not matter which order I list the TCP and UDP port only the TCP port is ever mapped through docker.
From docker ps -a:
0.0.0.0:53->53/tcp, 53/udp
I have looked through all the information I can find and I have seen older closed issues related to docker-py but nothing recent and seemingly no one else having this issue.
I have confirmed that running the container manually does work and map the ports correctly:
docker run ... -p 53:53 -p 53:53/udp ...
docker ps -a
... 0.0.0.0:53->53/tcp, 0.0.0.0:53->53/udp ...
So this confirms to me the source of this issue or misconfiguration is in either docker-py or SaltStack. Any information or configuration tips would be greatly appreciated. Thanks.
The source of this issue is the configuration of the HostIp: field. Using a blank value of "" is not acceptable. Instead setting the HostIp to "0.0.0.0" resolves this issue completely. The documentation in this regard is not clear.
Much thanks and credit goes to mosen on #salt channel on Freenode IRC.