On a local development environment (Ubuntu) Docker and Minishift are installed.
With Docker a container with mariadb:10.3.11 is started:
docker run -e MYSQL_ROOT_PASSWORD=root -p 3306:3306 --name mariadb 4f2e75647d2a
Running curl from the host system has the following output:
curl 172.17.0.1:3306
curl: (56) Recv failure: Connection reset by peer
5.5.5-10.3.11-MariaDB-1:10.3.11+maria~bionic
QpatFAPM���rv{(RC:7G#H+mysql_native_password!��#08S01Got packets out of order%
The target is to connect to this MariaDB from a pod running in Minishift. I followed this tutorial: https://docs.okd.io/latest/dev_guide/integrating_external_services.html
apiVersion: v1
kind: List
items:
- kind: "Service"
apiVersion: "v1"
metadata:
name: "external-mysql-service"
spec:
ports:
- name: "mysql"
protocol: "TCP"
port: 3306
targetPort: 3306
nodePort: 0
selector: {}
- kind: "Endpoints"
apiVersion: "v1"
metadata:
name: "external-mysql-service"
subsets:
- addresses:
- ip: "172.17.0.1"
ports:
- port: 3306
name: "mysql"
However, the pod cannot connect to the MariaDB with the following error:
2019-02-19 18:00:42 [ERROR] HikariPool:567 - HikariPool-1 - Exception during pool initialization.
java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=external-mysql-service)(port=3306)(type=master) : Connection refused (Connection refused)
If I connect to the pod via the terminal tab in the OpenShift Web Console and try
$ curl -v external-mysql-service:3306
* Rebuilt URL to: external-mysql-service:3306/
* Trying 172.30.23.62...
* TCP_NODELAY set
* connect to 172.30.23.62 port 3306 failed: Connection refused
* Failed to connect to external-mysql-service port 3306: Connection refused
* Closing connection 0
curl: (7) Failed to connect to external-mysql-service port 3306: Connection refused
How can I establish a connection from the Minishift pod to the Docker container?
The overall goal is to use arbitrary external services.
The IP of the Endpoint has to be the IP of the host system from the perspective of the vm-driver used with Minishift.
In my case the vm-driver is VirtualBox and the IP to access the host system is 10.0.2.2:
https://github.com/kubernetes/minikube/issues/352#issuecomment-237615642
Changing the IP of the endpoint worked for me:
- kind: "Endpoints"
apiVersion: "v1"
metadata:
name: "external-mysql-service"
subsets:
- addresses:
- ip: "10.0.2.2"
ports:
- port: 3306
name: "mysql"
Now the pod inside MiniShift can access the MariaDB running with docker on the host.
Related
What do I need to do in order to get my local browser to and request a resource to a web service running inside Minikube instance running locally on my machine?
I am getting a Connection refused when trying to kubectl port-forward.
My workflow is:
Creating Dockerfile with web service on
Start minikube in docker
Build docker image
Import image locally into Minikube
Created a deployment with one container and a NodePort service
Applied deployment/service
Ran kubectl port-forward (to hopefully forward requests to my container)
Open browser to 127.0.0.1:31000
Port Configuration Summary
Dockerfile:
Expose: 80
uvicorn: 80
Deployment
NodePort Service:
Port: 80
Target Port: 80
Node Port: 31000
Kubectl Command: 8500:31000
Browser: 127.0.0.1:8500
Setup and run through
dev.dockerfile (Step 1)
FROM python:3.11-buster # Some Debian Python image... I built my own
COPY ../sources/api/ /app/
RUN pip install --no-cache-dir --upgrade -r /app/requirements.txt
ENV PYTHONPATH=/app/
EXPOSE 80
CMD ["uvicorn", "app.main:app", "--proxy-headers", "--host", "0.0.0.0", "--port", "80"]
Build Sequence (Steps 2 to 4)
# 2 - start minikube
minikube start --bootstrapper=kubeadm --vm-driver=docker
minikube docker-env
## 3 - build image
docker build -f ../../service1/deploy/dev.dockerfile ../../service1 -t acme-app.service1:latest
## 4 - load image into minikube
minikube image load acme-app.service1:latest
Deployment (Step 5 and 6)
deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: acme-service-1-deployment
namespace: acme-app-dev
labels:
app: service-1
spec:
replicas: 1
selector:
matchLabels:
app: service-1
template:
metadata:
labels:
app: service-1
spec:
containers:
- name: service1-container
image: docker.io/library/acme-app.service1:latest
imagePullPolicy: Never
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: service-1-service
namespace: acme-app-dev
spec:
type: NodePort
selector:
app: service-1
ports:
- protocol: TCP
port: 80
targetPort: 80
nodePort: 31000
Deploy
kubectl apply -f deployment.yaml
kubectl port forward (Step 7)
Find Pod
kubectl get pods -n acme-app-dev
NAME READY STATUS RESTARTS AGE
acme-service-1-deployment-76748d7ff6-llcsr 1/1 Running 0 11s
Port Forward to pod
port-forward acme-service-1-deployment-76748d7ff6-llcsr 8500:31000 -n acme-app-dev
Forwarding from 127.0.0.1:8500 -> 31000
Forwarding from [::1]:8500 -> 31000
Test in Browser (Step 8)
Open favorite browser and navigate to 127.0.0.1:31000.
The console running the port forward now outputs:
E0123 14:54:16.208010 25932 portforward.go:406] an error occurred forwarding 8500 -> 31000: error forwarding port 31000 to pod d4c0fa6cb16ce02335a05cad904fbf2ab7818e2073d7c7ded8ad05f193aa37e7, uid : exit status 1: 2023/01/23 14:54:16 socat[39370] E connect(5, AF=2 127.0.0.1:31000, 16): Connection refused
E0123 14:54:16.213268 25932 portforward.go:234] lost connection to pod
What have I looked at?
I've tried looking through the docs on kubernetes website as well as issues on here (yes there are similar). This is pretty similar - although no marked answer and still an issue by the looks of it. I couldn't see a solution for my issue here.
NodePort exposed Port connection refused
I am running Minikube on Windows and I'm just setting out on a kubernetes journey.
The image itself works in docker from a docker compose. I can see the pod is up and running in minikube from the logs (minikube dashboard).
You got your wires crossed:
The pod is listening on port 80
The NodePort service is listening on port 31000 on the node, but its underlying ClusterIP service is listening on port 80 as well.
You are trying to port-forward to port 31000 on the Pod. This will not work.
Call one of the following instead:
kubectl port-forward -n acme-app-dev deploy/acme-service-1-deployment 8500:80
or kubectl port-forward -n acme-app-dev service/service-1-service 8500:80
or use minikube service -n acme-app-dev service-1-service and use the provided URL.
I have a celery instance running inside a pod in local kubernetes cluster whereas the redis server/broker it connects to is started on my localhost:6379 without kubernetes . How can i get my k8 pod to talk to locally deployed redis?
You can create a Headless Service and an Endpoint with statically defined IP address of the node where the redis server is running.
I've created an example to illustrate you how it works.
First, I created a Headless Service and an Endpoint.
NOTE: Endpoint has the IP address of the node where redis server is running:
# example.yml
---
apiVersion: v1
kind: Service
metadata:
name: redis
namespace: default
spec:
clusterIP: None
ports:
- name: redis
port: 6379
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: redis
namespace: default
subsets:
- addresses:
- ip: 10.156.0.58 # your node's IP address
ports:
- port: 6379
name: redis
protocol: TCP
After creating above resources, we are able to resolve the redis service name to the IP address:
# kubectl get svc,ep redis
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/redis ClusterIP None <none> 6379/TCP 28m
NAME ENDPOINTS AGE
endpoints/redis 10.156.0.58:6379 28m
# kubectl run dnsutils --image=gcr.io/kubernetes-e2e-test-images/dnsutils:1.3 -it --rm
/ # nslookup redis
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: redis.default.svc.cluster.local
Address: 10.156.0.58
Additionally, if your redis server is only listening on localhost, you need to modify the iptables rules. To configure port forwarding from port 6379 (default redis port) to localhost you can use:
NOTE: Instead of 10.156.0.58 use the IP address of the node where your redis server is running.
# iptables -t nat -A PREROUTING -p tcp -d 10.156.0.58 --dport 6379 -j DNAT --to-destination 127.0.0.1:6379
As you can see, it is easier if redis is listening not only on the localhost, as we don't have to modify the iptables rules then.
Finally, let's see if we can connect from Pod to the redis server on the host machine:
# kubectl exec -it redis-client -- bash
root#redis-client:/# redis-cli -h redis
redis:6379> SET key1 "value1"
OK
I am using the WSL2 based engine for Docker and I have enabled Kubernetes v1.19.3
I have several Kubernetes services and pods running and I want to connect to a website hosted on the WSL2 VM. How can I determine the IP address for that VM that I can use to connect from a pod?
I ran hostname -I on the VM and got an IP address for the machine.
I created a service and an endpoint:
apiVersion: v1
kind: Service
metadata:
name: test-viewer
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8280
apiVersion: v1
kind: Endpoints
metadata:
name: test-viewer
subsets:
- addresses:
- ip: 172.17.159.34
ports:
- port: 8280
I tried to use curl from one of the pods and got the following error:
curl http://test-viewer.default.svc.cluster.local:8080/index.html --output somefile
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0
curl: (7) Failed connect to test-viewer.default.svc.cluster.local:8080; No route to host
I run the following command on the WSL machine without issue:
curl http://172.17.159.34:8280/index.html --output somefile
Your problem is on the service the specified port.
So either change the service port to 80 use the port 8080 on your curl
apiVersion: v1
kind: Service
metadata:
name: test-viewer
spec:
ports:
- protocol: TCP
port: 80
targetPort: 8280
i'm new to kubernetes , i'm trying to learn it using minikube and i'm facing a problem with accessing apps outside the cluster. i created a deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 8080
To access it i need to expose it decoratively or imperatively. In the imperative way it works :
kubectl expose deployment nginx-deployment --port 80 --type NodePort
When i create a service declaratively i always end up with a connection refused error :
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
spec:
type : NodePort
ports:
- port : 8080
nodePort : 30018
protocol : TCP
selector:
app: nginx
curl -k http://NodeIP:NodePort returns :
curl: (7) Failed to connect to Node IP port NodePORT: Connection
refused
As #Ansil suggested, your nginx should be configured to listen on port 8080 if you want to refer to this port in your Service definition. By default it listens on port 80.
You cannot make it listen on different port like 8080 simply by specifying different containerPort in your Deployment definition as in your example:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 8080
You can easily verify it on your own by attaching to such Pod:
kubectl exec -ti <nginx-pod-name> -- /bin/bash
Once you're there, run:
ss -ntlp
And you should see on which port your nginx actually listens on.
Additionally you may:
cat /etc/nginx/conf.d/default.conf
It will also tell you on which port your nginx is configured to listen. That's all. It's really simple. You changed containerPort to 8080 but inside your container nothing actually listens on such port.
You can still expose it as a Service (no matter declaratively or imperatively) but it won't change anything as eventually it points to the wrong port on your container, on which nothing listens and you'll see message similar to this one:
curl: (7) Failed to connect to 10.1.2.3 port 30080: Connection refused
Once you create a service in minikube you can expose the service to the outside of the minikube VM (host machine) using the command
minikube service SERVICE_NAME
Refer: https://minikube.sigs.k8s.io/docs/reference/commands/service/
I can't connect to my app running with nginx ingress (Docker Desktop win 10).
The nginx-ingress controller pod is running, the app is healthy, and I have created an ingress. However, when I try to connect to my app on localhost, I get "connection refused".
I see this error in the log:
[14:13:13.028][VpnKit ][Info ] vpnkit.exe: Connected Ethernet interface f6:16:36:bc:f9:c6
[14:13:13.028][VpnKit ][Info ] vpnkit.exe: UDP interface connected on 10.96.181.150
[14:13:22.320][GoBackendProcess ][Info ] Adding vpnkit-k8s-controller tcp forward from 0.0.0.0:80 to 10.96.47.183:80
[14:13:22.323][ApiProxy ][Error ] time="2019-12-09T14:13:22-05:00" msg="Port 443 for service ingress-nginx is already opened by another service"
I think port 443 is used by another app, possibly zscaler security or skype.
Excerpt from netstat -a -b:
[svchost.exe]
TCP 0.0.0.0:443 0.0.0.0:0 LISTENING 16012
[com.docker.backend.exe]
TCP 0.0.0.0:443 0.0.0.0:0 LISTENING 8220
I don't know how to make the ingress work. Please help!
My ingress:
$ kubectl describe ing kbvalues-deployment-dev-ingress
Name: kbvalues-deployment-dev-ingress
Namespace: default
Address: localhost
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
localhost
/ kbvalues-deployment-dev-frontend:28000 (10.1.0.174:8080)
Annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/cors-allow-headers: X-Forwarded-For, X-app123-XPTO
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 42m nginx-ingress-controller Ingress default/kbvalues-deployment-dev-ingress
Normal UPDATE 6s (x5 over 42m) nginx-ingress-controller Ingress default/kbvalues-deployment-dev-ingress
My service:
$ kubectl describe svc kbvalues-deployment-dev-frontend
Name: kbvalues-deployment-dev-frontend
Namespace: default
Labels: chart=tomcat-sidecar-war-1.0.4
environment=dev
name=kbvalues-frontend-dev
release=kbvalues-test
tier=frontend
Annotations: <none>
Selector: app=kbvalues-dev
Type: ClusterIP
IP: 10.98.89.94
Port: <unset> 28000/TCP
TargetPort: 8080/TCP
Endpoints: 10.1.0.174:8080
Session Affinity: None
Events: <none>
I am trying to access the app at: http://localhost:28000/health. I verified that the /health URL is accessible locally within the web server container.
I appreciate any help you can offer.
Edit:
I tried altering the ingress-nginx service to remove HTTPS, as suggested here:
https://stackoverflow.com/a/56303330/166850
This got rid of the 443 error in the logs, but didn't fix my setup (still getting connection refused).
Edit 2: Here is the Ingress YAML definition (kubectl get -o yaml):
$ kubectl get ing -o yaml
apiVersion: v1
items:
- apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
creationTimestamp: "2019-12-09T18:47:33Z"
generation: 5
name: kbvalues-deployment-dev-ingress
namespace: default
resourceVersion: "20414"
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/kbvalues-deployment-dev-ingress
uid: 5c34bf7f-1ab4-11ea-80e4-00155d169409
spec:
rules:
- host: localhost
http:
paths:
- backend:
serviceName: kbvalues-deployment-dev-frontend
servicePort: 28000
path: /
status:
loadBalancer:
ingress:
- hostname: localhost
kind: List
metadata:
resourceVersion: ""
selfLink: ""
Edit 3: Output of kubectl get svc -A (ingress line only):
ingress-nginx ingress-nginx LoadBalancer 10.96.47.183 localhost 80:30470/TCP 21h
Edit 4: I tried to get the VM's IP address from windows HyperV, but it seems like the VM doesn't have an IP?
PS C:\> (Get-VMNetworkAdapter -VMName DockerDesktopVM)
Name IsManagementOs VMName SwitchName MacAddress Status IPAddresses
---- -------------- ------ ---------- ---------- ------ -----------
Network Adapter False DockerDesktopVM DockerNAT 00155D169409 {Ok} {}
Edit 5:
Output of netstat -a -n -o -b for port 80:
TCP 0.0.0.0:80 0.0.0.0:0 LISTENING 4
Can not obtain ownership information
I have managed to create Ingress resource in Kubernetes on Docker in Windows.
Steps to reproduce:
Enable Hyper-V
Install Docker for Windows and enable Kubernetes
Connect kubectl
Enable Ingress
Create deployment
Create service
Create ingress resource
Add host into local hosts file
Test
Enable Hyper-V
From Powershell with administrator access run below command:
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All
System could ask you to reboot your machine.
Install Docker for Windows and enable Kubernetes
Install Docker application with all the default options and enable Kubernetes
Connect kubectl
Install kubectl .
Enable Ingress
Run this commands:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml
Edit: Make sure no other service is using port 80
Restart your machine. From a cmd prompt running as admin, do:
net stop http
Stop the listed services using services.msc
Use: netstat -a -n -o -b and check for other processes listening on port 80.
Create deployment
Below is simple deployment with pods that will reply to requests:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello
spec:
selector:
matchLabels:
app: hello
version: 2.0.0
replicas: 3
template:
metadata:
labels:
app: hello
version: 2.0.0
spec:
containers:
- name: hello
image: "gcr.io/google-samples/hello-app:2.0"
env:
- name: "PORT"
value: "50001"
Apply it by running command:
$ kubectl apply -f file_name.yaml
Create service
For pods to be able for you to communicate with them you need to create a service.
Example below:
apiVersion: v1
kind: Service
metadata:
name: hello-service
spec:
type: NodePort
selector:
app: hello
version: 2.0.0
ports:
- name: http
protocol: TCP
port: 80
targetPort: 50001
Apply this service definition by running command:
$ kubectl apply -f file_name.yaml
Create Ingress resource
Below is simple Ingress resource using service created above:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: hello-ingress
spec:
rules:
- host: hello-test.internal
http:
paths:
- path: /
backend:
serviceName: hello-service
servicePort: http
Take a look at:
spec:
rules:
- host: hello-test.internal
hello-test.internal will be used as the hostname to connect to your pods.
Apply your Ingress resource by invoking command:
$ kubectl apply -f file_name.yaml
Add host into local hosts file
I found this Github link that will allow you to connect to your Ingress resource by hostname.
To achieve that add a line 127.0.0.1 hello-test.internal to your C:\Windows\System32\drivers\etc\hosts file and save it.
You will need Administrator privileges to do that.
Edit: The newest version of Docker Desktop for Windows already adds a hosts file entry:
127.0.0.1 kubernetes.docker.internal
Test
Display the information about Ingress resources by invoking command:
kubectl get ingress
It should show:
NAME HOSTS ADDRESS PORTS AGE
hello-ingress hello-test.internal localhost 80 6m2s
Now you can access your Ingress resource by opening your web browser and typing
http://kubernetes.docker.internal/
The browser should output:
Hello, world!
Version: 2.0.0
Hostname: hello-84d554cbdf-2lr76
Hostname: hello-84d554cbdf-2lr76 is the name of the pod that replied.
If this solution is not working please check connections with the command:
netstat -a -n -o
(with Administrator privileges) if something is not using port 80.
On Windows the Kubernetes cluster is running in a VM. Try to access ingress on that VM-s IP address instead of localhost.
i was facing similar issue while deploying ingress-nginx controller using the manual steps mentioned for bareMetal node at ingress-nginx-deploy however was facing an issue , however referred to the link Github link mentioned by #RMorrisey which leads to other threads where they have mentioned to install ingress-nginx using the steps mentioned for mac and it worked without making cny changes to host file , etc
The problem is that your service has a type of ClusterIP, which isn't accessible externally. You need it to be of type NodePort, which is what is done in Dawid Kruk's instructions.