I'm trying to install pihole on a Kubernetes cluster on Docker via helm. I'm following this guide to do so. Everything seems to go smoothly. I get a completion:
NAME: pihole
LAST DEPLOYED: Wed Sep 30 22:22:15 2020
NAMESPACE: pihole
STATUS: deployed
REVISION: 1
TEST SUITE: None
But the pihole never reaches the ready state, it just restarts after a couple minutes. Upon inspecting the pod I see:
lastState:
terminated:
containerID: docker://16e2a318b460d4d5aebd502175fb688fc150993940181827a506c086e2cb326a
exitCode: 0
finishedAt: "2020-09-30T22:01:55Z"
reason: Completed
startedAt: "2020-09-30T21:59:17Z"
How do I prevent this from continually restarting once it's complete?
Here is the output of kubectl logs <POD_NAME>:
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] 01-resolver-resolv: applying...
[fix-attrs.d] 01-resolver-resolv: exited 0.
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 20-start.sh: executing...
::: Starting docker specific checks & setup for docker pihole/pihole
[✓] Update local cache of available packages
[i] Existing PHP installation detected : PHP version 7.0.33-0+deb9u8
[i] Installing configs from /etc/.pihole...
[i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone!
[✓] Copying 01-pihole.conf to /etc/dnsmasq.d/01-pihole.conf
chown: cannot access '': No such file or directory
chmod: cannot access '': No such file or directory
chown: cannot access '/etc/pihole/dhcp.leases': No such file or directory
::: Pre existing WEBPASSWORD found
Using default DNS servers: 8.8.8.8 & 8.8.4.4
DNSMasq binding to default interface: eth0
Added ENV to php:
"PHP_ERROR_LOG" => "/var/log/lighttpd/error.log",
"ServerIP" => "0.0.0.0",
"VIRTUAL_HOST" => "pi.hole",
Using IPv4 and IPv6
::: Preexisting ad list /etc/pihole/adlists.list detected ((exiting setup_blocklists early))
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
https://mirror1.malwaredomains.com/files/justdomains
::: Testing pihole-FTL DNS: FTL started!
::: Testing lighttpd config: Syntax OK
::: All config checks passed, cleared for startup ...
::: Docker start setup complete
[✗] DNS resolution is currently unavailable
You are not alone with this issue.
Resolution is here - chown: cannot access '/etc/pihole/dhcp.leases': No such file or directory
This happens for me as well. I used that same tutorial to set up my
cluster. If you are using a persistent volume as well, use a ssh
connection to get to your drive and run these two commands.
ls -l ----> this will show the owner and user of each file they all should be www-data if not run this cmd
sudo chown -R www-data:www-data pihole from the /mnt/ssd directory described in the tutorial. This will allow you to add more whitelists/blacklists/adlists from the web portal.
Related
In a docker container I want to run k8s.
When I run kubeadm join ... or kubeadm init commands I see sometimes errors like
\"modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could
not open moddep file
'/lib/modules/3.10.0-1062.1.2.el7.x86_64/modules.dep.bin'.
nmodprobe:
FATAL: Module configs not found in directory
/lib/modules/3.10.0-1062.1.2.el7.x86_64",
err: exit status 1
because (I think) my container does not have the expected kernel header files.
I realise that the container reports its kernel based on the host that is running the container; and looking at k8s code I see
// getKernelConfigReader search kernel config file in a predefined list. Once the kernel config
// file is found it will read the configurations into a byte buffer and return. If the kernel
// config file is not found, it will try to load kernel config module and retry again.
func (k *KernelValidator) getKernelConfigReader() (io.Reader, error) {
possibePaths := []string{
"/proc/config.gz",
"/boot/config-" + k.kernelRelease,
"/usr/src/linux-" + k.kernelRelease + "/.config",
"/usr/src/linux/.config",
}
so I am bit confused what is simplest way to run k8s inside a container such that it consistently past this getting the kernel info.
I note that running docker run -it solita/centos-systemd:7 /bin/bash on a macOS host I see :
# uname -r
4.9.184-linuxkit
# ls -l /proc/config.gz
-r--r--r-- 1 root root 23834 Nov 20 16:40 /proc/config.gz
but running exact same on a Ubuntu VM I see :
# uname -r
4.4.0-142-generic
# ls -l /proc/config.gz
ls: cannot access /proc/config.gz
[Weirdly I don't see this FATAL: Module configs not found in directory error every time, but I guess that is a separate question!]
UPDATE 22/November/2019. I see now that k8s DOES run okay in a container. Real problem was weird/misleading logs. I have added an answer to clarify.
I do not believe that is possible given the nature of containers.
You should instead test your app in a docker container then deploy that image to k8s either in the cloud or locally using minikube.
Another solution is to run it under kind which uses docker driver instead of VirtualBox
https://kind.sigs.k8s.io/docs/user/quick-start/
It seems the FATAL error part was a bit misleading.
It was badly formatted by my test environment (all on one line.
When k8s was failing I saw the FATAL and assumed (incorrectly) that was root cause.
When I format the logs nicely I see ...
kubeadm join 172.17.0.2:6443 --token 21e8ab.1e1666a25fd37338 --discovery-token-unsafe-skip-ca-verification --experimental-control-plane --ignore-preflight-errors=all --node-name 172.17.0.3
[preflight] Running pre-flight checks
[WARNING FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 4.4.0-142-generic
DOCKER_VERSION: 18.09.3
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.3. Latest validated version: 18.06
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-142-generic/modules.dep.bin'\nmodprobe: FATAL: Module configs not found in directory /lib/modules/4.4.0-142-generic\n", err: exit status 1
[discovery] Trying to connect to API Server "172.17.0.2:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://172.17.0.2:6443"
[discovery] Failed to request cluster info, will try again: [the server was unable to return a response in the time allotted, but may still be processing the request (get configmaps cluster-info)]
There are other errors later, which I originally though were a side-effect of the nasty looking FATAL error e.g. .... "[util/etcd] Attempt timed out"]} but I now think root cause is Etcd part times out sometimes.
Adding this answer in case someone else puzzled like I was.
I am using Docker 1.13 community edition on a CentOS 7 x64 machine. When I was following a Docker Compose sample from Docker official tutorial, all things were OK until I added these lines to the docker-compose.yml file:
volumes:
- .:/code
After adding it, I faced the following error:
can't open file 'app.py': [Errno 13] Permission denied. It seems that the problem is due to a SELinux limit. Using this post I ran the following command:
su -c "setenforce 0"
to solve the problem temporarily, but running this command:
chcon -Rt svirt_sandbox_file_t /path/to/volume
couldn't help me.
Finally I found the correct rule to add to SELinux:
# ausearch -c 'python' --raw | audit2allow -M my-python
# semodule -i my-python.pp
I found it when I opened the SELinux Alert Browser and clicked on 'Details' button on the row related to this error. The more detailed information from SELinux:
SELinux is preventing /usr/local/bin/python3.4 from read access on the
file app.py.
***** Plugin catchall (100. confidence) suggests **************************
If you believe that python3.4 should be allowed read access on the
app.py file by default. Then you should report this as a bug. You can
generate a local policy module to allow this access. Do allow this
access for now by executing:
ausearch -c 'python' --raw | audit2allow -M my-python
semodule -i my-python.pp
I use an ansible script to load & start the https://hub.docker.com/r/rastasheep/ubuntu-sshd/ container.
so it starts well of course :
bash-4.4$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8bedbd3b7d88 rastasheep/ubuntu-sshd "/usr/sbin/sshd -D" 37 minutes ago Up 36 minutes 0.0.0.0:49154->22/tcp test
bash-4.4$
so after ansible failure on ssh access to it I tested manually from shell
this is also ok.
bash-4.4$ ssh root#172.17.0.2
The authenticity of host '172.17.0.2 (172.17.0.2)' can't be established.
ECDSA key fingerprint is SHA256:YtTfuoRRR5qStSVA5UuznGamA/dvf+djbIT6Y48IYD0.
ECDSA key fingerprint is MD5:43:3f:41:e9:89:45:06:6f:f6:42:c4:6a:70:37:f8:1d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.17.0.2' (ECDSA) to the list of known hosts.
root#172.17.0.2's password:
root#8bedbd3b7d88:~# logout
Connection to 172.17.0.2 closed.
bash-4.4$
so the step that failed is trying to get on it from ansible script & make access to ssh-copy-id
ansible error message is :
Fatal: [172.17.0.2]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '172.17.0.2' (ECDSA) to the list of known hosts.\r\nPermission denied (publickey,password).\r\n", "unreachable": true}
---
- hosts: 127.0.0.1
tasks:
- name: start docker service
service:
name: docker
state: started
- name: load and start the container we wanna use
docker_container:
name: test
image: rastasheep/ubuntu-sshd
state: started
ports:
- "49154:22"
- name: Wait maximum of 300 seconds for ports to be available
wait_for:
host: 0.0.0.0
port: 49154
state: started
- hosts: 172.17.0.2
vars:
passwordadmin: $6$pbE6yznA$AeFIdI.....K0
passwordroot: $6$TMrxQUxT$I8.JIzR.....TV1
ansible_ssh_extra_args: "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
tasks:
- name: Build test container root user rsa ssh-key
shell: docker exec test ssh-keygen -b 2048 -t rsa -f /root/.ssh/id_rsa -q -N ""
so I cannot even run the needed step to build ssh
how to do then ??
1st step (ansible task) : load docker container
2cd step (ansible task on only 172.17.0.2) : connect to it & setup it
there will be 3rd step to run application on it after that.
the problem occurs only when starting the 2cd step
Ok after many trys on a second container
conclusion is my procedure was bad
what I have done to solve that :
build a diroctory tree separating ./ ./inventory ./includes
build 1 yaml file by host (local, docker, labo)
build 1 main yaml file on ./
build 1 new host file in ./inventory
connect forced by sshpass to docker on default password
changed it
add the host key on authorized key to a login dedicated usage
installed pyhton (needed to answer ansible host else it makes
randomly module errors or refused connections depending on current
action)
setup a ssh login user in sudoers
then I can un the docker.yaml actions
then only at last I can run the labo.yaml actions.
Thanks for help
now I'm able to build the missing tools.
When running docker run hello-world I get an "Incorrect Usage" error (full output pasted below). I'm running the following:
Docker 17.05.0-ce, build 89658be
docker-containerd 0.2.3 (commit 9048e5e)
runc v1.0.0-rc4
Linux kernel 4.1.15
Using buildroot 2017.11 (commit 1f1a242) to generate custom toolchain/rootfs
systemd 234
Seems as though I can pull the hello-world image down properly, as it is included in docker images output. Wondering if there is an incompatibility between docker/containerd/runc? Or maybe something obvious? First time working with docker.
Additionally, I've run a docker check-config.sh script I found that states the only kernel configuration features I'm missing are optional. They are CONFIG_CGROUP_PIDS, CONFIG_CGROUP_HUGETLB, CONFIG_AUFS_FS, /dev/zfs, zfs command, and zpool command. Everything else, including all required, are enabled.
Output:
# docker run hello-world
[ 429.332968] device vethc0d83d1 entered promiscuous mode
[ 429.359681] IPv6: ADDRCONF(NETDEV_UP): vethc0d83d1: link is not ready
Incorrect Usage.
NAME:
docker-runc create - create a container
USAGE:
docker-runc create [command options] <container-id>
Where "<container-id>" is your name for the instance of the container that you
are starting. The name you provide for the container instance must be unique on
your host.
DESCRIPTION:
The create command creates an instance of a container for a bundle. The bundle
is a directory with a specification file named "config.json" and a root
filesystem.
The specification file includes an args parameter. The args parameter is used
to specify command(s) that get run when the container is started. To change the
command(s) that get executed on start, edit the args parameter of the spec. See
"runc spec --help" for more explanation.
OPTIONS:
--bundle value, -b value path to the root of the bundle directory, defaults to the current directory
--console-socket value path to an AF_UNIX socket which will receive a file descriptor referencing the master end of the console's pseudoterminal
--pid-file value specify the file to write the process id to
--no-pivot do not use pivot root to jail process inside rootfs. This should be used whenever the rootfs is on top of a ramdisk
--no-new-keyring do not create a new session keyring for the container. This will cause the container to inherit the calling processes session key
--preserve-fds value Pass N additional file descriptors to the container (stdio + $LISTEN_FDS + N in total) (default: 0)
flag provided but not defined: -console
[ 429.832198] docker0: port 1(vethc0d83d1) entered disabled state
[ 429.849301] device vethc0d83d1 left promiscuous mode
[ 429.859317] docker0: port 1(vethc0d83d1) entered disabled state
docker: Error response from daemon: oci runtime error: flag provided but not defined: -console.
The -console option was replaced with --console-socket in runc Dec 2016 for v1.0.0-rc4.
So I would guess you need an older version of runc or a newer version of Docker.
If you are building Docker yourself, use Docker 17.09.0-ce or an older release of runc. I'm not sure if that's v0.1.1 or just an earlier 1.0 like v1.0.0-rc2
If you were upgrading packages, something has gone wrong with the install. Probably purge everything and reinstall Docker.
I am trying to run ruby on rails under passenger with apache2 under fedora 19 and I got this error in log:
[Tue Feb 25 09:37:52.367683 2014] [passenger:error] [pid 2779] ***
Passenger could not be initialized because of this error: Unable to
start the Phusion Passenger watchdog because it encountered the
following error during startup: Cannot change the directory
'/tmp/passenger.1.0.2779/generation-1/buffered_uploads' its UID to 48
and GID to 48: Operation not permitted (errno=1)
That directory (/tmp/passenger.1.0.2779) doesn't even exist. I think that problem is with selinux. I tried to solve it about 4 hours. Httpd is running under user apache and group apache, I tried:
cat /var/log/audit/audit.log | grep passenger | audit2allow -M
passenger semodule -i passenger.pp
but still nothing.
In your case, you should switch SELinux into Permissive mode at first, then try to capture the audit log from starting Apache to run your application.1
Once you got the home page of your application, you can build your custom policy with the logs.
Switch SELinux into Permissive mode and clean audit.log
]# setenforce 0
]# rm /var/log/audit/audit.log
]# service auditd restart
Restart Apache
]# service httpd restart
Try to open your application with a web browser
It might give more information about what is happenning when you application is running.
Make a custom policy module to allow these actions
]# mkdir work
]# cd work
]# grep httpd /var/log/audit/audit.log | audit2allow -M passenger
]# ls
passenger.pp passenger.te
Load postgrey policy module using the 'semodule' command into the current SELinux policy:
]# semodule -i passenger.pp
]# setenforce 1
Restart Apache
]# service httpd restart
References:
http://wiki.centos.org/HowTos/SELinux#head-faa96b3fdd922004cdb988c1989e56191c257c01
I ran into a similar error, with a startup error about being unable to create a directory that did not exist. (logs, not tmp, but same sort of thing) I, too, battled with it for an hour and couldn't make sense of it. I created/deleted/chmod the directory many ways without success.
The fix for me was to change the parameters to passenger-start. Initially, my Docker container started passenger with:
exec bundle exec passenger start --auto --disable-security-update-check --min-instances 20 --max-pool-size 20 --max-request-queue-size 500
I removed all parameters, leaving just this:
exec bundle exec passenger start
At this point, passenger could create the log folder and file, and all was well. I could have restored the params at this point, but we decided they were not needed for the development environment so left them out moving ahead.
In hindsight, I have a hunch that I deleted the log directory while a file in it was still open, and the file system persisted that condition in some way. But that's just a hunch. Perhaps simply rebooting my Mac would have fixed it...