I have RedisDB that exists in a Docker container. Database restored from dump.rdb file and everything works well (I can get all data that stored in Redis before) until Redis tries to save data in the background. Log here:
1:M 28 Jan 2021 11:28:28.035 * 100 changes in 300 seconds. Saving...
1:M 28 Jan 2021 11:28:28.349 * Background saving started by pid 21
21:C 28 Jan 2021 11:30:23.589 * DB saved on disk
21:C 28 Jan 2021 11:30:23.930 * RDB: 0 MB of memory used by copy-on-write
And after that, I see in the folder a new file: backup.db, and all data I get from this file (instead of dump.rdb).
redis.conf:
appendonly yes
appendfilename "appendonly.aof"
dbfilename dump.rdb
dir ./
save 900 1
save 300 10
save 60 10000
Docker run command:
docker run --name redis -p 6379:6379 -v /redis/data:/data -v /redis/config/redis.conf:/usr/local/etc/redis/redis.conf -d redis
dump.rdb file stored into /redis/data folder and redis.conf stored into /redis/config folder
Redis version: 6.0.10
Question: How to avoid creation backup.db file?
Related
I am trying to launch a redis docker container using docker-compose, but I always get this error. This is my docker-compose run commands docker-compose -f docker-compose.yml build and docker-compose -f docker-compose.yml up -d --force-recreate. I am running the docker containers on aws ecs and using a t2.micro ec2 instance. I am not sure if that is the reason why. Any insight would be helpful.
I have also included my docker-compose.yml
version: '2.1'
services:
redis:
image: redis:latest
container_name: redis
volumes:
- redis_data:/data
ports:
- 6379:6379
app:
image: custom_image
build: .
depends_on:
redis:
condition: service_started
ports:
- 8003:8003
links:
- redis
volumes:
redis_data:
Error
1:C 11 Sep 00:18:34.345 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 11 Sep 00:18:34.348 # Redis version=4.0.11, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 11 Sep 00:18:34.348 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 11 Sep 00:18:34.349 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
1:M 11 Sep 00:18:34.349 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
1:M 11 Sep 00:18:34.349 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
1:M 11 Sep 00:18:34.350 * Running mode=standalone, port=6379.
1:M 11 Sep 00:18:34.350 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 11 Sep 00:18:34.350 # Server initialized
1:M 11 Sep 00:18:34.350 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 11 Sep 00:18:34.350 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 11 Sep 00:18:34.350 * Ready to accept connections
1:signal-handler (1536625117) Received SIGTERM scheduling shutdown...
1:M 11 Sep 00:18:37.375 # User requested shutdown...
1:M 11 Sep 00:18:37.375 * Saving the final RDB snapshot before exiting.
1:M 11 Sep 00:18:37.378 * DB saved on disk
1:M 11 Sep 00:18:37.378 # Redis is now ready to exit, bye bye...
Ran into the same issues. After a bit of digging we found that it was killed by systemd due to inactive.
Run systemctl show docker.service command show that the inactive and active enter timestamp match up with when the redis service stop and start again.
InactiveEnterTimestamp=Tue 2021-08-03 22:07:19 AEST
ActiveEnterTimestamp=Wed 2021-08-04 09:30:36 AEST
Our solution is just to perform some activity on redis so that it doesn't enter inactive state.
i've a docker with redis container
configuration of it
docker-compose.yml
# Redis
redis:
image: redis:4.0.6
build:
context: .
dockerfile: dockerfile_redis
volumes:
- "./redis.conf:/usr/local/etc/redis/redis.conf"
ports:
- "6379:6379"
dockerfile_redis
CMD ["chown", "redis:redis", "-R", "/etc"]
CMD ["chown", "redis:redis", "-R", "/var/lib"]
CMD ["chown", "redis:redis", "-R", "/run"]
CMD ["sudo", "chmod", "644", "/data/dump.rdb" ]
CMD ["sudo", "chmod", "755", "/etc" ]
CMD ["sudo", "chmod", "770", "/var/lib" ]
CMD ["sudo", "chmod", "777", "/run" ]
CMD [ "redis-server", "/usr/local/etc/redis/redis.conf" ]
Also i use django and celery, when celery works 4-6 hours, container of celery stopped, with error:
[2018-03-05 17:18:24,516: CRITICAL/MainProcess] Unrecoverable error: ResponseError('MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error.',)
Traceback (most recent call last):
File "/usr/local/lib/python3.4/site-packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
File "/usr/local/lib/python3.4/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/usr/local/lib/python3.4/site-packages/celery/bootsteps.py", line 370, in start
return self.obj.start()
File "/usr/local/lib/python3.4/site-packages/celery/worker/consumer/consumer.py", line 320, in start
blueprint.start(self)
File "/usr/local/lib/python3.4/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/usr/local/lib/python3.4/site-packages/celery/worker/consumer/consumer.py", line 596, in start
c.loop(*c.loop_args())
File "/usr/local/lib/python3.4/site-packages/celery/worker/loops.py", line 88, in asynloop
next(loop)
File "/usr/local/lib/python3.4/site-packages/kombu/async/hub.py", line 354, in create_loop
cb(*cbargs)
File "/usr/local/lib/python3.4/site-packages/kombu/transport/redis.py", line 1040, in on_readable
self.cycle.on_readable(fileno)
File "/usr/local/lib/python3.4/site-packages/kombu/transport/redis.py", line 337, in on_readable
chan.handlers[type]()
File "/usr/local/lib/python3.4/site-packages/kombu/transport/redis.py", line 714, in _brpop_read
**options)
File "/usr/local/lib/python3.4/site-packages/redis/client.py", line 680, in parse_response
response = connection.read_response()
File "/usr/local/lib/python3.4/site-packages/redis/connection.py", line 629, in read_response
raise response
redis.exceptions.ResponseError: MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error.
Import Error
-------------- celery#b17b82a69031 v4.1.0 (latentcall)
---- **** -----
--- * *** * -- Linux-4.4.0-34-generic-x86_64-with-debian-8.9 2018-03-05 07:24:00
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: backend:0x7f19e5745208
- ** ---------- .> transport: redis://redis:6379/0
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 20 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. CallbackNotifier
. FB posting
. FB token status
. MD posting
. MD token status
. OK posting
. OK token status
. TW posting
. TW token status
. VK posting
. VK token status
. api.controllers.message.scheduled_message
. backend.celery.debug_task
. stats.views.collect_stats
In my redis.conf file i disable snapshots
stop-writes-on-bgsave-error no
In redis logs:
1:M 06 Mar 07:40:04.037 * Background saving started by pid 8228
8228:C 06 Mar 07:40:04.038 # Failed opening the RDB file backupall.db (in server root dir /run) for saving: Permission denied
But, when i restart redis container i've get some warnings:
1:C 06 Mar 08:12:48.982 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 06 Mar 08:12:48.982 # Redis version=4.0.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 06 Mar 08:12:48.982 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 06 Mar 08:12:48.986 * Running mode=standalone, port=6379.
1:M 06 Mar 08:12:48.986 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 06 Mar 08:12:48.986 # Server initialized
1:M 06 Mar 08:12:48.987 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 06 Mar 08:12:48.988 * DB loaded from disk: 0.001 seconds
1:M 06 Mar 08:12:48.988 * Ready to accept connections
Permissions in dockerfile_redis is correct?
How configurate redis with my conf file?
What also i need to make the redis work well?
Please check this blogpost:
https://blog.huntingmalware.com/notes/LLMalware
It is very likely a malware causing the working directory of your redis to change, and redis tries to write RDB file to a directory owned by root, following the commands of a malicious script. As it does not run from root, and write access to /run directory is not granted to user 'redis', the writing fails.
So, do not expose your Redis server port to the Internet and it should fix the issue with malware being able to reach it.
If you do not really need to expose ports, just remove next lines:
ports:
- "6379:6379"
Simple question: Is there a docker command to view the files inside a volume?
I run docker for windows which creates a MobyLinuxVM on my machine to run Docker. I can't get a remote desktop connection onto this machine like I can with an Ubuntu VM (which I also have running on my machine).
Therefore, I can't see a way to see what is inside my host volumes (as they are actually inside the MobyLinuxVM), where as if I ran docker on my Ubuntu VM I could remote onto the machine and take a look.
Therefore, is there a way I can run some sort of docker volume command to list what's inside each volume?
You can use a temporary container for this. I tend to use busybox for these temporary containers:
$ docker volume ls
DRIVER VOLUME NAME
local jenkins-home
local jenkins-home2
local jenkinsblueocean_jenkins-data
...
$ docker run -it --rm -v jenkins-home:/vol busybox ls -l /vol
total 428
-rw-r--r-- 1 1000 1000 327 Jul 14 2016 com.dabsquared.gitlabjenkins.GitLabPushTrigger.xml
-rw-r--r-- 1 1000 1000 276 Aug 17 2016 com.dabsquared.gitlabjenkins.connection.GitLabConnectionConfig.xml
-rw-r--r-- 1 1000 1000 256 Aug 17 2016 com.nirima.jenkins.plugins.docker.DockerPluginConfiguration.xml
drwxr-xr-x 28 1000 1000 4096 Aug 17 2016 config-history
-rw-r--r-- 1 1000 1000 6460 Aug 17 2016 config.xml
-rw-r--r-- 1 1000 1000 174316 Jun 2 18:50 copy_reference_file.log
-rw-r--r-- 1 1000 1000 2875 Aug 9 2016 credentials.xml
...
For a host volume, you can just replace the volume mount with the host directory name (fully qualified) in the docker run cli.
$ docker run -it --rm -v /path/on/host:/vol busybox ls -l /vol
This isn't a direct answer to the question (because it was asking about a docker command) but in case anyone arrives here like I did:
If you have Docker Desktop (on Windows at least) you can explore into a volume using the Docker Desktop GUI. Just click on the volume, then switch to the "Data" tab at the top.
Quick and easy if you are just wanting to take a look around or copy out a file.
Not sure how widely applicable this is, but if you have root access I've just discovered that you can browse the contents of a volume at /var/lib/docker/volumes/<VOLUME_NAME>/_data. VOLUME_NAME is as shown by docker volume ls.
I'm looking at an Ubuntu 18.04 VM running Docker 19.03.5 - YMMV.
I've been looking into setting up a data volume for a Docker container that I'm running on my server. The container is from this FreePBX image https://hub.docker.com/r/jmar71n/freepbx/
Basically I want persistent data so I don't lose my VoIP extensions and settings in the case of Docker stopping. I've tried many guides, ones here on stack overflow, and on the Docker manpages, but I just can't quite get it to work.
Can anyone help me with what commands I need to run in order to attach a volume to the FreePBX image I linked above?
You can do this by running a container with the -v option and mapping to a host directory - you just need to know where the container's storing the data.
Looking at the Dockerfile for that image, I'm assuming that the data you're interested in is stored in MySql. In the MySql config the data directory the container's using is /var/lib/mysql.
So you can start your container like this, mapping the MySql data directory to /docker/pbx-data on your host:
> docker run -d -t -v /docker/pbx-data:/var/lib/mysql jmar71n/freepbx
20b45b8fb2eec63db3f4dcab05f89624ef7cb1ff067cae258e0f8a910762fb1a
Use docker inpect to confirm that the mount is mapped as expected:
> docker inspect --format '{{json .Mounts}}' 20b
[{"Source":"/docker/pbx-data",
"Destination":"/var/lib/mysql",
"Mode":"","RW":true,"Propagation":"rprivate"}]
When the container runs it bootstraps the database, so on the host you'll be able to see the contents of the MySql data directory the container is using:
> ls -l /docker/pbx-data
total 28684
-rw-r----- 1 103 root 2062 Sep 21 09:30 20b45b8fb2ee.err
-rw-rw---- 1 103 messagebus 18874368 Sep 21 09:30 ibdata1
-rw-rw---- 1 103 messagebus 5242880 Sep 21 09:30 ib_logfile0
-rw-rw---- 1 103 messagebus 5242880 Sep 21 09:30 ib_logfile1
drwx------ 2 103 root 4096 Sep 21 09:30 mysql
drwx------ 2 103 messagebus 4096 Sep 21 09:30 performance_schema
If you kill the container and run another one with the same volume mapping, it will have all the data files from the previous container, and your app state should be preserved.
I'm not familiar with FreePBX, but if there is state being stored in other directories, you can find the locations in config and map them to the host in the same way, with multiple -v options.
Hi Elton Stoneman and user3608260!
Yes, you assuming correctly for data saves in Mysql (records, users, configs, etc.).
But in asterisk, all configurations are saved in files '.conf' and similars.
In this case, the archives looked for user3608260 are storaged in '/etc/asterisk/*'
Your answer is perfectly with more one command: -v /local_to_save:/etc/asterisk
the final docker command:
docker run -d -t -v /docker/pbx-data:/var/lib/mysql -v /docker/pbx-asterisk:/etc/asterisk jmar71n/freepbx
[Assuming /docker/pbx-asterisk is a host directory. ]
I'll preface this with the fact that I haven't used PostgreSQL much. I tried using it with RoR but the fact that it uses an ORM, I never got why PostgreSQL was the flavor of choice.
After fighting with getting the damn thing installed on Ubuntu 14.04, I need to clone a repo that depends on it.
After about 30 minutes of dealing trying a few things, I discovered:
$ /usr/lib/postgresql/9.4/bin/postgres -d 3 -D /var/lib/postgresql/9.4/main -c config_file=/etc/postgresql/9.4/main/postgresql.conf
LOG: skipping missing configuration file "/var/lib/postgresql/9.4/main/postgresql.auto.conf"
2015-02-14 21:05:01 PST [7665-2] FATAL: data directory "/var/lib/postgresql/9.4/main" has wrong ownership
2015-02-14 21:05:01 PST [7665-3] HINT: The server must be started by the user that owns the data directory.
2015-02-14 21:05:01 PST [7665-4] DEBUG: shmem_exit(1): 0 before_shmem_exit callbacks to make
2015-02-14 21:05:01 PST [7665-5] DEBUG: shmem_exit(1): 0 on_shmem_exit callbacks to make
2015-02-14 21:05:01 PST [7665-6] DEBUG: proc_exit(1): 0 callbacks to make
2015-02-14 21:05:01 PST [7665-7] DEBUG: exit(1)
One, I don't know what this auto.conf file it's looking for as I'm specifying the conf file.
However... (edited to what I think are the appropriate line[s])
$ sudo gedit /etc/postgresql/9.4/main/pg_hba.conf
local all postgres 127.0.0.1 peer
(I added in the local IP after nothing working. Still doesn't work.)
And (/etc/postgresql/9.4/main/)
-rw-r--r-- 1 postgres postgres 315 Feb 14 20:20 environment
-rw-r--r-- 1 postgres postgres 143 Feb 14 20:20 pg_ctl.conf
-rw-r----- 1 postgres postgres 4641 Feb 14 20:55 pg_hba.conf
-rw-r----- 1 postgres postgres 4641 Feb 14 20:20 pg_hba.conf~
-rw-r----- 1 postgres postgres 1636 Feb 14 20:20 pg_ident.conf
-rw-r--r-- 1 postgres postgres 21461 Feb 14 20:20 postgresql.conf
-rw-r--r-- 1 postgres postgres 378 Feb 14 20:20 start.conf
Seems to me the configuration files are owned by postgres. What gives?
Update (9:30p)
Running the following command (as postgres) gives the same result.
$ su - postgres; /usr/lib/postgresql/9.4/bin/postgres -d 3 -D /var/lib/postgresql/9.4/main -c config_file=/etc/postgresql/9.4/main/postgresql.conf
Judging from the error message, ownership for the data directory seems to be misconfigured. If so, fix with (as privileged system user):
chown postgres:postgres /var/lib/postgresql/9.4
chown postgres:postgres /var/lib/postgresql/9.4/main
Use the "recursive" option -R if anything inside those directories is owned by different users.