I'm trying to avoid a list of 'command' modules in my ansible play, but there seems to be a void of ansible docs regarding tar/gz and the synch module seems... incomplete.
I'd like to gzip a tarball of a big directory then rsync it to another host. Even using 'command' seems to not work for me :<
"warnings": ["Consider using unarchive module rather than running tar"]}
[WARNING]: Consider using unarchive module rather than running tar
PLAY RECAP *********************************************************************
ctarlctarl : ok=2 changed=0 unreachable=0 failed=1
The 'unarchive' module seems to expect an already compressed/archived directory and doesn't appear to be the solution I want.
Related but unanswered: ansible playbook unable to continue as the `tar` fails due to `file change as we read`
(Edit) showing the task since it was asked if I remembered the z. =)
- name: tar ball app dir, exclude conf files
command: "tar -zcvf {{ item.code_dir }}.tar.gz --exclude '*config*' ."
args:
chdir: "{{ apps_home }}/{{ item.code_dir }}"
with_items:
- "{{ processes }}"
In version 2.2 ansible will get an archive module: https://docs.ansible.com/ansible/archive_module.html
With that you can archive, transfer and unarchive all with ansible modules and don't have to fiddle with command line arguments.
So, I got it working... just inelegantly.
With the v flag set, my ansible stdout was horrendously verbose (even without calling -vvvv with ansible-playbook). Turning off the switch inside the tar command made tracking down the problem easier.
The problem was actually the error "error: file changed as we read it" - no doubt because I was adding the archive file into the directory it was compressing. I solved it by simply moving the archived file up a directory when running.
This, however, leaves me with the 'command after command' solution - which is what I hope I'll eventually be able to avoid. Half way there, though.
This warning tip you can use the ansible unarchive module instead of tar commnad. The syntax is very easy just like below:
- unarchive: src=foo.tgz dest=/var/lib/foo
And more detail info you can got from here: unarchive_module
Related
I have a container that I want to export as a .tar file. I have used a podman run with a tar --exclude=/dir1 --exclude=/dir2 … that outputs to a file located on a bind-mounted host dir. But recently this has been giving me some tar: .: file changed as we read it errors, which podman/docker export would avoid. Besides the export I suppose is more efficient. So I'm trying to migrate to using the export, but the major obstacle is I can't seem to find a way to exclude paths from the tar stream.
If possible, I'd like to avoid modifying a tar archive already saved on disk, and instead modify the stream before it gets saved to a file.
I've been banging my head for multiple hours, trying useless advices from ChatGPT, looking at cpio, and attempting to pipe the podman export to tar --exclude … command. With the last I did have small success at some point, but couldn't make tar save the result to a particularly named file.
Any suggestions?
(note: I do not make distinction between docker and podman here as their export command is completely the same, and it's useful for searchability)
We have a Dockerfile which copies the folder to another folder.
Something like
COPY . /mycode
But the problem is that there is tons of files in the generated code, and it creates 10K+ lines on the jenkins log where we are running the CICD pipeline.
copying /bla/blah/bla to copying /bla/blah/bla 10k times.
Is there a way to make this COPY less verbose or silent? jenkins admin has already warned us that our log file is nearing his max limit.
You can tar/zip the files on the host, so there's only one file to copy. Then untar/unzip after it's been copied and direct the output of untar/unzip to /dev/null.
By definition the docker cp command doesn't have any "silent" switches, but perhaps redirecting the output may help, have you tried:
docker cp -a CONTAINER:SRC_PATH DEST_PATH|- &> /dev/null
I know it's not the most elegant, but if what you seek is to supress the console output, that may help.
Sources
[1] https://docs.docker.com/engine/reference/commandline/cp/
[2] https://csatlas.com/bash-redirect-stdout-stderr/
This error appears randomly when I'm working with docker-compose on Windows 10, sometimes after pycharm already working with docker-compose as interpreter.
I tried:
Make sure docker-compose file is valid, without tabs instead of spaces.
Use yml and yaml suffixes (sometimes yaml works and yml doesn't, sometimes both are working or not working)
Add project-compose to configuration files.
Problem is 'solved' just after rebooting, and then happened again.
Linux here
tl;dr: same solution as for windows
check path to docker-compose executable:
➤ which docker-compose
/home/voy/.pyenv/shims/docker-compose
Go to File | Settings | Build, Execution, Deployment | Docker | Tools | Docker Compose Executable and paste docker compose executable path from above
Restart pycharm
Here is a JetBrains issue about this:
https://youtrack.jetbrains.com/issue/WI-49652
And another post:
https://intellij-support.jetbrains.com/hc/en-us/community/posts/360000129510-Couldn-t-refresh-skeletons-for-remote-interpreter-Docker
They suggest multiple things, first I had to change the docker-compose executable path, as PyCharm found the docker-compose.txt first, I needed to set it to docker-compose.exe.
After this the problem still occured from time to time, but restarting PyCharm fixed it. Though it takes for a few minutes to index everything and reload the project.
Line endings can also be an issue, if in the docker-compose.yml set to use CRLF instead of LF, that can be a cause to fail parsing as well. I suggest to use a .editorconfig file to control your line endings, that seemed to help as well. Also setting your git autocrlf to 'input' might help if you use Windows.
Slowest one is posted on the forum:
remove pycharm helper containers: $ docker rm -f $(docker ps -a | grep pycharm_helper | awk '{print $1}')
invalidate caches and restart PyCharm
No a great solutions yet as I know, unfortunately.
After reading this thread, I am not able to 100% foward my key, I'm blocked at step where I should run "ssh-add".
Here is a part of my docker-compose.yml:
engine:
build: ./docker/engine/
volumes:
- "./:/var/www/docker:rw"
- "./docker/engine/php.ini:/usr/local/etc/php/conf.d/custom.ini:ro"
- $SSH_AUTH_SOCK:/tmp/agent.sock
environment:
- SSH_AUTH_SOCK=/tmp/agent.sock
On my container, I can see without problem the file "/tmp/agent.sock" and the ENV var "SSH_AUTH_SOCK" when I print them with printenv.
After that I start ssh-agent with:
`eval `ssh-agent -s`
Then I add the ssh-key:
ssh-add
But if I print the last error (echo $?), I get 1.
From doc: Exit status is 0 on success, 1 if the specified command fails, and 2
if ssh-add is unable to contact the authentication agent.
I tried so much thing, I have no other ideas, I have visited all the possible links on google (damn).
If you have a solution (or path to one), you will be my hero!
Thanks.
Same here,
No issue on linux while sharing ~/.ssh via volume, on windows tho it's another story. I managed to have my keys in the /.ssh folder in the container, but docker consider they are too opened (chmod > 600) so it's impossible to composer install or stuff like that. I don't know if this could be related to the way linux read NTFS Permission or something like that.
Either way, it's not working and I tried a lots of permission change
Edit: I've post this in Github, we might get some help from there :
https://github.com/docker/docker/issues/27685
I`ve created simple script which is based on inotify-tools, but finally after when i decided to monitor /remotepath, which was mounted from NAS by command mount.cifs, it wasnt work.
So after some investigation i found information, that inotify-tools has not support for remote folder.
Does any one of You have any expirience with simple tool which will give me a chance, to watch remote folder, and if something will change, then will run rsync.
Maybe i should go only with rsync and sync remote folder with new files only ?
Thanks for any ideas.
In the mean time i created some simple bash script which doing this what i want, but i fighting with a problem, what will happend if something will be deleted from destination folder and i dont want to synchronize this deleted file again.
Any idea how to fix this problem ?
#!/bin/bash
### Logs path
path="/var/log/compare"
log="compare.log"
listing1="listing1.log"
listing2="listing2.log"
### Path which will be monitored
destination="/path/to/destination/"
source="/path/to/remote/folder"
## Watching for content in source folder
ls -lh $source > $path/$listing1
### I`m checking if something was changed
echo "$(date)" 'INFO' 'I will compare listing files' >> "$path/$log"
if cmp -s "$path/$listing1" "$path/$listing2"
### Files are the same
then
echo "$(date)" 'INFO' 'Listings are the same' >> "$path/$log"
### Files are different
else
rsync -art $source $destination
echo "$(date)" 'INFO' 'Finished synchronization' >> "$path/$log"
fi
cp $path/$listing1 $path/$listing2
inotify is indeed the wrong tool for the job; it works by intercepting filesystem activity in the kernel, so remote activity will be totally missed.
An ideal solution would be to run inotify on the NAS box. This is certainly possible with some devices, but you don't say what device you have (and if you did I probably don't have the same one).
Any other tool that exists will just do exactly what your script does (albeit maybe more prettily), so I don't think you need to pursue that avenue.
In any case, your script is entirely redundant! If you just ran rsync periodically it would do exactly the same thing (rsync -t just compares the file names, sizes, and timestamps). The only difference is that rsync will compare the remote file list against your real files, not a cached copy of the file-list. Incidentally, -a implies both -r and -t, so rsync -a is sufficient.
One thing I would look into: running rsync on the NAS device directly. Accessing the file list through CIFS is less efficient that running it locally, so if your NAS can support rsync then do it that way. (Synology NAS boxes have rsync, but I don't know about other devices.)