I have a host where an application creates model directories/files. I want to tar up those files and send it over to another host for DR backup.
So far I have the following:
sudo tar -zcvf - /mnt/nfsshared | ssh app#<ip-address> "cat > /data/models/models_$(date +\%Y-\%m-\%d).tar.gz" > /dev/null 2>&1
This command will tar up all directories and files in /mnt/nfsshared and create the tarball on the remote host under /data/models/
However, I want to exclude the the /mnt/nfsshared/ parent directories and just tar up the directories/files in the /nfsshared directory.
I know I could cd to /mnt/nfshared and tar up all files.
I can also do this command
sudo -C /mnt/nfsshared -zcvf | ssh app#<ip-address> "cat > /data/models/models_$(date +\%Y-\%m-\%d).tar.gz" > /dev/null 2>&1
which would only tar up directories/files under nfsshared. However, what argument would I pass to the command: -zcvf
I can't use a tarball name since the tarball gets created on the remote host.
Within /mnt/nfsshared/ I have multiple directories containing additional subdirectories and files.
/mnt/nfsshared/model1/files
/model2/files
/model3/submodel1/files
/submodel2/files
/submodel3/files
/model4/submodel1/files
I want only to tar up the model, submodel directories and the files I want to exclude the /mnt/nfsshared directories
Cheers,
Roland
Related
I typically do :
tar czf $directory_path/file_${1}.tar.gz --directory=$specified_path/ *${1}.csv .
What if I just want to include only certain files of a directory, but not the directory itself?
cd [dir]; tar cz ./* >x.tgz
or
cd [dir]; tar cz file1 file2 >x.tgz
I want to use wildcard to select multiple files from a directory in a container and use docker cp to copy these files from container to docker host.
I couldn't find if support for using wildcard is available with docker cp yet or not.
docker cp fd87af99b650:/foo/metrics.csv* /root/metrices_testing/
This results with the error metrics.csv*: no such file or directory
I came across an example where for loop was used to select a few files and then sent to container, but i want to transfer files from container to host and want to do this on docker host itself as script is running on host only.
Using docker exec to select files first and then copying them using docker cp can be an option. But that is a 2 step process.
Can someone please help me do this in one step?
EDIT:
I tried this. A step close but still failing.
# for f in $(docker exec -it SPSRS bash -c "ls /opt/tpa/logs/metrics.csv*");
do docker cp SPSRS:$f /root/metrices_testing/;
done
: no such file or directory lstat /docker/overlay2/193d2ad0d8d087377e3b96cbfb672b0e39132ae5e961872127614c9396f8c068/merged/opt/SPS_18_5_R1/logs/metrics.csv.2018.07.10-08:45
: no such file or directory lstat /docker/overlay2/193d2ad0d8d087377e3b96cbfb672b0e39132ae5e961872127614c9396f8c068/merged/opt/SPS_18_5_R1/logs/metrics.csv.2018.07.10-09:00
: no such file or directory lstat /docker/overlay2/193d2ad0d8d087377e3b96cbfb672b0e39132ae5e961872127614c9396f8c068/merged/opt/SPS_18_5_R1/logs/metrics.csv.2018.07.10-09:15
: no such file or directory lstat /docker/overlay2/193d2ad0d8d087377e3b96cbfb672b0e39132ae5e961872127614c9396f8c068/merged/opt/SPS_18_5_R1/logs/metrics.csv.2018.07.10-09:30
: no such file or directory lstat /docker/overlay2/193d2ad0d8d087377e3b96cbfb672b0e39132ae5e961872127614c9396f8c068/merged/opt/SPS_18_5_R1/logs/metrics.csv.2018.07.10-09:45
: no such file or directory lstat /docker/overlay2/193d2ad0d8d087377e3b96cbfb672b0e39132ae5e961872127614c9396f8c068/merged/opt/SPS_18_5_R1/logs/metrics.csv.2018.07.10-10:00
: no such file or directory lstat /docker/overlay2/193d2ad0d8d087377e3b96cbfb672b0e39132ae5e961872127614c9396f8c068/merged/opt/SPS_18_5_R1/logs/metrics.csv.2018.07.10-10:15
: no such file or directory lstat /docker/overlay2/193d2ad0d8d087377e3b96cbfb672b0e39132ae5e961872127614c9396f8c068/merged/opt/SPS_18_5_R1/logs/metrics.csv.2018.07.10-10:30
: no such file or directory lstat /docker/overlay2/193d2ad0d8d087377e3b96cbfb672b0e39132ae5e961872127614c9396f8c068/merged/opt/SPS_18_5_R1/logs/metrics.csv.2018.07.10-10:45
: no such file or directory lstat /docker/overlay2/193d2ad0d8d087377e3b96cbfb672b0e39132ae5e961872127614c9396f8c068/merged/opt/SPS_18_5_R1/logs/metrics.csv.2018.07.10-11:00
: no such file or directory lstat /docker/overlay2/193d2ad0d8d087377e3b96cbfb672b0e39132ae5e961872127614c9396f8c068/merged/opt/SPS_18_5_R1/logs/metrics.csv.2018.07.10-11:15
: no such file or directory lstat /docker/overlay2/193d2ad0d8d087377e3b96cbfb672b0e39132ae5e961872127614c9396f8c068/merged/opt/SPS_18_5_R1/logs/metrics.csv.2018.07.10-11:30
: no such file or directory lstat /docker/overlay2/193d2ad0d8d087377e3b96cbfb672b0e39132ae5e961872127614c9396f8c068/merged/opt/SPS_18_5_R1/logs/metrics.csv.2018.07.10-11:45
: no such file or directory lstat /docker/overlay2/193d2ad0d8d087377e3b96cbfb672b0e39132ae5e961872127614c9396f8c068/merged/opt/SPS_18_5_R1/logs/metrics.csv.2018.07.10-12:00
: no such file or directory lstat /docker/overlay2/193d2ad0d8d087377e3b96cbfb672b0e39132ae5e961872127614c9396f8c068/merged/opt/SPS_18_5_R1/logs/metrics.csv.2018.07.10-12:15
: no such file or directory lstat /docker/overlay2/193d2ad0d8d087377e3b96cbfb672b0e39132ae5e961872127614c9396f8c068/merged/opt/SPS_18_5_R1/logs/metrics.csv.2018.07.10-12:30
: no such file or directory lstat /docker/overlay2/193d2ad0d8d087377e3b96cbfb672b0e39132ae5e961872127614c9396f8c068/merged/opt/SPS_18_5_R1/logs/metrics.csv.2018.07.10-12:45
In fact your solution can make your aims just need a little change:
for f in $(docker exec -it SPSRS bash -c "ls /opt/tpa/logs/metrics.csv*"); do docker cp SPSRS:$f /root/metrices_testing/; done
->
for f in $(docker exec SPSRS bash -c "ls /opt/tpa/logs/metrics.csv*"); do docker cp SPSRS:`echo $f | sed 's/\r//g'` /root/metrices_testing/; done
This is because docker exec SPSRS bash -c "ls /opt/tpa/logs/metrics.csv*" will have \r in every matched string, so finally the cp can not find the files in container.
So, we use echo $f | sed 's/\r//g' to get rid of \r for every file name, this could make you work.
NOTE: for alpine, we need to use sh to replace bash, meanwhile, -it should be deleted to avoid colorful print in alpine introduce some invisible characters like ^[[0;0m, etc.
Docker cp command supports to copy folder with all the contents inside a folder
docker cp -a container-id:/opt/tpa/logs/ /root/testing/
In the above example copying files from container folder /opt/tpa/logs to local machine /root/testing/ folder. Here all the files inside /logs/ will be copied to local. The trick here is using -a option along with docker cp
Docker cp still doesn't support wildcards. You can however use them in a Dockerfile in the following way:
COPY hom* /mydir/ # adds all files starting with "hom"
COPY hom?.txt /mydir/ # ? is replaced with any single character, e.g., "home.txt"
Reference: https://docs.docker.com/engine/reference/builder/#copy
Run this inside the container:
dcp() {
if [ "$#" -eq 1 ]; then
printf "docker cp %q .\n" "$(hostname):$(readlink -e "$1")"
else
local archive="$(mktemp -t "export-XXXXX.tgz")"
tar czf "$archive" "$#" --checkpoint=.52428800
printf "docker exec %q cat %q | tar xvz -C .\n" "$(hostname)" "$archive"
fi
}
Then select the files you want to copy out:
dcp /foo/metrics.csv*
It'll create an archive inside of the container and spit out a command for you to run. Run that command on the host.
e.g.
docker exec 1c75ed99fa42 cat /tmp/export-x9hg6.tgz | tar xvz -C .
Or, I guess you could do it without the temporary archive:
dcp() {
if [ "$#" -eq 1 ]; then
printf "docker cp %q .\n" "$(hostname):$(readlink -e "$1")"
else
printf "docker exec %q tar czC %q" "$(hostname)" "$PWD"
printf " %q" "$#"
printf " | tar xzvC .\n"
fi
}
Will generate a command for you, like:
docker exec 1c75ed99fa42 tar czC /root .cache .zcompdump .zinit .zshrc .zshrc.d foo\ bar | tar xzvC .
You don't even need the alias then, it's just a convenience.
docker cp accepts either files, or tar archives, so you can pack the list of files provided as arguments to an tar archive, return the archive to stdout and pipe to docker cp.
#!/bin/bash
if [[ "$#" -lt 2 || "$1" == "-h" || "$1" == "--help" ]]; then
printf "Copy files to docker container directory.\n\n"
echo "Usage: $(basename $0) files... container:directory"
exit 0
fi
SOURCE="${*%${!#}}"
TARGET="${#:$#}"
tar cf - $SOURCE | docker cp - $TARGET
I have a ~300Mb zipped local file that I add to a docker image. The next state then extracts the image.
The problem is that the ADD statement results in a commit that results in a new file system layer makes the image ~300Mb larger than it needs to be.
ADD /files/apache-stratos.zip /opt/apache-stratos.zip
RUN unzip -q apache-stratos.zip && \
rm apache-stratos.zip && \
mv apache-stratos-* apache-stratos
Question: Is there a work-around to ADD local files without causing a commit?
One option is to run a simple web server (e.g. python -m SimpleHTTPServer) before starting the docker build, and then using wget to retrieve the file, but that seems a bit messy:
RUN wget http://localhost:8000/apache-stratos.zip && \
unzip -q apache-stratos.zip && \
rm apache-stratos.zip && \
mv apache-stratos-* apache-stratos
Another option is to extract the zipped file at container start up instead of build time, but I would prefer to keep the start up as quick as possible.
According to the documentation, if you pass an archive file from the local filesystem (not a URL) to ADD in the Dockerfile (with a destination path, not a path + filename), it will uncompress the file into the directory given.
If <src> is a local tar archive in a recognized compression format
(identity, gzip, bzip2 or xz) then it is unpacked as a directory.
Resources from remote URLs are not decompressed. When a directory is
copied or unpacked, it has the same behavior as tar -x: the result is
the union of:
1) Whatever existed at the destination path and 2) The contents of the
source tree, with conflicts resolved in favor of "2." on a file-by-file basis.
try:
ADD /files/apache-stratos.zip /opt/
and see if the files are there, without further decompression.
With Docker 17.05+ you can use a multi-stage build to avoid creating extra layers.
FROM ... as stage1
# No need to clean up here, these layers will be discarded
ADD /files/apache-stratos.zip /opt/apache-stratos.zip
RUN unzip -q apache-stratos.zip
&& mv apache-stratos-* apache-stratos
FROM ...
COPY --from=stage1 apache-stratos/ apache-stratos/
You can use docker-squash to squash newly created layers. That should reduce the image size significantly.
Unfortunately the mentioned workarounds (RUN curl ... && unzip ... & rm ..., unpack on container start) are the only options at the moment (docker 1.11).
There are currently 3 options I can think of.
Option 1: you can switch to a tar or compressed tar format from the zip file and then allow ADD to decompress the file for you.
ADD /files/apache-stratos.tgz /opt/
Only downside is any other changes, like a directory rename, will trigger the copy on write again, so you need to make sure your tar file has the contents in the final directory structure.
Option 2: Use a multi-stage build. Extract the file in an early stage, perform any changes, and then copy the resulting directory to your final stage. This is a good option for any build engines that cannot use BuildKit. augurar's answer covers this so I won't repeat the same Dockerfile he already has.
Option 3: BuildKit (available in 18.09 and newer) allows you to mount files from other locations, including your build context, within a RUN command. This currently requires the experimental syntax. The resulting Dockerfile looks like:
# syntax=docker/dockerfile:experimental
FROM ...
...
RUN --mount=type=bind,source=/files/apache-stratos.zip,target=/opt/apache-stratos.zip \
unzip -q apache-stratos.zip && \
rm apache-stratos.zip && \
mv apache-stratos-* apache-stratos
Then to build that, you export a variable before running your build (you could also export it in your .bashrc or equivalent):
DOCKER_BUILDKIT=1 docker build -t your_image .
More details on BuildKit's experimental features are available here: https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md
I have the directory A/a A/b where both a and b are files.
When I run these commands I get the outputs.
-> tar --include A -v -cf A.tar A
a A
a A/a
a A/b
-> tar --include A/a -v -cf A.tar A
I don`t understand in the 2nd evocation why file A/a is not archived. I believe I do not understand how include works.
I am trying to give tar a list and have it create an archive with the contents. Please help.
Thank you.
The following will extract the files in /root/ directory. But it also creates the parent directories under root. What I need is that the files should be exactly under root folder and not in /root/data/mysql/...
# tar -xvf company_raw_2012-02-22.tgz --directory=/root/
data/mysql/company_raw/data_archive_r_20120222.MYD
data/mysql/company_raw/data_archive_r_20120222.MYI
data/mysql/company_raw/data_archive_r_20120222.frm
If that is not possible, how do I write a program to move these files to the required folder?
I have tried the following and it does work.
--strip-components=3
But I do not know how many folder will be there. So the number 3 may change.
Extract everything to the temp directory with full path and then just walk it moving files to the desired destination?
destdir=/root
tmpdir=/root/tmp
rm -rf $tmpdir
mkdir $tmpdir
tar xf archive.tar.gz -C $tmpdir
find -H $tmpdir -type f -exec mv '{}' $destdir \;