tf.keras.utils.get_file(<directory_name>, <tgz path>, untar = True)
How do I prevent the untar from the above get_file() call from creating duplicate "._*" files?
I JUST figured it out for me, maybe it's the same for you. It wasn't the functions fault, but instead the files you are working with do in fact have files with "." as a prefix. For me, it was because I have a mac and in the image folders I zipped there are hidden mac files that have . as a prefix.
I just rezipped my files using this command:
tar --disable-copyfile -czf weeds_image.tgz weed_images
Related
I recently moved my code repo from a One drive folder to my C: drive, trying to escape sync issues wiping files (thats another issue). Before doing so both my dev and prod compose files worked as expected (under source control).
Moving the files across (cloning from the repo and checking out to the branch) I ran my dev compose file which worked again as expected however my prod started complaining about "no such file or directory", "not found". I decided to snapshot the filesystem and check and could see the files and could cat the contents. So why is docker complaining?
My compose file does not use any hardcoded file paths, all relative and its evident the files are being copied across.
Compose errors:
exec docker/deployment/folder/scripts/run.sh: no such file or directory
Snapshot:
/app$ cat docker/deployment/folder/scripts/run.sh
#!/bin/sh
As previously mentioned this worked running from a diffirent local dir location. I can't imaging how that would make any diffirence.
Thanks in advance for any support.
Found the issue, vs code had updated the end-of-line sequence to crlf instead of lf.
Simplified problem: I have a tar.gz file: Example.tar.gz
When extracted, it produces 1 folder named 2014, and 3 files within that folder named as follows:
a1:00:test
a2:00:test
a3:00:test
Objective: When extraction is complete, I want the folder to still be named 2014, but have the files within the folder named as follows:
a1-00-test
a2-00-test
a3-00-test
These files are on an ext4 file system, but the extracted contents must be on a ntfs file system which is not possible due to the illegal file names (:).
These tar.gz files are very large (30 GB), and I have over 70 of them. It would not be possible to extract all the files on the ext4 file system and rename them due to limited storage.
tar -xzvf /media/joshua/output/Example.tar.gz -C /media/joshua/elements/
Above code works normally, but not when extracting to the ntfs file system (due to illegal file name).
tar --transform='s/:/-/'-xzvf /media/joshua/output/Example.tar.gz -C /media/joshua/elements/
Above code suggested by #Barmar changes the first : to a - but not the second :
I have tried suggestions here: how to rename files you put into a tar archive using linux 'tar' but I am new to Linux and having some trouble understanding the syntax, and applying it to my situation. Any help or general suggestions is appreciated!
Use the --transform option as described in the question you linked to:
tar --transform='s/:/-/g' -xzvf /media/joshua/output/Example.tar.gz -C /media/joshua/elements/
s/:/-/ replaces : with - in the filename.
Starting to use Docker here.
Right now im facing an issue with my project, where I need to copy multiple files from multiple directories to the docker image on start up.
Here is my current code
FROM heroiclabs/nakama:3.11.0
COPY src/*.lua /nakama/data/modules
COPY src/database/*.lua /nakama/data/modules
COPY src/managers/*.lua /nakama/data/modules
COPY src/modules/*.lua /nakama/data/modules
COPY local.yml /nakama/data
What is happening so far is that docker copies only the main.lua file from the starting directory, it either dont copy the remaining ones or copy them with the current data structure.
How can I actually copy it in order to get the lua files from database, manages and modules into the same root directory as main.lua?
To add to this, I get this error on the Nakama console that indicates that a file searched by main.lua module its not found.
{"level":"fatal","ts":"2022-04-28T21:19:51.163Z","caller":"main.go:154","msg":"Failed initializing runtime modules","error":"/nakama/data/modules/src/main.lua:2: module economyManager not found:\n\tno field package.preload['economyManager']\n\tno cached module 'economyManager', \nstack traceback:\n\t[G]: in function 'require'\n\t/nakama/data/modules/src/main.lua:2: in main chunk\n\t[G]: ?"}
So far so good, the / at the end of each COPY line did not do the trick.
Edit:
This is the full directory structure:
src
-database
--luascripts.lua
-managers
--luascripts.lua
-modules
--luascripts.lua
-main.lua
intellisense
-nakama.lua
local.yml
dorckerfile
docker-compose.yml
This helps better illustrate the error. As you can se on the log, docker its copying src directory over to nakama/data/modules, what I aim to do is to copy ONLY the content from src, but not the src directory.
Same can be said for other directories to a lesser degree, my aim is to not carry over directory structure to the destination path
Your command seems to be right, just add a trailing / to the destination docker image's path.
For example,
COPY *.lua /nakama/data/modules/
The error was on the docker-compose file, I was double mounting a volume and that was causing errors on the whole COPY operation.
Using java -jar .\swagger-codegen-cli.jar generate -l typescript-angular -i .\swagger.json, I can turn ".\swagger.json" into a typescript-angular file. However, the .json file must be in the same folder as the swagger-codegen-cli, which is impractical when you have many files to convert (You have to manually download and move every file). I've scoured the web, but haven't been able to find a practical solution. Is there a way to do this? Am I able to integrate swagger-codegen in such a way that I can "mass produce" files? I know I can change the output directory (-o) but that isn't what I'm looking for. Say I have a folder called "scripts" where I've got a lot of .json files. Am I able to call swagger-codegen-cli generate on any file inside of that folder, even if swagger-codegen-cli is located in a different folder?
The -i argument expects a fully-qualified or relative path to an OpenAPI file, or an URL where that file is located. That is, the file does not have to be in the same folder as the Swagger Codegen CLI.
Examples:
# The file is in the current working folder
-i swagger.json
-i ./swagger.json
# The file is in a subfolder of the current working folder
-i sub/folder/swagger.json
-i ./sub/folder/swagger.json
# Relative path to a file in another folder
-i ../../path/to/swagger.json
# Absolute path - Windows
-i C:/path/to/swagger.json
# Absolute path - *nix
-i ~/path/to/swagger.json
# URL
-i https://petstore.swagger.io/v2/swagger.json
in my dockerfile I have these two lines:
ADD /ansible/inventory /etc/ansible/hosts
ADD /ansible/. /ansiblerepo
The first line works, as I can run the container and see my hosts file has been populated with all the ips from my inventory file.
The second line doesn't appear to be working though. I'm just trying to copy all the files/subdirectories of ansible and copy them over to the ansiblerepo directory inside the new container.
There are no errors while building the image, but again ansiblerepo is just an empty directory and nothing has copied over to it. I assume I'm just missing a back slash or something.
Docker ADD and COPY commands work relative to the build directly, and only for files in that directory that weren't excluded with a .dockerignore file. The reason for this is that builds actually run on the docker host, which may be a remote machine. The first step of a docker build . is to package up all the files in the directory (in this case .) and send them to the host to run your build. Any absolute paths you provide are interpreted as relative to the build directory and anything you reference that wasn't sent to the server will be interpreted as a missing file.
The solution is to copy /ansible to your build directory (typically the same folder as your Dockerfile).
Make sure that in your ".dockerignore" file, it does not excluded everything. usually, dockerignore file has these lines
*
!obj\Docker\publish\*
!obj\Docker\empty\
this means that everything is ignored except publish and empty folders.
Removing trailing /. from source directory should fix the ADD command.
On a related note, Docker Best Practices suggest using COPY over ADD if you don't need the URL download feature.