I try to deploy example Kedro starter project (pandas-iris).
I successfuly run it locally (kedro run), and then, having kedro-docker install, init a Docker, build image and push it to my registry.
Unfortunately, both kedro docker run and docker run myDockerID/iris_image generate the same error:
DataSetError: Failed while loading data from data set
CSVDataSet(filepath=/home/kedro/data/01_raw/iris.csv, load_args={},
protocol=file, save_args={'index': False}).
[Errno 2] No such file or directory: '/home/kedro/data/01_raw/iris.csv'
It looks like the data catalog wasn't copied to the image/container.
I would appreciate Your help,
Many thanks :)
Andy
If data catalog wasn't copy then you probably won't get the path? Is the data actually lives there?
Problem solved: I had to comment #data entry in .dockerignore file. Original kedro-docker keeps data folder ignored.
#mediumnok: thank you for the comment, no problem with path :)
Related
I received a .tar docker file from a friend that told me that it should contain all dependences for a program that I've been struggling to get working and that all I need to do is "run" the Docker file. The Docker file is of a .tar format and is around 3.1 GB. The program this file was setup to run is call opensimrt. The GitHub link to the file is as follows:
https://github.com/mitkof6/OpenSimRT
The google drive link to the Docker file is as follows:
https://drive.google.com/file/d/1M-5RnnBKGzaoSB4MCktzsceU4tWCCr3j/view?usp=sharing
This program has many dependencies, some big ones to note is that it runs off ubuntu 18.04 and Opensim 4.1.
I'm not a computer scientist by any means, so I've been struggling to even learn to do docker basics like load and run a image. However, I desperately need this program to work. If you have any steps or advice on how to run this .tar I'd greatly appreciate it. Alternatively if you are able to find a way to get opensimrt up and running and can post those steps I'd be more than happy with that solution as well.
I've tried the commands "docker run" and "docker load" followed by their respective tags, file paths, args..etc. However, even when I fix various issues I always get stuck with a missing var/lib/docker/tmp/docker-import-....(random numbers) file. The numbers change every so often when trying to solve the issue, but eventually I always end up getting some variation of this error: Error response from daemon: open /var/lib/docker/tmp/docker-import-3640220538/bin/json: no such file or directory.
ps: I have extracted the .tar already and there is no install guide/instruction, .exe, install application. As a result I'm not sure how to get the program installed and running.
I have a question about the .dockerignore workflow which I wasn't really able to understand while browsing through the documentation and different internet topics.
Have the following folder structure:
home
|
|- folder_1
|- folder_2
Inside my dockerfile I want to copy the contents of home directory, so I use
COPY ./ /home
Inside .dockerignore I have:
*
!folder_1
!folder_3
I am referring to a non-existent folder - folder_3, which is supposed to be copied, right?
I ran it and it looks like there's no problem with that, thus .dockerignore somehow manages this situation.
If I tried to do the same thing without using .dockerignore, targeting a non-existent directory I would get an error.
If anybody can please clear this workflow, or if a duplicate, please attach some information so I can educate myself.
Thanks in advance!
First of all, .dockerignore works like .gitignore. Inside these files you set the rules on the basis of which files should be added, and which should not.
In your scenario you COPY the whole home directory which consists of folder_1 and folder_2. Your .dockerignore file sets the following rules:
* # ignore all files/directories
!folder_1 # do not ignore folder_1
!folder_3 # do not ignore folder_3
Regardless of whether there is a folder_1 or folder_3 in your local home directory or not, it won't show you any errors, because it just tries to find particular files/directories that are inside .dockerignore. If it finds this file/directory, it applies the rules. If it doesn't find this file/directory, it doesn't do anything with it.
Hope that's a little bit more clear now.
You'll occasionally see reference to a Docker build context. The build has two steps:
The docker build client application creates a tar file of its directory parameter, and sends it in an HTTP request to the Docker daemon.
The Docker daemon unpacks the tar file, finds the Dockerfile in it, and runs it using the file content it was given.
.dockerignore only affects the first step: it keeps docker build from sending the Docker daemon particular files. The .dockerignore file doesn't require there to be a folder_3 directory, it just says that if there is one it shouldn't be excluded. The second step on the Docker daemon side doesn't use .dockerignore at all, and when you COPY . /somewhere it copies the entire build context; that is, whatever was sent in the API request.
There are a couple of practical consequences of this workflow. If you have a very large local directory it can take time to send it to the Docker daemon, and the Docker daemon keeps a duplicate copy of it during the build, so it's often worthwhile to .dockerignore your .git directory and a build tree. This setup is also how docker build works with a Docker daemon on a different system or in a VM, and it's why if you try to COPY a file by name that doesn't exist (COPY folder_3 somewhere) you get an error message referencing a Docker-internal path.
I have just created a .env file to separate my environment variables from my main docker-compose file. I can run this document on my local machine fine with no errors or issues but when I try run it through my CD pipeline I get the following error.
[error]Top level object in 'C:\BuildAgent_work\r38\a\"Myproject Name"\drop\ .env' needs to be an object not 'class 'str'.
I first thought this was because I had set up my build/CI process wrong but I have played around with it and have had no luck.
I have also done some research online to find others with the same problem but none relate to DevOps in anyway so it has been unhelpful
I am not sure how to reproduce this problem but if anyone knows I can try provide some of my code if needed
Edit:
Here is a snippet of my .env file. Check comment below for my thoughts
ContainerInfrastructure_Version=6.7.93-beta.1
ContainerInfrastructureCore_Version=6.7.41-beta.1
AuthenticationWebService_Version=6.7.52-beta.1
CRM_Version=6.7.52-beta.1
Expected result:
Deploys successfully
What I'm getting during the docker-compose task:
[error]Top level object in 'C:\BuildAgent_work\r38\a\Goldpine.ReleaseManagement\drop.env' needs to be an object not 'class 'str'.
Ok so I figured it out. I'm not sure how to explain this briefly but I'll do my best.
So the problem was within DevOps itself not my code. It turns out a .env file only works/gets picked up if you run the docker-compose command from within the working directory that the docker-compose.yml file exists.
When it went into DevOps it was not running the command from within the downloaded artefact directory but by creating a path to it using the -f command.
So long story short, If you use a .env file you need to set the working directory within the CD pipeline to your artefact folder for it to be able to see the .env file correctlly.
I hope this is clear enough if not just let me know and I'll try change it accordingly :)
I am new in the Docker-World and i struggle very hard to create a Docker Image in Azure DevOps.
So First i created a Angular ASP.NET Core 2.2 Webapplication on VS 2019.
Then i committed this in Azure DevOps & configure the Pipline for manuell or automatic build process which create a docker image.
After then i create a Docker file in VS2019 -> Project -> ADD Docker Support -> Linux.
Conclusion when i start my pipeline so I get this error
COPY failed: stat
/var/lib/docker/tmp/docker-builder088029891/WebapplicationTestLinuxDocker/WebapplicationTestLinuxDocker.csproj:
no such file or directory /usr/bin/docker failed with return code: 1
Projectstructure:
My Docker File looks like:
And my .dockerignore have this:
I thank you in advance and I hope you can help me! :)
the source paths are relative to the Dockerfile. So in your Dockerfile it says:
["WebapplicationTestLinuxDocker/WebapplicationTestLinuxDocker.csproj","WebapplicationTestLinuxDocker/"]
change it to:
["WebapplicationTestLinuxDocker.csproj","WebapplicationTestLinuxDocker/"]
then go down two lines and where it says:
COPY . .
change it to:
COPY . WebapplicationTestLinuxDocker/
Hope that works!
Tim
Tray this:
copy ./path/path path
btw I did not see your file in the screenshot :)
Keep the docker file on the solution folder level instead of inside the project.
I've personal ASP.NET Core project which scrapes data from the web using Selenium and Chromium and saves it in local sqlite database.
I want to be able to run this app in Docker image on my Synology NAS. Managed to create and run Docker image (on my Mac), it displays data from sqlite db correctly, but getting error when trying to scrape:
The chromedriver file does not exist in the current directory or in a directory on the PATH environment variable.
From my very limited understanding of Dockers in general, I understand that I need to add chromiumdriver inside the docker somehow.
I've searched a lot, went trough ~30 different examples and still can't get this to work.
Any help is appreciated!
You need to build a new image based on the existing one, in which you add the chromedriver binary. In other words you need to extend your current image.
So create a directory containing a Dockerfile and the chromedriver binary.
Your Dockerfile should look like this:
FROM your_existing_image_name:version
COPY chromedriver desired_path_inside_container
Then open a terminal inside this directory and execute:
docker build -t your_existing_image_name:version++ .
After that you should be able to start a container from the newly created image.
Some notes:
I have assumed that your existing image has been tagged with a version. If it is not the case then remove :version from Dockerfile
Similarly, remove :version++ from the build command. However, is a good practice to include versioning in your images.
I have not add any entrypoint assuming that you do not need to change the existing one.