I am using IBM Cloud Code Engine. Building the container locally, pushing it to the IBM Container Registry and then creating the Code Engine app from it works. Now, I wanted to build the container image using Code Engine. The code is in a public GitHub repository. I am using the build strategy "Dockerfile" based on this Dockerfile.
When I submit a build using the console, it fails after a while and I see these lines in the output.
#13 1.368 Collecting ibm_db>=3.0.2
#13 1.374 Downloading ibm_db-3.1.2.tar.gz (1.1 MB)
#13 1.381 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 234.4 MB/s eta 0:00:00
#13 1.711 Installing build dependencies: started
#13 5.425 Installing build dependencies: finished with status 'done'
#13 5.430 Getting requirements to build wheel: started
#13 6.751 Getting requirements to build wheel: finished with status 'done'
#13 6.752 ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/tmp/tmpqm5wa7sq/output.json'
How can I investigate this further? Could the errors be from different tools creating the container image? How would I proceed?
There are several options on how to approach the issue:
First and most important: Obtain and check the logs for anything suspicious. The logs are available in the Code Engine browser console for the buildruns, or via command line:
ibmcloud ce buildrun logs --name mybuildrun
In addition, there is extra information on the buildrun by using either
ibmcloud ce buildrun get --name mybuildrun
or
ibmcloud ce buildrun events --name mybuildrun
On a new GitHub branch or better locally, modify the Dockerfile and add extra steps to access files of interest or print additional debugging output. The latter could be done be adding || cat /some/log/file to a failing step.
Make sure that all requirements and packages that are installed are correct and available.
Check the dependencies for known issues.
In the case that lead to my question I needed more debug output to find an issue with a broken software module from my requirements.txt.
Related
I've had a couple Cloud Run services live for several months. However, I attempted to make some updates to a service yesterday, and suddenly the scripts I have been using since the beginning are no longer functioning.
gcloud build submit
I've been using the following command to build my node/npm project via the remote docker container:
gcloud builds submit --tag gcr.io/PROJECT_ID/generator
I have a dockerfile and .dockerignore in the same directory as my package.json from where I run this script. However, yesterday I suddenly started getting an error which read that a dockerfile is required when using the --tag parameter and the image would not build.
Tentative Solution
After some research, I tried moving my build cnfig into a gcloudbuild-staging.json, which looks like this:
{
"steps": [
{
"name": "gcr.io/cloud-builders/docker",
"args": [
"build",
"-t",
"gcr.io/PROJECT_ID/generator",
"."
]
}
]
}
And I've chnaged my build script to:
gcloud builds submit --config=./gcloudbuild-staging.json
After doing this, the container will build - or as far as I can tell. The console output looks like this:
------------------------------------------------- REMOTE BUILD OUTPUT --------------------------------------------------
starting build "8ca1af4c-d337-4349-959f-0000577e4528"
FETCHSOURCE
Fetching storage object: gs://PROJECT_ID/source/1650660913.623365-8a689bcf007749b7befa6e21ab9086dd.tgz#1650660991205773
Copying gs://PROJECT_ID/source/1650660913.623365-8a689bcf007749b7befa6e21ab9086dd.tgz#1650660991205773...
/ [0 files][ 0.0 B/ 22.2 MiB]
/ [1 files][ 22.2 MiB/ 22.2 MiB]
-
Operation completed over 1 objects/22.2 MiB.
BUILD
Already have image (with digest): gcr.io/cloud-builders/docker
Sending build context to Docker daemon 785.4kB
Step 1/6 : FROM node:14-slim
14-slim: Pulling from library/node
8bd3f5a20b90: Pulling fs layer
3a665e454db5: Pulling fs layer
11fcaa1377c4: Pulling fs layer
bf0a7233d366: Pulling fs layer
0d4d73621610: Pulling fs layer
bf0a7233d366: Waiting
0d4d73621610: Waiting
3a665e454db5: Verifying Checksum
3a665e454db5: Download complete
bf0a7233d366: Verifying Checksum
bf0a7233d366: Download complete
8bd3f5a20b90: Verifying Checksum
8bd3f5a20b90: Download complete
0d4d73621610: Verifying Checksum
0d4d73621610: Download complete
11fcaa1377c4: Verifying Checksum
11fcaa1377c4: Download complete
8bd3f5a20b90: Pull complete
3a665e454db5: Pull complete
11fcaa1377c4: Pull complete
bf0a7233d366: Pull complete
0d4d73621610: Pull complete
Digest: sha256:9ea3dfdff723469a060d1fa80577a090e14ed28157334d649518ef7ef8ba5b9b
Status: Downloaded newer image for node:14-slim
---> 913d072dc4d9
Step 2/6 : WORKDIR /usr/src/app
---> Running in 96bc104b9501
Removing intermediate container 96bc104b9501
---> 3b1b05ea0470
Step 3/6 : COPY package*.json ./
---> a6eca4a75ddd
Step 4/6 : RUN npm ci --only=production
---> Running in 7e870db13a9b
> protobufjs#6.11.2 postinstall /usr/src/app/node_modules/protobufjs
> node scripts/postinstall
added 237 packages in 7.889s
Removing intermediate container 7e870db13a9b
---> 6a86cc961a09
Step 5/6 : COPY . ./
---> 9e1f0f7a69a9
Step 6/6 : CMD [ "node", "index.js" ]
---> Running in d1b4d054a974
Removing intermediate container d1b4d054a974
---> 672075ef5897
Successfully built 672075ef5897
Successfully tagged gcr.io/PROJECT_ID/generator:latest
PUSH
DONE
------------------------------------------------------------------------------------------------------------------------
ID CREATE_TIME DURATION SOURCE IMAGES STATUS
8ca1af4c-d337-4349-959f-0000577e4528 2022-04-22T20:56:31+00:00 31S gs://PROJECT_ID/source/1650660913.623365-8a689bcf007749b7befa6e21ab9086dd.tgz - SUCCESS
There are no errors in the online logs.
gcloud run deploy
Here is the code I use to deploy the container:
gcloud run deploy generator --image gcr.io/PROJECT_ID/generator --region=us-central1 --set-env-vars ENVIRONMENT=DEV
The console output for this is:
Deploying container to Cloud Run service [generator] in project [PROJECT_ID] region [us-central1]
✓ Deploying... Done.
✓ Creating Revision...
✓ Routing traffic...
Done.
Service [generator] revision [generator-00082-kax] has been deployed and is serving 100 percent of traffic.
Service URL: https://generator-SERVICE_ID-uc.a.run.app
No errors in the run console, either. It shows the deployment as if everything is fine.
The Problem
Nothing is changing. Locally, running this service with the front-end app which accesses it produces successful results. However, my staging version of the app hosted on Firebase is still acting as if the old version of the code is active.
What I've Tried
I've made sure I'm testing and deploying on the same git branch
I've done multiple builds and deployments in case there was some kind of fluke.
I've tried using the gcloud command to update a service's traffic to its latest revision
I've made sure my client app is using the correct service URL. It doesn't appear to have changed but I copy/pasted it anyway just in case
My last successful deployment was on March 19, 2022. Since them, the only thing I've done is update all my WSL linux apps - which would include gcloud. I couldn't tell what version I ws on before, but I'm now on 38.0.0 of the Google Cloud CLI.
I've tried searching for my issue but nothing relevant is coming up. I'm totally stumped as to why all of this has stopped working and I'm receiving no errors whatsoever. Any suggestions or further info I can provide?
gcloud builds submit should (!?) continue to work with --tag as long as there is a Dockerfile in the folder from which you're running the command or you explicitly specify a source folder.
I'm not disputing that you received an error but it would be helpful to see the command you used and the error that resulted. You shouldn't have needed to switch to a build config file. Although that isn't the problem.
Using latest as a tag value is challenging. The term suggests that the latest version of a container image will be used but this is often not what happens. It is particularly challenging when a service like Cloud Run is running an image tagged latest and a developer asks the service to run -- what the developer knows (!) is a different image -- but also tagged latest.
As far as most services are concerned, same tag means same image and so it's possible (!) either that Cloud Run is not finding a different image or you're not providing it with a different image. I'm unclear which alternative is occurring but I'm confident that your use of latest is causing some of your issues.
So.... for starters, please consider using a system in which every time you create a new container, you tag it with a unique identifier. A common way to do this is to use a commit hash (as these change with every commit). Alternatively you can use the container's digest (instead of a tag) to reference an image version. This requires image references of the form {IMG}#sha256:{HASH}.
Lastly, gcloud run now (has always?) supported deployment from source (folder) to running service (it does the Cloud Build process for you and deploys the result to Cloud Run. It may be worth using this flow to reduce your steps and thereby the possibility of error.
See: Deploying from source code
I'm attempting to learn how to create a Laravel Docker image by following a tutorial on DigitalOcean using WSL. Following the instructions on the Docker Hub page, however, yields an error:
❯ docker run --rm --interactive --tty -v $(pwd):/app composer install
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 94 installs, 0 updates, 0 removals
- Installing voku/portable-ascii (1.4.10): Failed to download voku/portable-ascii from dist: Could not delete /app/vendor/voku/portable-ascii/src/voku/helper:
Now trying to download from source
- Installing voku/portable-ascii (1.4.10):
[RuntimeException]
Could not delete /app/vendor/voku/portable-ascii/src/voku/helper:
install [--prefer-source] [--prefer-dist] [--dry-run] [--dev] [--no-dev] [--no-custom-installers] [--no-autoloader] [--no-scripts] [--no-progress] [--no-suggest] [-v|vv|vvv|--verbose] [-o|--optimize-autoloader] [-a|--classmap-authoritative] [--apcu-autoloader] [--ignore-platform-reqs] [--] [<packages>]...
How can I diagnose what I'm doing wrong?
It turns out that the underlying problem had nothing to do with Docker at all. In fact, Composer was trying to tell me what the problem was all along, but I dismissed it as just a symptom of a deeper issue:
[RuntimeException]
Could not delete /app/vendor/voku/portable-ascii/src/voku/helper:
This message was the crux of it all. I noticed that the directory mentioned, [...]/helper, was empty, so I tried to remove it by hand with rmdir. Instead, I got a No such file or directory error message. I attempted many other was to kill this directory, the entire project directory with rm -rf ~/laravel-app from the home folder, etc. Nothing worked.
Some digging around on the internet suggested that it could be an NTFS corruption if I was running into this issue on Windows. Since I am, indeed, attempting this on WSL (Windows Subsystem for Linux), I gave their suggested fix a try: running chkdsk /F in CMD/PowerShell. A reboot was necessary to complete this task, but after getting everything back up and trying those first few tutorial steps again, I was able to get composer to install the Laravel dependencies without a hitch.
Bottom line: If you run into this sort of issue on WSL, please try running chkdsk /F and reboot. You might just have a similar file system corruption.
We have two possibilities for this error:
1 - You did not execute the command inside the directory :
cd ~/laravel-app
2 - I'm sure there is an internal proxy that is blocking the download of packages.
I am running a tensorflow serving container referring to this , all the previous steps are good, but in the last block I met some problems:
git clone --recurse-submodules https://github.com/tensorflow/serving
cd serving/
bazel build -c opt tensorflow_serving/...
root#15bb1c2766e3:/serving# bazel build -c opt tensorflow_serving/...
ERROR:
/root/.cache/bazel/_bazel_root/f8d1071c69ea316497c31e40fe01608c/external/org_tensorflow/third_party/clang_toolchain/cc_configure_clang.bzl:3:1:
file '#bazel_tools//tools/cpp:cc_configure.bzl' does not contain
symbol 'cc_autoconf_impl'. ERROR: error loading package '': Extension
file 'third_party/clang_toolchain/cc_configure_clang.bzl' has errors.
ERROR: error loading package '': Extension file
'third_party/clang_toolchain/cc_configure_clang.bzl' has errors. INFO:
Elapsed time: 0.107s ERROR: Couldn't start the build. Unable to run
tests. And in my container, the bazel version is 0.9.0:
I just came across this error. First please check whether your Bazel version is 0.5.4 by typing the command bazel version.
If the bazel version is 0.5.4 you need to upgrade it to 0.12.0 . For updating you could change Dockerfile.devel BAZEL_VERSION to 0.12.0 and re run all the steps.
Or you could update bazel directly in the docker container by
Download bazel--installer-linux-x86_64.sh from
https://github.com/bazelbuild/bazel/releases location
chmod +x ./bazel--installer-linux-x86_64.sh
./bazel--installer-linux-x86_64.sh
I have already answered this on github and it worked. Please refer to the links https://github.com/tensorflow/serving/issues/851 and https://github.com/tensorflow/serving/issues/854
I am trying to setup TensorFlow Serving using Docker on a Windows machine by following the instructions from https://tensorflow.github.io/serving/docker.html
I have successfully built and run the container and cloned the TensorFlow servings sub-modules. However, while running the command
"bazel test tensorflow_serving/..." the error
Linking of rule: '//tensorflow_serving/batching/test_util:puppet_batch_scheduler_test' failed:gcc failed: error executing command /usr/bin/gcc .... is displayed
The complete stack trace is shown in the below image.
Any help is appreciated.
I don't know how to solve the problem directly, but a workaround could be pulling a docker image where the TF Serving is already compiled such as this one.
After git clone from dotcloud/docker
cd docker
sudo make VERBOSE=1
Fetching https://net/http/cookiejar?go-get=1
https fetch failed
**
unrecognized import path "net/http/cookiejar"
Can any one please tell me what should i be looking at. I have Go 1.1 installed. The reason i wanted to build docker from src it to remove sys_rawio from lxc.cap.drop. I needed to make CUDA work with lxc containers. I was able to compile cuda code but while running it. I keep getting driver related error that says Operation not permitted.
Installed Go from https://go.googlecode.com/files/go1.1.1.linux-amd64.tar.gz
As suggested by user creack in above comments.
cd docker/docker; go build; ./docker -d& ./docker version
I have docker build successfully now.
Docker can now be used to build docker and is the recommended (and only supported) method, see:
https://docs.docker.com/project/set-up-dev-env/