tldr; When I try to pull an AWS-CLI image from Docker Hub using Bazel, I'm getting odd 404 errors. Pulling other images in the same way works fine.
I'm trying to use Bazel in my monorepo to (among many other things) create several Docker images. One of the Docker images I'm creating uses the verified AWS CLI image as a base.
I'm following along with the rules_docker documentation along with examples provided in that repo.
WORKSPACE File:
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_file")
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "io_bazel_rules_docker",
sha256 = "b1e80761a8a8243d03ebca8845e9cc1ba6c82ce7c5179ce2b295cd36f7e394bf",
urls = ["https://github.com/bazelbuild/rules_docker/releases/download/v0.25.0/rules_docker-v0.25.0.tar.gz"],
)
load(
"#io_bazel_rules_docker//repositories:repositories.bzl",
container_repositories = "repositories",
)
container_repositories()
load("#io_bazel_rules_docker//repositories:deps.bzl", container_deps = "deps")
container_deps()
load(
"#io_bazel_rules_docker//container:container.bzl",
"container_pull",
)
load("#io_bazel_rules_docker//contrib:dockerfile_build.bzl",
"dockerfile_image")
container_pull(
name = "alpine_linux_amd64",
digest = "sha256:954b378c375d852eb3c63ab88978f640b4348b01c1b3456a024a81536dafbbf4",
registry = "index.docker.io",
repository = "library/alpine",
# tag field is ignored since digest is set
tag = "3.8",
)
container_pull(
name = "aws_cli",
digest = "sha256:abb7e318502e78ec99d85bfa0121d5fbc11d8c49bb95f7f12db0b546ebd5ff99",
registry = "index.docker.io",
repository = "library/amazon",
# tag field is ignored since digest is set
tag = "2.9.9",
)
http_file(
name = "sam_archive",
downloaded_file_path = "aws-sam-cli-linux-x86_64.zip",
sha256 = "74264b224f133461e324e7877ed8218fe38ac2320ba498024f0c297de7bb3e95",
urls = [
"https://github.com/aws/aws-sam-cli/releases/download/v1.67.0/aws-sam-cli-linux-x86_64.zip",
],
)
And BUILD file:
load("#io_bazel_rules_docker//container:container.bzl", "container_image", "container_layer")
load("#io_bazel_rules_docker//contrib:test.bzl", "container_test")
load("#io_bazel_rules_docker//docker/util:run.bzl", "container_run_and_commit")
# Includes the aws-cli installation archive
container_image(
name = "aws_cli",
base = "#aws_cli//image"
)
container_image(
name = "basic_alpine",
base = "#alpine_linux_amd64//image",
cmd = ["Hello World!"],
entrypoint = ["echo"],
)
Building basic_alpine works fine:
$ bazel build //:basic_alpine
INFO: Analyzed target //:basic_alpine (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:basic_alpine up-to-date:
bazel-bin/basic_alpine-layer.tar
INFO: Elapsed time: 1.140s, Critical Path: 0.99s
INFO: 50 processes: 16 internal, 34 linux-sandbox.
INFO: Build completed successfully, 50 total actions
Admittedly new to Bazel and maybe I'm not doing this correctly, but building aws_cli fails:
$ bazel build //:aws_cli
INFO: Repository aws_cli instantiated at:
/home/jdibling/repos/stream-ai.io/products/filedrop/monorepo/WORKSPACE:38:15: in <toplevel>
Repository rule container_pull defined at:
/home/jdibling/.cache/bazel/_bazel_jdibling/4ce73e7de2c4ac9889a94fb9b2da25fc/external/io_bazel_rules_docker/container/pull.bzl:294:33: in <toplevel>
ERROR: An error occurred during the fetch of repository 'aws_cli':
Traceback (most recent call last):
File "/home/jdibling/.cache/bazel/_bazel_jdibling/4ce73e7de2c4ac9889a94fb9b2da25fc/external/io_bazel_rules_docker/container/pull.bzl", line 240, column 13, in _impl
fail("Pull command failed: %s (%s)" % (result.stderr, " ".join([str(a) for a in args])))
Error in fail: Pull command failed: 2022/12/23 08:31:25 Running the Image Puller to pull images from a Docker Registry...
2022/12/23 08:31:29 Image pull was unsuccessful: reading image "index.docker.io/library/amazon#sha256:abb7e318502e78ec99d85bfa0121d5fbc11d8c49bb95f7f12db0b546ebd5ff99": GET https://index.docker.io/v2/library/amazon/manifests/sha256:abb7e318502e78ec99d85bfa0121d5fbc11d8c49bb95f7f12db0b546ebd5ff99: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/amazon Type:repository]]
(/home/jdibling/.cache/bazel/_bazel_jdibling/4ce73e7de2c4ac9889a94fb9b2da25fc/external/go_puller_linux_amd64/file/downloaded -directory /home/jdibling/.cache/bazel/_bazel_jdibling/4ce73e7de2c4ac9889a94fb9b2da25fc/external/aws_cli/image -os linux -os-version -os-features -architecture amd64 -variant -features -name index.docker.io/library/amazon#sha256:abb7e318502e78ec99d85bfa0121d5fbc11d8c49bb95f7f12db0b546ebd5ff99)
ERROR: /home/jdibling/repos/stream-ai.io/products/filedrop/monorepo/WORKSPACE:38:15: fetching container_pull rule //external:aws_cli: Traceback (most recent call last):
File "/home/jdibling/.cache/bazel/_bazel_jdibling/4ce73e7de2c4ac9889a94fb9b2da25fc/external/io_bazel_rules_docker/container/pull.bzl", line 240, column 13, in _impl
fail("Pull command failed: %s (%s)" % (result.stderr, " ".join([str(a) for a in args])))
Error in fail: Pull command failed: 2022/12/23 08:31:25 Running the Image Puller to pull images from a Docker Registry...
2022/12/23 08:31:29 Image pull was unsuccessful: reading image "index.docker.io/library/amazon#sha256:abb7e318502e78ec99d85bfa0121d5fbc11d8c49bb95f7f12db0b546ebd5ff99": GET https://index.docker.io/v2/library/amazon/manifests/sha256:abb7e318502e78ec99d85bfa0121d5fbc11d8c49bb95f7f12db0b546ebd5ff99: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/amazon Type:repository]]
(/home/jdibling/.cache/bazel/_bazel_jdibling/4ce73e7de2c4ac9889a94fb9b2da25fc/external/go_puller_linux_amd64/file/downloaded -directory /home/jdibling/.cache/bazel/_bazel_jdibling/4ce73e7de2c4ac9889a94fb9b2da25fc/external/aws_cli/image -os linux -os-version -os-features -architecture amd64 -variant -features -name index.docker.io/library/amazon#sha256:abb7e318502e78ec99d85bfa0121d5fbc11d8c49bb95f7f12db0b546ebd5ff99)
ERROR: /home/jdibling/repos/stream-ai.io/products/filedrop/monorepo/BUILD:6:16: //:aws_cli depends on #aws_cli//image:image in repository #aws_cli which failed to fetch. no such package '#aws_cli//image': Pull command failed: 2022/12/23 08:31:25 Running the Image Puller to pull images from a Docker Registry...
2022/12/23 08:31:29 Image pull was unsuccessful: reading image "index.docker.io/library/amazon#sha256:abb7e318502e78ec99d85bfa0121d5fbc11d8c49bb95f7f12db0b546ebd5ff99": GET https://index.docker.io/v2/library/amazon/manifests/sha256:abb7e318502e78ec99d85bfa0121d5fbc11d8c49bb95f7f12db0b546ebd5ff99: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/amazon Type:repository]]
(/home/jdibling/.cache/bazel/_bazel_jdibling/4ce73e7de2c4ac9889a94fb9b2da25fc/external/go_puller_linux_amd64/file/downloaded -directory /home/jdibling/.cache/bazel/_bazel_jdibling/4ce73e7de2c4ac9889a94fb9b2da25fc/external/aws_cli/image -os linux -os-version -os-features -architecture amd64 -variant -features -name index.docker.io/library/amazon#sha256:abb7e318502e78ec99d85bfa0121d5fbc11d8c49bb95f7f12db0b546ebd5ff99)
ERROR: Analysis of target '//:aws_cli' failed; build aborted: Analysis failed
INFO: Elapsed time: 4.171s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
Just a quick sanity check - should that be library/amazonlinux? AFAICT library/amazon does not exist. However, that one does not have a tag with the sha265 that you specify.
The link you have in the intro is for the amazon/aws-cli image, which does have that tag, so maybe that's the one that you mean to pull?
Related
I have installed k8s 1.24 version and containerd (containerd://1.5.9) is the CR for my setup (ubuntu 20.04).
I have also installed docker on my VM and have added my private repository under /etc/docker/daemon.json with the following changes:
{ "insecure-registries" : ["myPvtRepo.com:5028"] }
When I am running docker pull myPvtRepo:123/image after login to my pvt repo by using docker login myPvtRepo:123 command, I am able to pull the images while running the same command with crictl pull myPvtRepo:123/image, I am facing:
E0819 06:49:01.200489 162610 remote_image.go:218] "PullImage from
image service failed" err="rpc error: code = Unknown desc = failed to
pull and unpack image "myPvtRepo.com:5028/centos:latest": failed to
resolve reference "myPvtRepo.com:5028/centos:latest": failed to do
request: Head https://myPvtRepo.com::5028/v2/centos/manifests/latest:
x509: certificate signed by unknown authority"
image="myPvtRepo.com::5028/centos:latest" FATA[0000] pulling image:
rpc error: code = Unknown desc = failed to pull and unpack image
"myPvtRepo.com::5028/centos:latest": failed to resolve reference
"myPvtRepo.com:5028/centos:latest": failed to do request: Head
https://myPvtRepo.com::5028/v2/centos/manifests/latest: x509:
certificate signed by unknown authority
FYI, I have modified /etc/containerd/config.toml with below content.
version = 2
[plugin."io.containerd.grpc.v1.cri".registry.configs."myPvtRepo.com:5028".tls]
insecure_skip_verify = true
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://myPvtRepo.com:5028", "https://myPvtRepo.com:5038", "https://myPvtRepo.com:5037",
"https://myPvtRepo.com:5039"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."IP:5000"]
endpoint = ["http://IP:5000"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."IP:5000"]
endpoint = ["http://IP:5000"]
I have also modified containerd's endpoint to point to containerd's sock.
Can you please help me out to understand and fix that even after setting insecure_skip_verify = true for my pvt repository and restarting the containerd service why I am getting this issue.
I got a solution:
cd /usr/local/share/ca-certificates/
curl -L --remote-name http://your-artifacts.com/xyz-bundle.crt
/usr/sbin/update-ca-certificates
This one work for me.
Also make sure to update your endpoints under /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: ""
timeout: 0
debug: false
pull-image-on-create: false
disable-pull-on-run: false
You will need to specify the hosts.toml file for the private registry and add skip-verify = true.
ref: https://github.com/containerd/containerd/blob/main/docs/hosts.md
Steps:
create folders: mkdir -p /etc/containerd/certs.d/<your registry>
add these config in /etc/containerd/config.toml:
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"
create and edit hosts.toml under the just created folder
server = "https://<your registry>"
[host."https://<your registry>"]
capabilities = ["pull", "resolve"]
skip_verify = true
I'm trying to connect to my local docker engine running on OSX (m1 chip) in order to create a dynamic inventory.
I've created a host file with the following config
I made sure that docker_containers module is well installed.
plugin: community.docker.docker_containers
docker_host: "unix://Users/ME/.docker/run/docker-cli-api.sock"
Then I run ansible-inventory --graph -i ./hosts/hosts-docker-local.yaml.
But I'm getting the following error:
[WARNING]: * Failed to parse /Users/ME/Projects/ansible-test/hosts/hosts-docker-local.yaml with auto plugin: inventory source '/Users/ME/Projects/ansible-test/hosts/hosts-docker-local.yaml' could not be
verified by inventory plugin 'community.docker.docker_containers'
[WARNING]: * Failed to parse /Users/ME/Projects/ansible-test/hosts/hosts-docker-local.yaml with yaml plugin: Plugin configuration YAML file, not YAML inventory
[WARNING]: * Failed to parse /Users/ME/Projects/ansible-test/hosts/hosts-docker-local.yaml with ini plugin: Invalid host pattern 'plugin:' supplied, ending in ':' is not allowed, this character is reserved to
provide a port.
[WARNING]: Unable to parse /Users/ME/Projects/ansible-test/hosts/hosts-docker-local.yaml as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
#all:
|--#ungrouped:
I tried
ansible-doc -t inventory -l | grep docker
community.docker.docker_containers Ansible dynamic inv...
community.docker.docker_machine Docker Machine inve...
community.docker.docker_swarm Ansible dynamic inv...
but somehow if I do this
ansible localhost -i ./hosts/hosts-docker-local.yaml -m community.docker.docker_containers
It complains
localhost | FAILED! => {
"msg": "The module community.docker.docker_containers was not found in configured module paths"
}
maybe something wrong with my module path, something wierd with OSX? (I installed Ansible with brew)
The inventory file must end in docker.yaml, as pointed out by #Zeitounator.
Uses a YAML configuration file that ends with docker.[yml|yaml].
https://docs.ansible.com/ansible/latest/collections/community/docker/docker_containers_inventory.html#synopsis
I know in Dockerfile I can extend existing docker image using:
FROM python/python
RUN pip install request
But how to extend it in bazel?
I am not sure if I should use container_import, but with that I am getting the following error:
container_import(
name = "postgres",
base_image_registry = "some.artifactory.com",
base_image_repository = "/existing-image:v1.5.0",
layers = [
"//docker/new_layer",
],
)
root#ba5cc0a3f0b7:/tcx# bazel build pkg:postgres-instance --verbose_failures --sandbox_debug
ERROR: /tcx/docker/postgres-operator/BUILD.bazel:12:17: in container_import rule //docker/postgres-operator:postgres:
Traceback (most recent call last):
File "/root/.cache/bazel/_bazel_root/2f47bbce04529f9da11bfed0fc51707c/external/io_bazel_rules_docker/container/import.bzl", line 98, column 35, in _container_import_impl
"config": ctx.files.config[0],
Error: index out of range (index is 0, but sequence has 0 elements)
ERROR: Analysis of target '//pkg:postgres-instance' failed; build aborted: Analysis of target '//docker/postgres-operator:postgres' failed
INFO: Elapsed time: 0.209s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (1 packages loaded, 2 targets configured)
container_import is the correct rule to import an existing image. However, all it does is import, it doesn't pull it from anywhere. I think you're looking for container_pull instead, which will pull an image from a repository and then automatically use container_import to translate it for other rules_docker rules.
To add a new layer, use container_image, with base set to the imported image and tars set to the additional files you want to add. Or, if you want to add things in other formats, see the docs for alternates to tars (like debs or files).
Putting it all together, something like this in your WORKSPACE:
container_pull(
name = "postgres",
registry = "some.artifactory.com",
repository = "existing-image",
tag = "v1.5.0",
)
and then this in a BUILD file:
container_image(
name = "postgres_plus",
base = "#postgres//image",
tars = ["//docker/new_layer"],
)
The specific problem you're running into is that container_pull.layers isn't for adding new layers, it's for specifying the layers of the image you're importing. You could import those some other way (http_archive, check in the tar files, etc) and then specify them all by hand instead of using container_pull if you're doing something unusual.
I needed a custom centos image with docker installed. So I built it using centos image and tagged it custom (shown below).
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
centos custom 84766562f881 4 hours ago 664MB
centos/systemd latest 05d3c1e2d0c1 7 weeks ago 202MB
I am trying to deploy couple of containers using Terraform on my local machine, each with a unique name that comes from another file. The docker images are on local machine. Here's the TF code.
$ cat main.tf
provider "docker" {
}
resource "docker_image" "centos" {
name = "centos:custom"
}
resource "docker_container" "app_swarm" {
image = "${docker_image.centos.custom}"
count = "${length(var.docker_cont)}"
name = "${element(var.docker_cont, count.index)}"
}
When I run terraform apply, I get this error which I am not sure how to fix. Can someone point me in the right direction please?
Error: Error running plan: 1 error(s) occurred:
* docker_container.app_swarm: 3 error(s) occurred:
* docker_container.app_swarm[0]: Resource 'docker_image.centos' does not have attribute 'custom' for variable 'docker_image.centos.custom'
* docker_container.app_swarm[1]: Resource 'docker_image.centos' does not have attribute 'custom' for variable 'docker_image.centos.custom'
* docker_container.app_swarm[2]: Resource 'docker_image.centos' does not have attribute 'custom' for variable 'docker_image.centos.custom'
Yes, the other file exists with names, its a simple list.
EDIT:
Thanks David, tried your suggestion and amended the code to look like-
provider "docker" {
}
resource "docker_image" "centos" {
name = "centos:custom"
}
resource "docker_container" "app_swarm" {
image = "${docker_image.centos.latest}"
count = "${length(var.docker_cont)}"
name = "${element(var.docker_cont, count.index)}"
}
But now I get this error.
Error: Error applying plan:
1 error(s) occurred:
* docker_image.centos: 1 error(s) occurred:
* docker_image.centos: Unable to read Docker image into resource: Unable to pull image centos:custom: error pulling image centos:custom: Error response from daemon: manifest for centos:custom not found
I guess I will have to setup a local Docker repository to get this working but I am not sure?
You can only use the specific fields listed in the docker_image resource documentation in the ${docker_image.centos...} interpolation. In particular, even though you don't use the tag :latest, you need a .latest property reference:
image = "${docker_image.centos.latest}"
(If the image actually is one you've built locally, you may also want to specify the keep_locally option on your docker_image resource so that terraform destroy won't delete it.)
I am trying to start up network using following command
./network_setup.sh up
After running this command I am receiving this errro
#
# Generating anchor peer update for Org2MSP
########################################################### 2017-06-05 18:16:35.716 CST [common/configtx/tool] main -> INFO 001
Loading configuration 2017-06-05 18:16:35.719 CST
[common/configtx/tool] doOutputAnchorPeersUpdate -> INFO 002
Generating anchor peer update 2017-06-05 18:16:35.719 CST
[common/configtx/tool] doOutputAnchorPeersUpdate -> INFO 003 Writing
anchor peer update
Pulling cli (hyperledger/fabric-tools:latest)...
ERROR: repository hyperledger/fabric-tools not found: does not exist or no pull access
ERROR !!!! Unable to pull the images
How I can remove this error?please help me
You can manually pull this (and any other Hyperledger Fabric image) image from DockerHub. There was a period when the fabric-tools image was not included in the helper script download-dockerimages.sh.
docker pull hyperledger/fabric-tools:x86_64-1.0.0-beta
docker tag hyperledger/fabric-tools:x86_64-1.0.0-beta hyperledger/fabric-tools
Note that it might be worth reviewing the set of published tags on DockerHub to be sure you are getting the latest.
https://hub.docker.com/u/hyperledger/
you write the command line : this :
docker pull hyperledger/fabric-tools:x86_64-1.1.0-rc1
after that
docker tag hyperledger/fabric-tools:x86_64-1.1.0-rc1 hyperledger/fabric-tools:latest