Using google cloud registry with docker while offline - docker

I'm using Google Cloud Registry, associating with Docker using gcloud auth configure-docker.
https://cloud.google.com/sdk/gcloud/reference/auth/configure-docker
However, when my computer is offline and I run docker-compose up I get an error where it tries to communicate/authenticate with Google.
how can I use docker offline now that I've started using GCR?
$ docker-compose up --build --force-recreate -d
Building solr
ERROR: (gcloud.auth.docker-helper) There was a problem refreshing your current auth tokens: Unable to find the server at www.googleapis.com
Please run:
$ gcloud auth login
to obtain new credentials, or if you have already logged in with a
different account:
$ gcloud config set account ACCOUNT
to select an already authenticated account to use.
Traceback (most recent call last):
File "site-packages/dockerpycreds/store.py", line 74, in _execute
File "subprocess.py", line 336, in check_output
File "subprocess.py", line 418, in run
subprocess.CalledProcessError: Command '['/Users/me/google-cloud-sdk/bin/docker-credential-gcloud', 'get']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "site-packages/docker/auth.py", line 129, in _resolve_authconfig_credstore
File "site-packages/dockerpycreds/store.py", line 35, in get
File "site-packages/dockerpycreds/store.py", line 87, in _execute
dockerpycreds.errors.StoreError: Credentials store docker-credential-gcloud exited with "".
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker-compose", line 6, in <module>
File "compose/cli/main.py", line 71, in main
File "compose/cli/main.py", line 127, in perform_command
File "compose/cli/main.py", line 1080, in up
File "compose/cli/main.py", line 1076, in up
File "compose/project.py", line 475, in up
File "compose/service.py", line 342, in ensure_image_exists
File "compose/service.py", line 1082, in build
File "site-packages/docker/api/build.py", line 251, in build
File "site-packages/docker/api/build.py", line 307, in _set_auth_headers
File "site-packages/docker/auth.py", line 96, in resolve_authconfig
File "site-packages/docker/auth.py", line 146, in _resolve_authconfig_credstore
docker.errors.DockerException: Credentials store error: StoreError('Credentials store docker-credential-gcloud exited with "".',)
[26419] Failed to execute script docker-compose

Docker-compose will read your YAML file to configure your application’s services , so if in your YAML you are using a Docker image or a personalized images and you don't have them on local, docker will try download them from the principal registry in your configuration, in this case Google Cloud Registry and if your are offline you will get an error.

Related

Configure gsutil s3 in Google Cloud SDK Docker

I have Google Cloud SDK docker configured and running on my windows machine after following this.
https://hub.docker.com/r/google/cloud-sdk/
I'm trying to run this command to list a s3 bucket
docker run --rm -ti --volumes-from gcloud-config google/cloud-sdk gsutil ls s3://bucketname
Authentication fails due to not setting the AWS keys. I presume from .boto file not having aws_access_key_id and aws_secret_access_key set. I can't seem to figure out how to set those variables.
I tried to run this to generate a .boto file but the bucket was shared with me and I don't have access keys.
docker run --rm -ti --volumes-from gcloud-config google/cloud-sdk gsutil config -a
Am I missing something or is there any other way to set these AWS credentials? Maybe with gcloud config set?
Here is the error log
ERROR 1202 03:16:07.326810 utils.py] Caught exception reading instance data
Traceback (most recent call last):
File "/usr/lib/python3.7/urllib/request.py", line 1324, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/usr/lib/python3.7/http/client.py", line 1260, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1306, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1255, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1030, in _send_output
self.send(msg)
File "/usr/lib/python3.7/http/client.py", line 970, in send
self.connect()
File "/usr/lib/python3.7/http/client.py", line 942, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/lib/python3.7/socket.py", line 727, in create_connection
raise err
File "/usr/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/utils.py", line 220, in retry_url
r = opener.open(req, timeout=timeout)
File "/usr/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/usr/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/usr/lib/python3.7/urllib/request.py", line 1352, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/usr/lib/python3.7/urllib/request.py", line 1326, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
ERROR 1202 03:16:07.328018 utils.py] Unable to read instance data, giving up
Traceback (most recent call last):
File "/usr/lib/google-cloud-sdk/platform/gsutil/gsutil", line 21, in <module>
gsutil.RunMain()
File "/usr/lib/google-cloud-sdk/platform/gsutil/gsutil.py", line 122, in RunMain
sys.exit(gslib.__main__.main())
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 444, in main
user_project=user_project)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 783, in _RunNamedCommandAndHandleExceptions
_HandleUnknownFailure(e)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 640, in _RunNamedCommandAndHandleExceptions
user_project=user_project)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 412, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/ls.py", line 683, in RunCommand
listing_helper.ExpandUrlAndPrint(storage_url))
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/utils/ls_helper.py", line 372, in ExpandUrlAndPrint
print_initial_newline=False)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/utils/ls_helper.py", line 449, in _RecurseExpandUrlAndPrint
bucket_listing_fields=self.bucket_listing_fields):
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 540, in IterAll
expand_top_level_buckets=expand_top_level_buckets):
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 174, in __iter__
fields=bucket_listing_fields):
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/boto_translation.py", line 447, in ListObjects
headers=headers)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/storage_uri.py", line 166, in list_bucket
bucket = self.get_bucket(headers=headers)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/storage_uri.py", line 181, in get_bucket
conn = self.connect()
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/storage_uri.py", line 117, in connect
**connection_args)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/s3/connection.py", line 205, in __init__
validate_certs=validate_certs, profile_name=profile_name)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/connection.py", line 573, in __init__
host, config, self.provider, self._required_auth_capability())
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/auth.py", line 1032, in get_auth_handler
'Check your credentials' % (len(names), str(names)))
boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['S3HmacAuthV4Handler'] Check your credentials
I edited the .boto file in the legacy config and got this to work.
docker restart gcloud-config
docker exec -u 0 -it <container-id-here>/bin/bash
apt-get install nano
nano root/.config/gcloud/legacy_credentials/***/.boto
add under [Credentials]
aws_access_key_id = ***
aws_secret_access_key = ***
enjoy
docker run --rm -ti --volumes-from gcloud-config google/cloud-sdk gsutil ls s3://bucketname

Docker: error using docker-compose up on the official getting started tutorial

I'm new to docker and I'm trying to follow this simple "getting started" tutorial https://docs.docker.com/compose/gettingstarted/ using a newly first time installation of docker (for Windows 10) dowloaded from here: https://hub.docker.com/editions/community/docker-ce-desktop-windows/.
At step 4 of this tutorial i get this error:
PS D:\composetest> docker-compose up
Building web
Traceback (most recent call last):
File "site-packages\docker\credentials\store.py", line 80, in _execute
File "subprocess.py", line 395, in check_output
File "subprocess.py", line 487, in run
subprocess.CalledProcessError: Command '['C:\\Program Files\\Docker\\Docker\\resources\\bin\\docker-credential-desktop.EXE', 'list']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker-compose", line 6, in <module>
File "compose\cli\main.py", line 72, in main
File "compose\cli\main.py", line 128, in perform_command
File "compose\cli\main.py", line 1078, in up
File "compose\cli\main.py", line 1074, in up
File "compose\project.py", line 548, in up
File "compose\service.py", line 367, in ensure_image_exists
File "compose\service.py", line 1106, in build
File "site-packages\docker\api\build.py", line 261, in build
File "site-packages\docker\api\build.py", line 308, in _set_auth_headers
File "site-packages\docker\auth.py", line 302, in get_all_credentials
File "site-packages\docker\credentials\store.py", line 71, in list
File "site-packages\docker\credentials\store.py", line 93, in _execute
docker.credentials.errors.StoreError: Credentials store docker-credential-desktop exited with "error listing credentials - err: exit status 1, out: `Impossibile trovare elemento.`".
[13284] Failed to execute script docker-compose
EDIT: The accepted solution in docker-compose unable to start, unfortunately, did not work.
What is going wrong?
I have the same issue...
Try:
$nano ~/.docker/config.json
In this file change credsStore to credStore
Now run your docker-compose. If its not works try sudo
try to add to the environment path :
C:\Program Files\Docker\Docker\resources\bin
If you are using WSL2 run this command
sudo ln -s /mnt/c/Program\ Files/Docker/Docker/resources/bin/docker-credential-desktop.exe /usr/bin/docker-credential-desktop.exe

RuntimeError: Can not reuse socket after connection was closed using docker

I am following this tutorial to run Ethereum crawler using Docker on window 10 using docker
after executing
$ MYSQL_DATA_PATH="$HOME/indexer-data/mysql" GETH_DATA_PATH="$HOME/indexer-data/geth"docker-compose up
in this line i got error:
409 Client Error: Conflict for url: http+docker://localnpipe/v1.25/containers/ee2b46142bae704d4963853e22a77ba896a8f841120ecf5ac97befca91847672/attach?logs=0&stdout=1&stderr=1&stream=1
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "threading.py", line 916, in
_bootstrap_inner File "threading.py", line 864, in run File "compose\cli\log_printer.py", line 233, in watch_events File
"compose\container.py", line 215, in attach_log_stream File
"compose\container.py", line 307, in attach File
"site-packages\docker\utils\decorators.py", line 19, in wrapped File
"site-packages\docker\api\container.py", line 61, in attach File
"site-packages\docker\api\client.py", line 400, in _read_from_socket
File "site-packages\docker\api\client.py", line 311, in
_get_raw_response_socket File "site-packages\docker\api\client.py", line 263, in _raise_for_status File
"site-packages\docker\errors.py", line 19, in
create_api_error_from_http_exception File
"site-packages\requests\models.py", line 880, in json File
"site-packages\requests\models.py", line 828, in content File
"site-packages\requests\models.py", line 750, in generate File
"site-packages\urllib3\response.py", line 496, in stream File
"site-packages\urllib3\response.py", line 444, in read File
"http\client.py", line 449, in read File "http\client.py", line 493,
in readinto File "site-packages\docker\transport\npipesocket.py",
line 209, in readinto File
"site-packages\docker\transport\npipesocket.py", line 20, in wrapped
RuntimeError: Can not reuse socket after connection was closed.
git config --global user.name "usrname"
git config --global user.password "paswd"
git clone https://github.com/usrame/eth-indexer.git
cd eth-indexer
touch .env
echo "MYSQL_DATA_PATH=~/indexer-data/mysql" > .env
echo "GETH_DATA_PATH=~/indexer-data/geth" > .env
git add -f .env
docker-compose build
mkdir -p ~/indexer-data/mysql ~/indexer-data/geth
# Create database sechema
MYSQL_DATA_PATH="$HOME/indexer-data/mysql" docker-compose up idx-database idx-migration
MYSQL_DATA_PATH="$HOME/indexer-data/mysql" GETH_DATA_PATH="$HOME/indexer-data/geth" docker-compose up

Kiwi Backup restore fails with python errors

I export database json doing commands explained here:
http://kiwitcms.org/blog/atodorov/2018/07/30/how-to-backup-docker-volumes-for-kiwi-tcms/
I'm running latest version of Kiwi.
docker exec -it kiwi_web /bin/bash -c '/Kiwi/manage.py sqlflush | /Kiwi/manage.py dbshell'
2.cat database.json | docker exec -i kiwi_web /Kiwi/manage.py loaddata --format json -
and I get this error:
Traceback (most recent call last):
File "/venv/lib/python3.6/site-packages/django/db/models/options.py", line 564, in get_field
return self.fields_map[field_name]
KeyError: 'description'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/venv/lib/python3.6/site-packages/django/core/serializers/json.py", line 69, in Deserializer
yield from PythonDeserializer(objects, **options)
File "/venv/lib/python3.6/site-packages/django/core/serializers/python.py", line 116, in Deserializer
field = Model._meta.get_field(field_name)
File "/venv/lib/python3.6/site-packages/django/db/models/options.py", line 566, in get_field
sh-4.2$ cat database.json | ./manage.py loaddata --format json -
Traceback (most recent call last):
File "/venv/lib/python3.6/site-packages/django/db/models/options.py", line 564, in get_field
return self.fields_map[field_name]
KeyError: 'description'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/venv/lib/python3.6/site-packages/django/core/serializers/json.py", line 69, in Deserializer
yield from PythonDeserializer(objects, **options)
File "/venv/lib/python3.6/site-packages/django/core/serializers/python.py", line 116, in Deserializer
field = Model._meta.get_field(field_name)
File "/venv/lib/python3.6/site-packages/django/db/models/options.py", line 566, in get_field
raise FieldDoesNotExist("%s has no field named '%s'" % (self.object_name, field_name))
django.core.exceptions.FieldDoesNotExist: Classification has no field named 'description'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "./manage.py", line 12, in <module>
execute_from_command_line(sys.argv)
File "/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/venv/lib/python3.6/site-packages/django/core/management/base.py", line 316, in run_from_argv
self.execute(*args, **cmd_options)
File "/venv/lib/python3.6/site-packages/django/core/management/base.py", line 353, in execute
output = self.handle(*args, **options)
File "/venv/lib/python3.6/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
self.loaddata(fixture_labels)
File "/venv/lib/python3.6/site-packages/django/core/management/commands/loaddata.py", line 113, in loaddata
self.load_label(fixture_label)
File "/venv/lib/python3.6/site-packages/django/core/management/commands/loaddata.py", line 168, in load_label
for obj in objects:
File "/venv/lib/python3.6/site-packages/django/core/serializers/json.py", line 73, in Deserializer
raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing fixture '-':
In which version of Kiwi TCMS did you make your backup ?
It looks like backup is from an older version because 6.5 ships with migrations that remove Build.description and Classification.description fields!
I am not sure it is technically possible to handle this gracefully. Please file an issue on GitHub so we can investigate in more details and link back to this SO thread.
A work around for you will be to launch not the latest version of Kiwi TCMS but the version in which you made the backup. Then restore your data, upgrade to the latest version, run the migrations which will change DB schema and then backup again.
If you do not keep around your older docker image you will have to build it from source.

JupyterHub - oauth_client_id not found

I am using Azure to run python notebook using Jupyterhub. After spinning up the VM, I was able to access the notebooks just by using my username and password (just like ssh). However, one day later when I switched to another network (I am not claiming that the network might have been a problem) I am unable to access the link. It gives me The site can't be reached error.
So I tried rerunning the process again, and since then I have been struggling to make it run again. I have searched for similar issues on GitHub, but they aren't helpful either.
After the kill the process using kill pid command, I tried running the jupyterhub through this command:
/anaconda/envs/py35/bin/python /anaconda/envs/py35/bin/jupyterhub-singleuser --port=50387 --notebook-dir="~/notebooks" --config=/etc/jupyterhub/jupyterhub_config.py
And it gives me the error:
JUPYTERHUB_API_TOKEN env is required to run jupyterhub-singleuser. Did you launch it manually?
So I searched through github issues similar to this. I tried generating token manually using:
jupyterhub token username
And I added that token to JUPYTERHUB_API_TOKEN via export JUPYTERHUB_API_TOKEN=token. I also added token:username to c.Authenticator.tokens in jupyterhub_config.py. Now I get this error:
Traceback (most recent call last):
File "/anaconda/envs/py35/lib/python3.5/site-packages/traitlets/traitlets.py", line 528, in get
value = obj._trait_values[self.name]
KeyError: 'oauth_client_id'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/anaconda/envs/py35/bin/jupyterhub-singleuser", line 6, in <module>
main()
File "/anaconda/envs/py35/lib/python3.5/site-packages/jupyterhub/singleuser.py", line 455, in main
return SingleUserNotebookApp.launch_instance(argv)
File "/anaconda/envs/py35/lib/python3.5/site-packages/jupyter_core/application.py", line 267, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/anaconda/envs/py35/lib/python3.5/site-packages/traitlets/config/application.py", line 657, in launch_instance
app.initialize(argv)
File "<decorator-gen-7>", line 2, in initialize
File "/anaconda/envs/py35/lib/python3.5/site-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/anaconda/envs/py35/lib/python3.5/site-packages/notebook/notebookapp.py", line 1296, in initialize
self.init_webapp()
File "/anaconda/envs/py35/lib/python3.5/site-packages/jupyterhub/singleuser.py", line 393, in init_webapp
self.init_hub_auth()
File "/anaconda/envs/py35/lib/python3.5/site-packages/jupyterhub/singleuser.py", line 388, in init_hub_auth
if not self.hub_auth.oauth_client_id:
File "/anaconda/envs/py35/lib/python3.5/site-packages/traitlets/traitlets.py", line 556, in __get__
return self.get(obj, cls)
File "/anaconda/envs/py35/lib/python3.5/site-packages/traitlets/traitlets.py", line 535, in get
value = self._validate(obj, dynamic_default())
File "/anaconda/envs/py35/lib/python3.5/site-packages/traitlets/traitlets.py", line 593, in _validate
value = self._cross_validate(obj, value)
File "/anaconda/envs/py35/lib/python3.5/site-packages/traitlets/traitlets.py", line 599, in _cross_validate
value = obj._trait_validators[self.name](obj, proposal)
File "/anaconda/envs/py35/lib/python3.5/site-packages/traitlets/traitlets.py", line 907, in __call__
return self.func(*args, **kwargs)
File "/anaconda/envs/py35/lib/python3.5/site-packages/jupyterhub/services/auth.py", line 439, in _ensure_not_empty
raise ValueError("%s cannot be empty." % proposal.trait.name)
ValueError: oauth_client_id cannot be empty.
I am not sure where I went wrong in this process. Anybody familiar with this issue?
Try running jupyterhub instead of jupyterhub-singleuser
For your specific use case, the command would be as follows:
sudo /anaconda/envs/py35/bin/python /anaconda/envs/py35/bin/jupyterhub --port=50387 --notebook-dir="~/notebooks" --config=/etc/jupyterhub/jupyterhub_config.py
Make sure that jupyterhub is installed (correctly) in the path you mentioned.

Resources