"sam local start-api" fails with "invalid volume specification" on (Windows 10 Pro, Docker for Windows) - aws-sam-cli

I am using SAM CLI 0.6.0 and I am getting the error below when running sam local start-api with the app generated using sam init --runtime java
PS C:\Users\Kiran\AWS\SAM\java-sample\sam-app> sam local start-api
2018-09-03 10:49:49 Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]
2018-09-03 10:49:49 You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2018-09-03 10:49:49 * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
2018-09-03 10:49:58 Invoking helloworld.App::handleRequest (java8)
2018-09-03 10:49:58 Found credentials in shared credentials file: ~/.aws/credentials
2018-09-03 10:49:58 Decompressing C:\Users\Kiran\AWS\SAM\java-sample\sam-app\target\HelloWorld-1.0.jar
Fetching lambci/lambda:java8 Docker container image......
2018-09-03 10:49:59 Mounting C:\Users\Kiran\AppData\Local\Temp\tmp7f8z0_zj as /var/task:ro inside runtime container
2018-09-03 10:49:59 Exception on /hello [GET]
Traceback (most recent call last):
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\api\client.py", line 229, in _raise_for_status
response.raise_for_status()
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\requests\models.py", line 939, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://192.168.1.145:2376/v1.35/containers/create
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\flask\app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\flask\app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\flask\app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\flask\_compat.py", line 35, in reraise
raise value
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\flask\app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\flask\app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\samcli\local\apigw\local_apigw_service.py", line 140, in _request_handler
self.lambda_runner.invoke(route.function_name, event, stdout=stdout_stream, stderr=self.stderr)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\samcli\commands\local\lib\local_lambda.py", line 80, in invoke
self.local_runtime.invoke(config, event, debug_context=self.debug_context, stdout=stdout, stderr=stderr)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\samcli\local\lambdafn\runtime.py", line 79, in invoke
self._container_manager.run(container)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\samcli\local\docker\manager.py", line 61, in run
container.create()
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\samcli\local\docker\container.py", line 120, in create
real_container = self.docker_client.containers.create(self._image, **kwargs)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\models\containers.py", line 824, in create
resp = self.client.api.create_container(**create_kwargs)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\api\container.py", line 411, in create_container
return self.create_container_from_config(config, name)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\api\container.py", line 422, in create_container_from_config
return self._result(res, True)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\api\client.py", line 235, in _result
self._raise_for_status(response)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\api\client.py", line 231, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("invalid volume specification: 'C:\Users\Kiran\AppData\Local\Temp\tmp7f8z0_zj:/var/task:ro'")
2018-09-03 10:49:59 127.0.0.1 - - [03/Sep/2018 10:49:59] "GET /hello HTTP/1.1" 502 -
The path mentioned in the last line of the error message (C:\Users\Kiran\AppData\Local\Temp\tmp7f8z0_zj) doesn't seem to exist on my machine. I do have the jar specified code URI of the template file.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
sam-app
Sample SAM Template for sam-app
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 20
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: target/HelloWorld-1.0.jar
Handler: helloworld.App::handleRequest
Runtime: java8
Environment: # More info about Env Vars: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#environment-object
Variables:
PARAM1: VALUE
Events:
HelloWorld:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /hello
Method: get
Outputs:
HelloWorldApi:
Description: "API Gateway endpoint URL for Prod stage for Hello World function"
Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
HelloWorldFunction:
Description: "Hello World Lambda Function ARN"
Value: !GetAtt HelloWorldFunction.Arn
HelloWorldFunctionIamRole:
Description: "Implicit IAM Role created for Hello World function"
Value: !GetAtt HelloWorldFunctionRole.Arn
Appreciate any inputs to fix this error.
Thanks!

Same errors with docker toolbox
Check out this issue.

This is a bug in SAM CLI and fixed with latest release.
Do 'Sam --version' and you should see the following.
sam --version
SAM CLI, version 0.6.1.
This issue affected people running SAM on windows 10 with docker toolbox.
Vyas

Related

Local CosmosDB connectivity with pymongo for gramex

Not able to connect cosmos db using gramex mongo adapter.
From python, We can connect to local cosmosDB instance using:
# --------------------------------------
import pymongo
uri = r"mongodb://localhost:Tm%2BOzpb8BrV7DrZHSrGm3GMKyx9r%2Frl5ue9letmD1XRUUiafHFUyIQNenAQDla85nqVDrb8tr%2FtB0LR4azi1FQ%3D%3D#localhost:10255/admin?ssl=true"
client = pymongo.MongoClient(uri,
tls=True,
tlsCAFile='./documentdbemulatorcert.cer')
db = client.admin
print(db.command("serverStatus"))
# --------------------------------------
Please mind “tlsCAFile” parameter
However in gramex when I connect using:
# --------------------------------------
url:
envvartest-app-data:
pattern: /$YAMLURL/appdata
handler: FormHandler
kwargs:
url: "mongodb://localhost:Tm%2BOzpb8BrV7DrZHSrGm3GMKyx9r%2Frl5ue9letmD1XRUUiafHFUyIQNenAQDla85nqVDrb8tr%2FtB0LR4azi1FQ%3D%3D#localhost:10255/admin?ssl=true"
database: galaxy-dev
collection: threats
id: record_number
connect_args:
tls: True
tlsCAFile: $YAMLPATH/documentdbemulatorcert.cer
# ssl:
# ssl_ca: $YAMLPATH/documentdbemulatorcert.cer
# --------------------------------------
the connection fails reading:
ERROR 17-Feb 14:05:06 formhandler 9988 envvartest-app-data: filter failed
Traceback (most recent call last):
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\gramex\handlers\formhandler.py", line 158, in get
result[key] = yield val
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\tornado\gen.py", line 1133, in run
value = future.result()
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\concurrent\futures\_base.py", line 428, in result
return self.__get_result()
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\concurrent\futures\_base.py", line 384, in __get_result
raise self._exception
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\concurrent\futures\thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\gramex\data.py", line 235, in filter
data = method(url=url, controls=controls, args=args, query=query, **kwargs)
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\gramex\data.py", line 1474, in _filter_mongodb
meta_cols = pd.DataFrame(list(table.find().limit(100)))
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\pymongo\cursor.py", line 1238, in next
if len(self.__data) or self._refresh():
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\pymongo\cursor.py", line 1130, in _refresh
self.__session = self.__collection.database.client._ensure_session()
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\pymongo\mongo_client.py", line 1935, in _ensure_session
return self.__start_session(True, causal_consistency=False)
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\pymongo\mongo_client.py", line 1883, in __start_session
server_session = self._get_server_session()
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\pymongo\mongo_client.py", line 1921, in _get_server_session
return self._topology.get_server_session()
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\pymongo\topology.py", line 520, in get_server_session
session_timeout = self._check_session_support()
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\pymongo\topology.py", line 502, in _check_session_support
None)
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\pymongo\topology.py", line 220, in _select_servers_loop
(self._error_message(selector), timeout, self.description))
pymongo.errors.ServerSelectionTimeoutError: localhost:10255: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1091), Timeout: 30s, Topology Description: <TopologyDescription id: 620e089c395cf9273486aa57, topology_type: Single, servers: [<ServerDescription ('localhost', 10255) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:10255: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1091)')>]>
ERROR 17-Feb 14:05:06 __init__ 9988 500 GET /appdata (127.0.0.1) 30795.58ms envvartest-app-data
Not able to connect cosmos db using gramex mongo adapter.
Steps to reproduce
Install CosmosDB emulator
Generate access key:
.\Microsoft.Azure.Cosmos.Emulator.exe /GenKeyFile=D:\99exps\cosmosdb\key
Start cosmosDB emulator with mongoDB support:
.\Microsoft.Azure.Cosmos.Emulator.exe /FailOnSslCertificateNameMismatch /EnableMongoDbEndpoint=3.6 /EnableMongoDbEndpoint=3.2 /KeyFile=D:\99exps\cosmosdb\key
Create Gramex Application with above mentioned configuration
Observe the error
Edit 1:
The error is changed now:
ERROR 19-Feb 00:10:33 formhandler 9988 galaxy-app-data: filter failed
Traceback (most recent call last):
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\gramex\handlers\formhandler.py", line 158, in get
result[key] = yield val
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\tornado\gen.py", line 1133, in run
value = future.result()
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\concurrent\futures\_base.py", line 428, in result
return self.__get_result()
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\concurrent\futures\_base.py", line 384, in __get_result
raise self._exception
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\concurrent\futures\thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\gramex\data.py", line 239, in filter
columns=columns, **kwargs)
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\gramex\data.py", line 1521, in _filter_mongodb
cols = [k for k in table.find().limit(1)[0].keys()]
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\pymongo\cursor.py", line 694, in __getitem__
raise IndexError("no such item for Cursor instance")
IndexError: no such item for Cursor instance
WARNING 19-Feb 00:10:33 web 500 GET /appdata (::1): IndexError('no such item for Cursor instance')
ERROR 19-Feb 00:10:33 __init__ 9988 500 GET /appdata (::1) 846.11ms galaxy-app-data
As of Gramex 1.76.0 (Feb 2022), FormHandler does not connect to empty MongoDB collections, since there is no schema defined.
There is a plan to support empty MongoDB collections by specifying the schema explicitly.
But for now, you should be able to access the collection once you've added at least one row to it.

ClientError 404, cannot connect mongodb to neo4j

Rather than going into painful process of bringing everything into neo4j, we I resorted to easier solution. As configuration is not my expertise https://medium.com/neo4j/transform-mongodb-collections-automagically-into-graphs-9ea085d6e3ef
I followed instructions from here to configure (I want to import data in mongo to neo4j). I am in OSX Catalina. What endpoint should I be using? Following is my settings:
I have a virutal environment created with pipenv.
mongo-connector version: 3.1.1
Python version: 3.9.1
pymongo version: 3.11.3
MongoDB version: 4.4.3
neo4j_doc_manager version: unknown (well installed following instructions from the above).
Now when I try to connect and run following
mongo-connector -m localhost:27017 -t http://localhost:11005/data/db -d neo4j_doc_manager
I used endpoint because I have following endpoints
bolt: https://localhost:11004
http: https://localhost:11005
https: https://localhost:7473
my neo4j location
/Users/Library/Application Support/com.Neo4j.Relate/Data/dbmss/dbms-bbb9318b-9083-4ebf-a934-e17ef055ae22
I have disabled authentication:
dbms.security.auth_enabled=false
I have no idea why I get HTTP 404:
2021-02-05 08:11:15,577 [ALWAYS] mongo_connector.connector:50 - Starting mongo-connector version: 3.1.1
2021-02-05 08:11:15,578 [ALWAYS] mongo_connector.connector:50 - Python version: 3.9.1 (default, Jan 8 2021, 17:17:17)
[Clang 12.0.0 (clang-1200.0.32.28)]
2021-02-05 08:11:15,612 [ALWAYS] mongo_connector.connector:50 - Platform: macOS-10.15.7-x86_64-i386-64bit
2021-02-05 08:11:15,612 [ALWAYS] mongo_connector.connector:50 - pymongo version: 3.11.3
2021-02-05 08:11:15,619 [ALWAYS] mongo_connector.connector:50 - Source MongoDB version: 4.4.3
2021-02-05 08:11:15,619 [ALWAYS] mongo_connector.connector:50 - Target DocManager: mongo_connector.doc_managers.neo4j_doc_manager version: unknown
2021-02-05 08:11:15,633 [CRITICAL] mongo_connector.oplog_manager:713 - Exception during collection dump
Traceback (most recent call last):
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 258, in get
response = self.__base.get(headers=headers, redirect_limit=redirect_limit, **kwargs)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/packages/httpstream/http.py", line 966, in get
return self.__get_or_head("GET", if_modified_since, headers, redirect_limit, **kwargs)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/packages/httpstream/http.py", line 943, in __get_or_head
return rq.submit(redirect_limit=redirect_limit, **kwargs)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/packages/httpstream/http.py", line 452, in submit
return Response.wrap(http, uri, self, rs, **response_kwargs)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/packages/httpstream/http.py", line 489, in wrap
raise inst
py2neo.packages.httpstream.http.ClientError: 404 Not Found
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/util.py", line 33, in wrapped
return f(*args, **kwargs)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/doc_managers/neo4j_doc_manager.py", line 78, in bulk_upsert
tx = self.graph.cypher.begin()
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 661, in cypher
metadata = self.resource.metadata
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 213, in metadata
self.get()
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 267, in get
raise_from(self.error_class(message, **content), error)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/util.py", line 235, in raise_from
raise exception
py2neo.error.GraphError: HTTP GET returned response 404
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/oplog_manager.py", line 668, in do_dump
upsert_all(dm)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/oplog_manager.py", line 651, in upsert_all
dm.bulk_upsert(docs_to_dump(from_coll), mapped_ns, long_ts)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/util.py", line 44, in wrapped
raise new_type(str(exc_value)).with_traceback(exc_tb)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/util.py", line 33, in wrapped
return f(*args, **kwargs)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/doc_managers/neo4j_doc_manager.py", line 78, in bulk_upsert
tx = self.graph.cypher.begin()
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 661, in cypher
metadata = self.resource.metadata
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 213, in metadata
self.get()
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 267, in get
raise_from(self.error_class(message, **content), error)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/util.py", line 235, in raise_from
raise exception
mongo_connector.doc_managers.error_handler.Neo4jOperationFailed: HTTP GET returned response 404
2021-02-05 08:11:15,634 [ERROR] mongo_connector.oplog_manager:723 - OplogThread: Failed during dump collection cannot recover! Collection(Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True, replicaset='rs0'), 'local'), 'oplog.rs')
2021-02-05 08:11:16,630 [ERROR] mongo_connector.connector:408 - MongoConnector: OplogThread <OplogThread(Thread-2, started 123145708253184)> unexpectedly stopped! Shutting down
You should check whether the version of py2neo you are using is compatible with the version of Neo4j. I suspect it isn't, since py2neo.packages.httpstream is a wildly out of date package name, and probably refers too a deprecated/removed HTTP endpoint.

How to change Python default version in Docker?

I am using aws ec2 Ubuntu Server 20.04 LTS and having trouble with Python and Docker.
The python default version on ec2 is Python 3.8.5. I have also installed Python version 2.7.18.
But when I run docker or my complete network, docker is using the default version, which is python 3.8.5. So this causes errors for all peers, oderer and couchdb e.g.
ERROR: for ca_peerOrg2 a bytes-like object is required, not 'str'
ERROR: for orderer.example.com a bytes-like object is required, not 'str'
ERROR: for orderer4.example.com a bytes-like object is required, not 'str'
and also the following error
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/docker/api/client.py", line 261, in _raise_for_status
response.raise_for_status()
File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.22/containers/26d88164408dec7c247e07d47f803fd3767731bb0cf9748d9d0285de89c1f49f/start
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/compose/service.py", line 625, in start_container
container.start()
File "/usr/lib/python3/dist-packages/compose/container.py", line 241, in start
return self.client.start(self.id, **options)
File "/usr/lib/python3/dist-packages/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
File "/usr/lib/python3/dist-packages/docker/api/container.py", line 1095, in start
self._raise_for_status(res)
File "/usr/lib/python3/dist-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/lib/python3/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("b'driver failed programming external connectivity on endpoint couchdb3 (4cc5.....): Bind for 0.0.0.0:8984 failed: port is already allocated'")
I found a solution here, which I couldn't apply to mine.
Do you know, how I can change it, so that my docker runs without the errors? Thank you.

Connecting to external networks from inside minikube VM behind proxy in docker container

I have an active kubernetes cluster inside Minikube VM (using VirtualBox as driver), so for deploying new containers I am able to download the images as this connection is already laid out using istio service, now if I ssh into my minikube VM first of all I am not able to wget https content but http contents are connected after setting proxies and no_proxies but if I want to access any link from inside of my containers, say simple pod with python image and urllib library and I want to connect from inside this pod and then print the contents from any link (eg.http://python.org) I am not able to do so, all I am getting is no route to host error in logs which points to some problem with the connection due to proxies.
def basic():
import urllib.request
print("inside basic funtion")
with urllib.request.urlopen('http://python.org/') as response:
html = response.read()
print(html)
this is the python code I am running from inside my container as a pipeline component.
Most recent error I got-
Traceback (most recent call last):
File "/usr/local/lib/python3.7/urllib/request.py", line 1317, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/usr/local/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1016, in _send_output
self.send(msg)
File "/usr/local/lib/python3.7/http/client.py", line 956, in send
self.connect()
File "/usr/local/lib/python3.7/http/client.py", line 928, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/local/lib/python3.7/socket.py", line 727, in create_connection
raise err
File "/usr/local/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 11, in <module>
File "<string>", line 3, in basic
File "/usr/local/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/usr/local/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/usr/local/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.7/urllib/request.py", line 1345, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/usr/local/lib/python3.7/urllib/request.py", line 1319, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 110] Operation timed out>
I have started minikube as-
minikube start --cpus 6 --memory 12288 --disk-size=80g --extra-config=apiserver.service-account-issuer=api --extra-config=apiserver.service-account-signing-key-file=/var/lib/minikube/certs/apiserver.key --extra-config=apiserver.service-account-api-audiences=api --kubernetes-version v1.14.0
after setting the env variables as well.
Update:
I created a different container just to check the curl from inside the component as- (I am using kfp libraries for creating containers)
def curl_op(text):
return dsl.ContainerOp(
name='curl',
image='tutum/curl',
command=['sh', '-c'],
arguments=['curl -x http://<proxy-server>:<proxy-port> "$0"', text]
)
so using the above argument I am able to connect to external links, which again makes it certain that i need to create the containers with proxies set.
So for running the above python code I mentioned as pipeline component.
I added the environment variables using the os library and this individual piece was able to connect to external networks.
Updated python code-
def basic():
import urllib.request
import os
proxy = 'http://proxy-path:port'
os.environ['http_proxy'] = proxy
os.environ['HTTP_PROXY'] = proxy
os.environ['https_proxy'] = proxy
os.environ['HTTPS_PROXY'] = proxy
print("inside basic funtion")
with urllib.request.urlopen('http://python.org/') as response:
html = response.read()
print(html)
And if the docker image is created from scratch without taking help of pipeline library function then we need to just add the env details into our dockerfile the usual way after the base image call-
ENV HTTP_PROXY http://proxy-path:port
ENV HTTPS_PROXY http://proxy-path:port

Airflow docker operator Internal Server Error ("b'Mounts denied: EOF'") MacOS

I'm trying to use the docker operator on an airflow pipeline. This is the code I'm using:
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta
from airflow.operators.docker_operator import DockerOperator
default_args = {
'owner' : 'airflow',
'description' : 'Use of the DockerOperator',
'depend_on_past' : False,
'start_date' : datetime(2018, 1, 3),
'email_on_failure' : False,
'email_on_retry' : False,
'retries' : 1,
'retry_delay' : timedelta(minutes=5)
}
with DAG('docker_dag', default_args=default_args, schedule_interval="5 * * * *", catchup=False) as dag:
t1 = BashOperator(
task_id='print_current_date',
bash_command='date'
)
t2 = DockerOperator(
task_id='docker_command',
image='alpine:3.7',
api_version='auto',
auto_remove=True,
command="/bin/sleep 30",
docker_url="unix://var/run/docker.sock",
network_mode="bridge"
)
t3 = BashOperator(
task_id='print_hello',
bash_command='echo "hello world"'
)
t1 >> t2 >> t3
The original source is: How to use the DockerOperator in Apache Airflow
I run my DAG using the airflow UI , but in the docker_command task I get this error:
*** Reading local file: /Users/SoftwareDeveloper/airflow/logs/docker_dag/docker_command/2020-10-04T01:47:06.692609+00:00/2.log
[2020-10-03 20:52:21,857] {taskinstance.py:670} INFO - Dependencies all met for <TaskInstance: docker_dag.docker_command 2020-10-04T01:47:06.692609+00:00 [queued]>
[2020-10-03 20:52:21,876] {taskinstance.py:670} INFO - Dependencies all met for <TaskInstance: docker_dag.docker_command 2020-10-04T01:47:06.692609+00:00 [queued]>
[2020-10-03 20:52:21,877] {taskinstance.py:880} INFO -
--------------------------------------------------------------------------------
[2020-10-03 20:52:21,877] {taskinstance.py:881} INFO - Starting attempt 2 of 2
[2020-10-03 20:52:21,877] {taskinstance.py:882} INFO -
--------------------------------------------------------------------------------
[2020-10-03 20:52:21,886] {taskinstance.py:901} INFO - Executing <Task(DockerOperator): docker_command> on 2020-10-04T01:47:06.692609+00:00
[2020-10-03 20:52:21,889] {standard_task_runner.py:54} INFO - Started process 33165 to run task
[2020-10-03 20:52:21,929] {standard_task_runner.py:77} INFO - Running: ['airflow', 'run', 'docker_dag', 'docker_command', '2020-10-04T01:47:06.692609+00:00', '--job_id', '72', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/docker_operator.py', '--cfg_path', '/var/folders/9m/6w_b7jmn11s9yn6f3jyfd7rr0000gs/T/tmp98jxpu6w']
[2020-10-03 20:52:21,932] {standard_task_runner.py:78} INFO - Job 72: Subtask docker_command
[2020-10-03 20:52:21,977] {logging_mixin.py:112} INFO - Running %s on host %s <TaskInstance: docker_dag.docker_command 2020-10-04T01:47:06.692609+00:00 [running]> 64.1.168.192.in-addr.arpa
[2020-10-03 20:52:22,126] {docker_operator.py:210} INFO - Starting docker container from image alpine:3.7
[2020-10-03 20:52:22,192] {taskinstance.py:1150} ERROR - 500 Server Error: Internal Server Error ("b'Mounts denied: EOF'")
Traceback (most recent call last):
File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/docker/api/client.py", line 259, in _raise_for_status
response.raise_for_status()
File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.40/containers/49bf0d929c8d2524a778b5bdc255544c5a3e4915530bbea379b8e147d765a5c6/start
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
result = task_copy.execute(context=context)
File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/airflow/operators/docker_operator.py", line 277, in execute
return self._run_image()
File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/airflow/operators/docker_operator.py", line 233, in _run_image
self.cli.start(self.container['Id'])
File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/docker/api/container.py", line 1108, in start
self._raise_for_status(res)
File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/docker/api/client.py", line 261, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("b'Mounts denied: EOF'")
[2020-10-03 20:52:22,202] {taskinstance.py:1187} INFO - Marking task as FAILED. dag_id=docker_dag, task_id=docker_command, execution_date=20201004T014706, start_date=20201004T015221, end_date=20201004T015222
[2020-10-03 20:52:26,855] {local_task_job.py:102} INFO - Task exited with return code 1
Update: This exercise I did it on a Mac, but I tried to do the same on a Linux OS (Ubuntu 18.04 distribution) and everything is working fine. So, I'm afraid this is related with something about my configuration on macOS or the permissions I have on the computer.
On Mac I have Catalina version 10.15.5.
Do you have any idea why I'm getting that error?

Resources