Airflow docker operator Internal Server Error ("b'Mounts denied: EOF'") MacOS - docker

I'm trying to use the docker operator on an airflow pipeline. This is the code I'm using:
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta
from airflow.operators.docker_operator import DockerOperator
default_args = {
'owner' : 'airflow',
'description' : 'Use of the DockerOperator',
'depend_on_past' : False,
'start_date' : datetime(2018, 1, 3),
'email_on_failure' : False,
'email_on_retry' : False,
'retries' : 1,
'retry_delay' : timedelta(minutes=5)
}
with DAG('docker_dag', default_args=default_args, schedule_interval="5 * * * *", catchup=False) as dag:
t1 = BashOperator(
task_id='print_current_date',
bash_command='date'
)
t2 = DockerOperator(
task_id='docker_command',
image='alpine:3.7',
api_version='auto',
auto_remove=True,
command="/bin/sleep 30",
docker_url="unix://var/run/docker.sock",
network_mode="bridge"
)
t3 = BashOperator(
task_id='print_hello',
bash_command='echo "hello world"'
)
t1 >> t2 >> t3
The original source is: How to use the DockerOperator in Apache Airflow
I run my DAG using the airflow UI , but in the docker_command task I get this error:
*** Reading local file: /Users/SoftwareDeveloper/airflow/logs/docker_dag/docker_command/2020-10-04T01:47:06.692609+00:00/2.log
[2020-10-03 20:52:21,857] {taskinstance.py:670} INFO - Dependencies all met for <TaskInstance: docker_dag.docker_command 2020-10-04T01:47:06.692609+00:00 [queued]>
[2020-10-03 20:52:21,876] {taskinstance.py:670} INFO - Dependencies all met for <TaskInstance: docker_dag.docker_command 2020-10-04T01:47:06.692609+00:00 [queued]>
[2020-10-03 20:52:21,877] {taskinstance.py:880} INFO -
--------------------------------------------------------------------------------
[2020-10-03 20:52:21,877] {taskinstance.py:881} INFO - Starting attempt 2 of 2
[2020-10-03 20:52:21,877] {taskinstance.py:882} INFO -
--------------------------------------------------------------------------------
[2020-10-03 20:52:21,886] {taskinstance.py:901} INFO - Executing <Task(DockerOperator): docker_command> on 2020-10-04T01:47:06.692609+00:00
[2020-10-03 20:52:21,889] {standard_task_runner.py:54} INFO - Started process 33165 to run task
[2020-10-03 20:52:21,929] {standard_task_runner.py:77} INFO - Running: ['airflow', 'run', 'docker_dag', 'docker_command', '2020-10-04T01:47:06.692609+00:00', '--job_id', '72', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/docker_operator.py', '--cfg_path', '/var/folders/9m/6w_b7jmn11s9yn6f3jyfd7rr0000gs/T/tmp98jxpu6w']
[2020-10-03 20:52:21,932] {standard_task_runner.py:78} INFO - Job 72: Subtask docker_command
[2020-10-03 20:52:21,977] {logging_mixin.py:112} INFO - Running %s on host %s <TaskInstance: docker_dag.docker_command 2020-10-04T01:47:06.692609+00:00 [running]> 64.1.168.192.in-addr.arpa
[2020-10-03 20:52:22,126] {docker_operator.py:210} INFO - Starting docker container from image alpine:3.7
[2020-10-03 20:52:22,192] {taskinstance.py:1150} ERROR - 500 Server Error: Internal Server Error ("b'Mounts denied: EOF'")
Traceback (most recent call last):
File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/docker/api/client.py", line 259, in _raise_for_status
response.raise_for_status()
File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.40/containers/49bf0d929c8d2524a778b5bdc255544c5a3e4915530bbea379b8e147d765a5c6/start
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
result = task_copy.execute(context=context)
File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/airflow/operators/docker_operator.py", line 277, in execute
return self._run_image()
File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/airflow/operators/docker_operator.py", line 233, in _run_image
self.cli.start(self.container['Id'])
File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/docker/api/container.py", line 1108, in start
self._raise_for_status(res)
File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/docker/api/client.py", line 261, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("b'Mounts denied: EOF'")
[2020-10-03 20:52:22,202] {taskinstance.py:1187} INFO - Marking task as FAILED. dag_id=docker_dag, task_id=docker_command, execution_date=20201004T014706, start_date=20201004T015221, end_date=20201004T015222
[2020-10-03 20:52:26,855] {local_task_job.py:102} INFO - Task exited with return code 1
Update: This exercise I did it on a Mac, but I tried to do the same on a Linux OS (Ubuntu 18.04 distribution) and everything is working fine. So, I'm afraid this is related with something about my configuration on macOS or the permissions I have on the computer.
On Mac I have Catalina version 10.15.5.
Do you have any idea why I'm getting that error?

Related

ClientError 404, cannot connect mongodb to neo4j

Rather than going into painful process of bringing everything into neo4j, we I resorted to easier solution. As configuration is not my expertise https://medium.com/neo4j/transform-mongodb-collections-automagically-into-graphs-9ea085d6e3ef
I followed instructions from here to configure (I want to import data in mongo to neo4j). I am in OSX Catalina. What endpoint should I be using? Following is my settings:
I have a virutal environment created with pipenv.
mongo-connector version: 3.1.1
Python version: 3.9.1
pymongo version: 3.11.3
MongoDB version: 4.4.3
neo4j_doc_manager version: unknown (well installed following instructions from the above).
Now when I try to connect and run following
mongo-connector -m localhost:27017 -t http://localhost:11005/data/db -d neo4j_doc_manager
I used endpoint because I have following endpoints
bolt: https://localhost:11004
http: https://localhost:11005
https: https://localhost:7473
my neo4j location
/Users/Library/Application Support/com.Neo4j.Relate/Data/dbmss/dbms-bbb9318b-9083-4ebf-a934-e17ef055ae22
I have disabled authentication:
dbms.security.auth_enabled=false
I have no idea why I get HTTP 404:
2021-02-05 08:11:15,577 [ALWAYS] mongo_connector.connector:50 - Starting mongo-connector version: 3.1.1
2021-02-05 08:11:15,578 [ALWAYS] mongo_connector.connector:50 - Python version: 3.9.1 (default, Jan 8 2021, 17:17:17)
[Clang 12.0.0 (clang-1200.0.32.28)]
2021-02-05 08:11:15,612 [ALWAYS] mongo_connector.connector:50 - Platform: macOS-10.15.7-x86_64-i386-64bit
2021-02-05 08:11:15,612 [ALWAYS] mongo_connector.connector:50 - pymongo version: 3.11.3
2021-02-05 08:11:15,619 [ALWAYS] mongo_connector.connector:50 - Source MongoDB version: 4.4.3
2021-02-05 08:11:15,619 [ALWAYS] mongo_connector.connector:50 - Target DocManager: mongo_connector.doc_managers.neo4j_doc_manager version: unknown
2021-02-05 08:11:15,633 [CRITICAL] mongo_connector.oplog_manager:713 - Exception during collection dump
Traceback (most recent call last):
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 258, in get
response = self.__base.get(headers=headers, redirect_limit=redirect_limit, **kwargs)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/packages/httpstream/http.py", line 966, in get
return self.__get_or_head("GET", if_modified_since, headers, redirect_limit, **kwargs)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/packages/httpstream/http.py", line 943, in __get_or_head
return rq.submit(redirect_limit=redirect_limit, **kwargs)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/packages/httpstream/http.py", line 452, in submit
return Response.wrap(http, uri, self, rs, **response_kwargs)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/packages/httpstream/http.py", line 489, in wrap
raise inst
py2neo.packages.httpstream.http.ClientError: 404 Not Found
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/util.py", line 33, in wrapped
return f(*args, **kwargs)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/doc_managers/neo4j_doc_manager.py", line 78, in bulk_upsert
tx = self.graph.cypher.begin()
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 661, in cypher
metadata = self.resource.metadata
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 213, in metadata
self.get()
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 267, in get
raise_from(self.error_class(message, **content), error)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/util.py", line 235, in raise_from
raise exception
py2neo.error.GraphError: HTTP GET returned response 404
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/oplog_manager.py", line 668, in do_dump
upsert_all(dm)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/oplog_manager.py", line 651, in upsert_all
dm.bulk_upsert(docs_to_dump(from_coll), mapped_ns, long_ts)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/util.py", line 44, in wrapped
raise new_type(str(exc_value)).with_traceback(exc_tb)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/util.py", line 33, in wrapped
return f(*args, **kwargs)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/doc_managers/neo4j_doc_manager.py", line 78, in bulk_upsert
tx = self.graph.cypher.begin()
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 661, in cypher
metadata = self.resource.metadata
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 213, in metadata
self.get()
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 267, in get
raise_from(self.error_class(message, **content), error)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/util.py", line 235, in raise_from
raise exception
mongo_connector.doc_managers.error_handler.Neo4jOperationFailed: HTTP GET returned response 404
2021-02-05 08:11:15,634 [ERROR] mongo_connector.oplog_manager:723 - OplogThread: Failed during dump collection cannot recover! Collection(Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True, replicaset='rs0'), 'local'), 'oplog.rs')
2021-02-05 08:11:16,630 [ERROR] mongo_connector.connector:408 - MongoConnector: OplogThread <OplogThread(Thread-2, started 123145708253184)> unexpectedly stopped! Shutting down
You should check whether the version of py2neo you are using is compatible with the version of Neo4j. I suspect it isn't, since py2neo.packages.httpstream is a wildly out of date package name, and probably refers too a deprecated/removed HTTP endpoint.

How to change Python default version in Docker?

I am using aws ec2 Ubuntu Server 20.04 LTS and having trouble with Python and Docker.
The python default version on ec2 is Python 3.8.5. I have also installed Python version 2.7.18.
But when I run docker or my complete network, docker is using the default version, which is python 3.8.5. So this causes errors for all peers, oderer and couchdb e.g.
ERROR: for ca_peerOrg2 a bytes-like object is required, not 'str'
ERROR: for orderer.example.com a bytes-like object is required, not 'str'
ERROR: for orderer4.example.com a bytes-like object is required, not 'str'
and also the following error
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/docker/api/client.py", line 261, in _raise_for_status
response.raise_for_status()
File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.22/containers/26d88164408dec7c247e07d47f803fd3767731bb0cf9748d9d0285de89c1f49f/start
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/compose/service.py", line 625, in start_container
container.start()
File "/usr/lib/python3/dist-packages/compose/container.py", line 241, in start
return self.client.start(self.id, **options)
File "/usr/lib/python3/dist-packages/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
File "/usr/lib/python3/dist-packages/docker/api/container.py", line 1095, in start
self._raise_for_status(res)
File "/usr/lib/python3/dist-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/lib/python3/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("b'driver failed programming external connectivity on endpoint couchdb3 (4cc5.....): Bind for 0.0.0.0:8984 failed: port is already allocated'")
I found a solution here, which I couldn't apply to mine.
Do you know, how I can change it, so that my docker runs without the errors? Thank you.

Can dask parallelization be encapsulated in a class?

Is it possible to encapsulate dask parallelization within a class? In its final form my class will do a lot of initialization before calling run - I trimmed my issue down to frame the question. Note that the code works for a LocalCluster and also distributed calcs outside of a class works fine on the same HPC cluster. Here is the boiled down code along with the corresponding error messages:
import numpy as np
from dask_jobqueue import PBSCluster
from dask.distributed import Client
from dask.distributed import wait
class Simulate:
def __init__(self):
pass
def run(self):
cluster = PBSCluster(cores=12, memory='1GB', queue='low', project='classtest', name='classtest_dask',
walltime='02:00:00', local_directory='/scratch/mmatthews')
cluster.scale(10) # Ask for # workers
client = Client(cluster)
seeds = list(np.arange(100))
a = client.map(self.run_trial, seeds)
wait(a)
trial_results = [a[i].result() for i in range(len(a))]
cluster.scale(0)
cluster.close()
def run_trial(self, trial_seed):
np.random.seed(trial_seed)
rst = np.random.randint
print('Simulation Finished rst=%s' % rst)
return rst
simob = Simulate()
simob.run()
Error to StdErr:
> distributed.client - ERROR - Failed to reconnect to scheduler after
> 10.00 seconds, closing client distributed.utils - ERROR - Traceback (most recent call last): File
> "/nfs/system/miniconda3_dev/envs/rosetta_dev/lib/python3.7/site-packages/distributed/utils.py",
> line 666, in log_errors
> yield File "/nfs/system/miniconda3_dev/envs/rosetta_dev/lib/python3.7/site-packages/distributed/client.py",
> line 1268, in _close
> await gen.with_timeout(timedelta(seconds=2), list(coroutines)) concurrent.futures._base.CancelledError distributed.utils - ERROR -
> Traceback (most recent call last): File
> "/nfs/system/miniconda3_dev/envs/rosetta_dev/lib/python3.7/site-packages/distributed/utils.py",
> line 666, in log_errors
> yield File "/nfs/system/miniconda3_dev/envs/rosetta_dev/lib/python3.7/site-packages/distributed/client.py",
> line 998, in _reconnect
> await self._close() File "/nfs/system/miniconda3_dev/envs/rosetta_dev/lib/python3.7/site-packages/distributed/client.py",
> line 1268, in _close
> await gen.with_timeout(timedelta(seconds=2), list(coroutines)) concurrent.futures._base.CancelledError
Error in PBS error file:
$ cat classtest_dask.e156272
distributed.nanny - INFO - Start Nanny at: 'tcp://160.84.192.224:40753'
distributed.diskutils - INFO - Found stale lock file and directory '/scratch/mmatthews/worker-bnjpcqmq', purging
distributed.worker - INFO - Start worker at: tcp://160.84.192.224:44564
distributed.worker - INFO - Listening to: tcp://160.84.192.224:44564
distributed.worker - INFO - dashboard at: 160.84.192.224:35232
distributed.worker - INFO - Waiting to connect to: tcp://160.84.192.193:39664
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 12
distributed.worker - INFO - Memory: 1000.00 MB
distributed.worker - INFO - Local Directory: /scratch/mmatthews/worker-kbw6dtj_
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Registered to: tcp://160.84.192.193:39664
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.dask_worker - INFO - Exiting on signal 15
distributed.nanny - INFO - Closing Nanny at 'tcp://160.84.192.224:40753'
distributed.dask_worker - INFO - End worker
Traceback (most recent call last):
File "/nfs/system/miniconda3_dev/envs/rosetta_dev/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/nfs/system/miniconda3_dev/envs/rosetta_dev/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/nfs/system/miniconda3_dev/envs/rosetta_dev/lib/python3.7/site-packages/distributed/cli/dask_worker.py", line 410, in <module>
go()
File "/nfs/system/miniconda3_dev/envs/rosetta_dev/lib/python3.7/site-packages/distributed/cli/dask_worker.py", line 406, in go
main()
File "/nfs/system/miniconda3_dev/envs/rosetta_dev/lib/python3.7/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/nfs/system/miniconda3_dev/envs/rosetta_dev/lib/python3.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/nfs/system/miniconda3_dev/envs/rosetta_dev/lib/python3.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/nfs/system/miniconda3_dev/envs/rosetta_dev/lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/nfs/system/miniconda3_dev/envs/rosetta_dev/lib/python3.7/site-packages/distributed/cli/dask_worker.py", line 397, in main
raise TimeoutError("Timed out starting worker.") from None
tornado.util.TimeoutError: Timed out starting worker.
Can dask parallelization be encapsulated in a class?
Yes. Dask calls are just normal Python calls. There is nothing that stops them from interacting with the rest of the language.
Your actual error seems to be entirely unrelated. It looks like something killed your worker
distributed.dask_worker - INFO - Exiting on signal 15
Unfortunately there is no information about what that was. I recommend checking with your system administrator.

Apache Spark and Apache Airflow connection in Docker based solution

I have Spark and Airflow cluster, I want to send a spark job from Airflow container to Spark container. But I am new about Airflow and I don't know which configuration I need to perform. I copied spark_submit_operator.py under the plugins folder.
from airflow import DAG
from airflow.contrib.operators.spark_submit_operator import SparkSubmitOperator
from datetime import datetime, timedelta
args = {
'owner': 'airflow',
'start_date': datetime(2018, 7, 31)
}
dag = DAG('spark_example_new', default_args=args, schedule_interval="*/10 * * * *")
operator = SparkSubmitOperator(
task_id='spark_submit_job',
conn_id='spark_default',
java_class='Simple',
application='/spark/abc.jar',
total_executor_cores='1',
executor_cores='1',
executor_memory='2g',
num_executors='1',
name='airflow-spark-example',
verbose=False,
driver_memory='1g',
application_args=["1000"],
conf={'master':'spark://master:7077'},
dag=dag,
)
master is the hostname of our Spark Master container. When I run the dag, it produce following error:
[2018-09-20 05:57:46,637] {{models.py:1569}} INFO - Executing <Task(SparkSubmitOperator): spark_submit_job> on 2018-09-20T05:57:36.756154+00:00
[2018-09-20 05:57:46,637] {{base_task_runner.py:124}} INFO - Running: ['bash', '-c', 'airflow run spark_example_new spark_submit_job 2018-09-20T05:57:36.756154+00:00 --job_id 4 --raw -sd DAGS_FOLDER/firstJob.py --cfg_path /tmp/tmpn2hznb5_']
[2018-09-20 05:57:47,002] {{base_task_runner.py:107}} INFO - Job 4: Subtask spark_submit_job [2018-09-20 05:57:47,001] {{settings.py:174}} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
[2018-09-20 05:57:47,312] {{base_task_runner.py:107}} INFO - Job 4: Subtask spark_submit_job [2018-09-20 05:57:47,311] {{__init__.py:51}} INFO - Using executor CeleryExecutor
[2018-09-20 05:57:47,428] {{base_task_runner.py:107}} INFO - Job 4: Subtask spark_submit_job [2018-09-20 05:57:47,428] {{models.py:258}} INFO - Filling up the DagBag from /usr/local/airflow/dags/firstJob.py
[2018-09-20 05:57:47,447] {{base_task_runner.py:107}} INFO - Job 4: Subtask spark_submit_job [2018-09-20 05:57:47,447] {{cli.py:492}} INFO - Running <TaskInstance: spark_example_new.spark_submit_job 2018-09-20T05:57:36.756154+00:00 [running]> on host e6dd59dc595f
[2018-09-20 05:57:47,471] {{logging_mixin.py:95}} INFO - [2018-09-20 05:57:47,470] {{spark_submit_hook.py:283}} INFO - Spark-Submit cmd: ['spark-submit', '--master', 'yarn', '--conf', 'master=spark://master:7077', '--num-executors', '1', '--total-executor-cores', '1', '--executor-cores', '1', '--executor-memory', '2g', '--driver-memory', '1g', '--name', 'airflow-spark-example', '--class', 'Simple', '/spark/ugur.jar', '1000']
[2018-09-20 05:57:47,473] {{models.py:1736}} ERROR - [Errno 2] No such file or directory: 'spark-submit': 'spark-submit'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1633, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/spark_submit_operator.py", line 168, in execute
self._hook.submit(self._application)
File "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/spark_submit_hook.py", line 330, in submit
**kwargs)
File "/usr/local/lib/python3.6/subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python3.6/subprocess.py", line 1344, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'spark-submit': 'spark-submit'
It's running command:
Spark-Submit cmd: ['spark-submit', '--master', 'yarn', '--conf', 'master=spark://master:7077', '--num-executors', '1', '--total-executor-cores', '1', '--executor-cores', '1', '--executor-memory', '2g', '--driver-memory', '1g', '--name', 'airflow-spark-example', '--class', 'Simple', '/spark/ugur.jar', '1000']
but I didn't use yarn.
If you use SparkSubmitOperator the connection to master will be set as "Yarn" by default regardless master which you set on your python code, however, you can override master by specifying conn_id through its constructor with the condition you have already created the aforementioned conn_id at "Admin->Connection" menu in Airflow Web Interface. I hope it can help you.
I guess you need to set master in extra options in your connections for this spark conn id (spark_default).By default is yarn, so try{"master":"your-con"}
Also does your airflow user has spark-submit in the classpath?, from logs looks like it doesnt.

"sam local start-api" fails with "invalid volume specification" on (Windows 10 Pro, Docker for Windows)

I am using SAM CLI 0.6.0 and I am getting the error below when running sam local start-api with the app generated using sam init --runtime java
PS C:\Users\Kiran\AWS\SAM\java-sample\sam-app> sam local start-api
2018-09-03 10:49:49 Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]
2018-09-03 10:49:49 You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2018-09-03 10:49:49 * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
2018-09-03 10:49:58 Invoking helloworld.App::handleRequest (java8)
2018-09-03 10:49:58 Found credentials in shared credentials file: ~/.aws/credentials
2018-09-03 10:49:58 Decompressing C:\Users\Kiran\AWS\SAM\java-sample\sam-app\target\HelloWorld-1.0.jar
Fetching lambci/lambda:java8 Docker container image......
2018-09-03 10:49:59 Mounting C:\Users\Kiran\AppData\Local\Temp\tmp7f8z0_zj as /var/task:ro inside runtime container
2018-09-03 10:49:59 Exception on /hello [GET]
Traceback (most recent call last):
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\api\client.py", line 229, in _raise_for_status
response.raise_for_status()
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\requests\models.py", line 939, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://192.168.1.145:2376/v1.35/containers/create
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\flask\app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\flask\app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\flask\app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\flask\_compat.py", line 35, in reraise
raise value
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\flask\app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\flask\app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\samcli\local\apigw\local_apigw_service.py", line 140, in _request_handler
self.lambda_runner.invoke(route.function_name, event, stdout=stdout_stream, stderr=self.stderr)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\samcli\commands\local\lib\local_lambda.py", line 80, in invoke
self.local_runtime.invoke(config, event, debug_context=self.debug_context, stdout=stdout, stderr=stderr)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\samcli\local\lambdafn\runtime.py", line 79, in invoke
self._container_manager.run(container)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\samcli\local\docker\manager.py", line 61, in run
container.create()
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\samcli\local\docker\container.py", line 120, in create
real_container = self.docker_client.containers.create(self._image, **kwargs)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\models\containers.py", line 824, in create
resp = self.client.api.create_container(**create_kwargs)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\api\container.py", line 411, in create_container
return self.create_container_from_config(config, name)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\api\container.py", line 422, in create_container_from_config
return self._result(res, True)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\api\client.py", line 235, in _result
self._raise_for_status(response)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\api\client.py", line 231, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("invalid volume specification: 'C:\Users\Kiran\AppData\Local\Temp\tmp7f8z0_zj:/var/task:ro'")
2018-09-03 10:49:59 127.0.0.1 - - [03/Sep/2018 10:49:59] "GET /hello HTTP/1.1" 502 -
The path mentioned in the last line of the error message (C:\Users\Kiran\AppData\Local\Temp\tmp7f8z0_zj) doesn't seem to exist on my machine. I do have the jar specified code URI of the template file.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
sam-app
Sample SAM Template for sam-app
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 20
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: target/HelloWorld-1.0.jar
Handler: helloworld.App::handleRequest
Runtime: java8
Environment: # More info about Env Vars: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#environment-object
Variables:
PARAM1: VALUE
Events:
HelloWorld:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /hello
Method: get
Outputs:
HelloWorldApi:
Description: "API Gateway endpoint URL for Prod stage for Hello World function"
Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
HelloWorldFunction:
Description: "Hello World Lambda Function ARN"
Value: !GetAtt HelloWorldFunction.Arn
HelloWorldFunctionIamRole:
Description: "Implicit IAM Role created for Hello World function"
Value: !GetAtt HelloWorldFunctionRole.Arn
Appreciate any inputs to fix this error.
Thanks!
Same errors with docker toolbox
Check out this issue.
This is a bug in SAM CLI and fixed with latest release.
Do 'Sam --version' and you should see the following.
sam --version
SAM CLI, version 0.6.1.
This issue affected people running SAM on windows 10 with docker toolbox.
Vyas

Resources