I'm using spark-submit in "cluster" mode with a Python script against a Spark cluster running on Mesos, and a custom Docker image for the executor set in spark.mesos.executor.docker.image.
My script file is already baked into the Docker image (let's say at path /app/script.py), so I don't want to use spark-submit's feature to download the script via HTTP before executing.
Per https://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management this seems possible by specifying the application script as a local: URL, e.g. spark-submit [...options...] local:/app/script.py. However this didn't work, I'm seeing errors like the following on Mesos (stderr of the spark driver task, scheduled by the spark-dispatcher framework):
I0727 20:31:50.164263 9207 fetcher.cpp:533] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/root","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"\/app\/script.py"}}],"sandbox_directory":"\/data\/mesos\/slaves\/GUID\/frameworks\/GUID\/executors\/driver-TIMESTAMP\/runs\/GUID","user":"root"}
I0727 20:31:50.170289 9207 fetcher.cpp:444] Fetching URI '/app/script.py'
I0727 20:31:50.170361 9207 fetcher.cpp:285] Fetching directly into the sandbox directory
I0727 20:31:50.170413 9207 fetcher.cpp:222] Fetching URI '/app/script.py'
cp: cannot stat ‘/app/script.py’: No such file or directory
E0727 20:31:50.174051 9207 fetcher.cpp:579] EXIT with status 1: Failed to fetch '/app/script.py': Failed to copy '/app/script.py': exited with status 1
s 1
After browsing through https://spark.apache.org/docs/latest/running-on-mesos.html, my guess is that the local: path is interpreted by a "MesosClusterDispatcher", which is a daemon that spins up a container for the Spark driver process (using my custom spark executor Docker image). Since this dispatcher doesn't itself run in the custom Docker image/container, it can't find the file.
Is there any other way to tell spark-submit to not download the application script and just use the script already present in the Docker image?
Related
I just downloaded this docker image to set up a spark cluster with two worker nodes. Cluster is up and running however I want to submit my scala file to this cluster. I am not able to start spark-shell in this.
When I was using another docker image, I was able to start it using spark-shell.
Can someone please explain if I need to install scala separately in the image or there is a different way to start
UPDATE
Here is the error bash: spark-shell: command not found
bash: spark-shell: command not found
root#a7b0682ff17d:/opt/spark# ls /home/shangupta/Scripts/
ProfileData.json demo.scala queries.scala
TestDataGeneration.sql input.scala
root#a7b0682ff17d:/opt/spark# spark-shell /home/shangupta/Scripts/input.scala
bash: spark-shell: command not found
root#a7b0682ff17d:/opt/spark#
You're getting command not found because PATH isn't correctly established
Use the absolute path /opt/spark/bin/spark-shell
Also, I'd suggest packaging your Scala project as an uber jar to submit unless you have no external dependencies or like to add --packages/--jars manually
I am trying to customize tuna-app chaincode of the tuna-app example. I want to use cid package inside my chaincode to make ABAC decisions about who is allowed to run the chaincode. When I try to install chaincode, I get the following error:
Error: Error getting chaincode code chaincode:
Error getting chaincode package bytes: Error obtaining dependencies for github.com/hyperledger/fabric/core/chaincode/lib/cid:
<go, [list -f {{ join .Deps "\n"}} github.com/hyperledger/fabric/core/chaincode/lib/cid]>: failed with error: "exit status 1"
cannot load package: package github.com/hyperledger/fabric/core/chaincode/lib/cid: cannot find package "github.com/hyperledger/fabric/core/chaincode/lib/cid" in any of:
/opt/go/src/github.com/hyperledger/fabric/core/chaincode/lib/cid (from $GOROOT)
/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/lib/cid (from $GOPATH)
I am usind Docker to run peer, orderer, ca, and cli containers. The Docker image which is used to build chaincode is hyperledger/fabric-ccenv. This image is created using Dockerfile; the interesting line I found was:
ADD payload/goshim.tar.bz2 $GOPATH/src/
which adds the tar.bz2 inside the $GOPATH/src folder (I believe). The .tar.bz2 file contains all Go packages used by chaincode. I tried to insert the cid package and to create a new .tar.bz2 file with the package inside. Then I rebuilt the image. The image now contains the cid package, but I still get the same error.
Why is it still missing the package?
In the startFabric.sh from your tuna-app, you launch the cli container using:
docker-compose -f ./docker-compose.yml up -d cli
Have a look at the mounting declaration of the persistent volumes in your compose yaml file. You should see something like this because the tuna-app is based on fabcar from the fabric-samples:
./../chaincode/:/opt/gopath/src/github.com/
If you see this declaration, copy in your local machine the folder /hyperledger/fabric/core/chaincode/lib/cid into your chaincode folder. You should find it in chaincode/abac if you are using the last version of fabric samples (https://github.com/hyperledger/fabric-samples).
I think you should not create a new goshim.tar.bz2. If you think it is easier make sure cid is in the correct path within the archive, e.g. github.com/hyperledger/fabric/core/chaincode/lib/cid
To test this you can make a debug output:
ADD payload/goshim.tar.bz2 $GOPATH/src/
RUN ls $GOPATH/src/github.com/hyperledger/fabric/core/chaincode/lib/cid
I would recommend to download cid within the Dockerfile:
RUN go get -d github.com/hyperledger/fabric/core/chaincode/lib/cid
According to the documentation at bazelbuild/rules_docker, it should be possible to work with these container images on OSX, and it also claims that it's possible to do so without docker.
These rules do not require / use Docker for pulling, building, or pushing images. This means:
They can be used to develop Docker containers on Windows / OSX without boot2docker or docker-machine installed.
They do not require root access on your workstation.
How do I do that? Here's a simple rule:
go_image(
name = "helloworld_image",
importpath = "github.com/nictuku/helloworld",
library = ":go_default_library",
visibility = ["//visibility:public"],
)
I can build the image with bazel build :helloworld_image. It produces a tar ball in blaze-bin, but it won't run it:
INFO: Running command line: bazel-bin/helloworld_image
Loaded image ID: sha256:08d312b529d30431c68741fd3a31468a02533f27a8c2c29eedc969dae5a39852
Tagging 08d312b529d30431c68741fd3a31468a02533f27a8c2c29eedc969dae5a39852 as bazel:helloworld_image
standard_init_linux.go:185: exec user process caused "exec format error"
ERROR: Non-zero return code '1' from command: Process exited with status 1.
It's trying to run the linux this is OSX, which is silly.
I also tried doing a "docker load" on the .tar content but it doesn't seem to like that format.
$ docker load -i bazel-bin/helloworld_image-layer.tar
open /var/lib/docker/tmp/docker-import-330829602/app/json: no such file or directory
Help? Thanks!
You are building for your host platform by default so you need to build for the container platform if you want to do that.
Since you are using a go binary, you can do cross compilation by specifying --cpu=k8 on the command line. Ideally we would be able to just say that the docker image needs a linux binary (so no need to specify the --cpu command-line flag) but this is still a work in progress in Bazel.
I have created a cluster on Digital Ocean (DC/OS 1.9) using terraform following these instructions here
Everything seems to have installed correctly, to pull from a private docker repo, I need to add a compressed .docker file to my /core/home/ and fetch it during deployment by including it in my JSON.
"fetch":[
{
"uri":"file:///home/core/docker.tar.gz"
}
]
Based on these instructions: https://docs.mesosphere.com/1.9/deploying-services/momee/docker-creds-agent/
And I'm still getting errors:
Failed to launch container:
Failed to fetch all URIs for container 'abc123-xxxxx' with exit status: 256
Upon looking at the logs of one of the agents:
Starting container '123-abc-xxx' for task 'my-docker-image-service.321-dfg-xxx' (and executor 'my-docker-image-service.397d20cb-1
Begin fetcher log (stderr in sandbox) for container 123-abc-xxx from running command: /opt/mesosphere/packages/mesos--aaedd03eee0d57f5c0d49c
Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/94af100c-4dc2-416d-b6d7-eec0d947a1a6-S11","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"executable":false,"extract":true,"value":"file:\/\/\/home\/core\/docker.tar.gz"}}],"sandbox_directory":"\/var\/lib\/mesos\/slave\/slaves\/94af100c-4dc2-416d-b6d7-eec0d947a1a6-S11\/frameworks\/94af100c-4dc2-416...
Fetching URI 'file:///home/core/docker.tar.gz'
Fetching directly into the sandbox directory
Fetching URI 'file:///home/core/docker.tar.gz'
Copied resource '/home/core/docker.tar.gz' to '/var/lib/mesos/slave/slaves/94af100c-4dc2-416d-b6d7-eec0d947a1a6-S11/frameworks/94af100c-4dc2-416d-b6d7-eec0d947a1a6-0
Failed to obtain the IP address for 'digitalocean-dcos-agent-20'; the DNS service may not be able to resolve it: Name or service not known
End fetcher log for container 123-abc-xxx
Failed to run mesos-fetcher: Failed to fetch all URIs for container '123-abc-xxx' with exit status: 256
You are missing the extract instruction:
"fetch":[
{
"uri":"file:///home/core/docker.tar.gz",
"extract":true
}
]
I am following this tutorial at https://gettech1.wordpress.com/2016/05/26/setting-up-kubernetes-cluster-on-ubuntu-14-04-lts/ to setup kubernet multi node with 2 minions and 1 master node on remote ubuntu machines, after following all the steps it goes OK. But when I am trying to run the ./kube-up.sh bash file. It returns the following errors
ubuntu#ip-XXX-YYY-ZZZ-AAA:~/kubernetes/cluster
$ ./kube-up.sh
Starting cluster in us-central1-b using provider gce ... calling
verify-prereqs Can't find gcloud in PATH, please fix and retry. The
Google Cloud SDK can be downloaded from
https://cloud.google.com/sdk/.
Edit : I have fixed above issue after exporting different environment variables like
$ export KUBE_VERSION=2.2.1
$ export FLANNEL_VERSION=0.5.5
$ export ETCD_VERSION=1.1.8
but after that it is generating this issue
kubernet gzip: stdin: not in gzip format tar: Child returned status 1
tar: Error is not recoverable: exiting now
The command you should be executing is KUBERNETES_PROVIDER=ubuntu ./kube-up.sh
Without setting that environment variable kube-up.sh tries to deploy VMs on Google Compute Engine and to do so it needs the gcloud binary that you don't have installed.