I have set up OSRM (Open Source Routing Machine) on a Windows 11 machine with WSL (Ubuntu 20.04.1 LTS) and Docker installed.
docker pull osrm/osrm-backend
Using default tag: latest
latest: Pulling from osrm/osrm-backend
aed007321795: Pull complete
595e916ac67a: Pull complete
69f309e04f11: Pull complete
e73ad3e5bcf3: Pull complete
4c20f809069e: Pull complete
Digest: sha256:af5d4a83fb90086a43b1ae2ca22872e6768766ad5fcbb07a29ff90ec644ee409
Status: Downloaded newer image for osrm/osrm-backend:latest
docker.io/osrm/osrm-backend:latest
docker run -t -v /mnt/c/docker:/data osrm/osrm-backend osrm-extract -p /opt/bicycle.lua /data/south_west_highways.osm.pbf
docker run -t -v /mnt/c/docker:/data osrm/osrm-backend osrm-partition /data/south_west_highways.osrm
docker run -t -v /mnt/c/docker:/data osrm/osrm-backend osrm-customize /data/south_west_highways.osrm
docker run --name osrm -t -i -p 5000:5000 -v /mnt/c/docker:/data osrm/osrm-backend osrm-routed --algorithm mld /data/south_west_highways.osrm
It runs correctly and I'm able match bike GPX tracks to OSM ways.
I would now like to modify bicycle.lua to allow routes on 'main' roads - but I can't find /opt/bicycle.lua ?
https://github.com/Project-OSRM/osrm-backend/blob/master/profiles/bicycle.lua
Where should this file be located on my system?
I've found the answer! In the command
docker run -t -v /mnt/c/docker:/data osrm/osrm-backend osrm-extract -p /opt/bicycle.lua /data/south_west_highways.osm.pbf
just point to the new file that you want to refer to e.g.
docker run -t -v /mnt/c/docker:/data osrm/osrm-backend osrm-extract -p /my_location/my_lua_file.lua /data/south_west_highways.osm.pbf
I am trying to setup "Deploying Hortonworks Sandbox on Docker" on MAC M1
Official Cloudera Installation directory
https://www.cloudera.com/tutorials/sandbox-deployment-and-install-guide/3.html
Reference installation
https://javamana.com/2021/02/20210206191356735v.html
below is the configuration
Ram - 16gb
Rosetta2 Installed
Docker installed Mac M1
Docker hub image link
https://hub.docker.com/r/hortonworks/sandbox-hdp/tags
command used
docker pull hortonworks/sandbox-hdp:3.0.1 --platform linux/amd64
SCRIPT
#!/usr/bin/env sh
#This script downloads HDP sandbox along with their proxy docker container
set -x
# CAN EDIT THESE VALUES
registry="hortonworks"
name="sandbox-hdp"
version="3.0.1"
proxyName="sandbox-proxy"
proxyVersion="1.0"
flavor="hdp"
# NO EDITS BEYOND THIS LINE
# housekeeping
echo $flavor > sandbox-flavor
# create necessary folders for nginx and copy over our rule generation script there
mkdir -p sandbox/proxy/conf.d
mkdir -p sandbox/proxy/conf.stream.d
# pull and tag the sandbox and the proxy container
docker pull "$registry/$name:$version"
docker pull "$registry/$proxyName:$proxyVersion"
# start the docker container and proxy
if [ "$flavor" == "hdf" ]; then
hostname="sandbox-hdf.hortonworks.com"
elif [ "$flavor" == "hdp" ]; then
hostname="sandbox-hdp.hortonworks.com"
fi
version=$(docker images | grep $registry/$name | awk '{print $2}');
# Create cda docker network
docker network create cda 2>/dev/null
# Deploy the sandbox into the cda docker network
docker run --privileged --name $name -h $hostname --network=cda --network-alias=$hostname -d "$registry/$name:$version"
echo " Remove existing postgres run files. Please wait"
sleep 2
docker exec -t "$name" sh -c "rm -rf /var/run/postgresql/*; systemctl restart postgresql-9.6.service;"
#Deploy the proxy container.
sed 's/sandbox-hdp-security/sandbox-hdp/g' assets/generate-proxy-deploy-script.sh > assets/generate-proxy-deploy-script.sh.new
mv -f assets/generate-proxy-deploy-script.sh.new assets/generate-proxy-deploy-script.sh
chmod +x assets/generate-proxy-deploy-script.sh
assets/generate-proxy-deploy-script.sh 2>/dev/null
#check to see if it's windows
if uname | grep MINGW; then
sed -i -e 's/\( \/[a-z]\)/\U\1:/g' sandbox/proxy/proxy-deploy.sh
fi
chmod +x sandbox/proxy/proxy-deploy.sh 2>/dev/null
sandbox/proxy/proxy-deploy.sh
LOG
➜ Documents sh docker-deploy-hdp30.sh
+ registry=hortonworks
+ name=sandbox-hdp
+ version=3.0.1
+ proxyName=sandbox-proxy
+ proxyVersion=1.0
+ flavor=hdp
+ echo hdp
+ mkdir -p sandbox/proxy/conf.d
+ mkdir -p sandbox/proxy/conf.stream.d
+ docker pull hortonworks/sandbox-hdp:3.0.1
3.0.1: Pulling from hortonworks/sandbox-hdp
70799bbf2226: Pull complete
40963917cdad: Pull complete
3fe9adbb8d7e: Pull complete
ee3ec4e8cb3d: Pull complete
7ea5917732c0: Pull complete
2d951411620c: Pull complete
f4c5e354e7fd: Pull complete
22ffa6ef360f: Pull complete
2060aa0f3751: Pull complete
ca01ba34744d: Pull complete
83326dded077: Pull complete
eb3d71b90b73: Pull complete
bdd1cab41c81: Pull complete
500cc770c4bd: Pull complete
0cb1decd5474: Pull complete
b9591f4b6855: Pull complete
f28e56086127: Pull complete
e7de4e7d0bca: Pull complete
ec77967d2166: Pull complete
4fdcae170114: Pull complete
6347f5df8ffc: Pull complete
6a6ecc232709: Pull complete
ea845898ff50: Pull complete
02135573b1bf: Pull complete
cb0176867cd8: Pull complete
3c08321268fd: Pull complete
82e82a97c465: Pull complete
8aaaa48ed101: Pull complete
74b321ac2ac5: Pull complete
569da02c0a66: Pull complete
af40820407ef: Pull complete
Digest: sha256:7b767af7b42030fb1dd0f672b801199241e6bef1258e3ce57361edb779d95921
Status: Downloaded newer image for hortonworks/sandbox-hdp:3.0.1
docker.io/hortonworks/sandbox-hdp:3.0.1
+ docker pull hortonworks/sandbox-proxy:1.0
1.0: Pulling from hortonworks/sandbox-proxy
951bdea65c93: Pull complete
4b9047c5fbbb: Pull complete
773156407aae: Pull complete
d8524176841d: Pull complete
Digest: sha256:42e4cfbcbb76af07e5d8f47a183a0d4105e65a1e7ef39fe37ab746e8b2523e9e
Status: Downloaded newer image for hortonworks/sandbox-proxy:1.0
docker.io/hortonworks/sandbox-proxy:1.0
+ '[' hdp == hdf ']'
+ '[' hdp == hdp ']'
+ hostname=sandbox-hdp.hortonworks.com
++ docker images
++ grep hortonworks/sandbox-hdp
++ awk '{print $2}'
+ version=3.0.1
+ docker network create cda
ce52e1d04aa49c2da70dfd7405b95fc2e2184fed9625feaea0ddafb9ab5c361d
+ docker run --privileged --name sandbox-hdp -h sandbox-hdp.hortonworks.com --network=cda --network-alias=sandbox-hdp.hortonworks.com -d hortonworks/sandbox-hdp:3.0.1
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
3ab0f0281918fc10d008c93aa7a844302e974b696e00788df76cde779fdec46d
+ echo ' Remove existing postgres run files. Please wait'
Remove existing postgres run files. Please wait
+ sleep 2
+ docker exec -t sandbox-hdp sh -c 'rm -rf /var/run/postgresql/*; systemctl restart postgresql-9.6.service;'
Failed to get D-Bus connection: No such file or directory
+ sed s/sandbox-hdp-security/sandbox-hdp/g assets/generate-proxy-deploy-script.sh
+ mv -f assets/generate-proxy-deploy-script.sh.new assets/generate-proxy-deploy-script.sh
+ chmod +x assets/generate-proxy-deploy-script.sh
+ assets/generate-proxy-deploy-script.sh
+ uname
+ grep MINGW
+ chmod +x sandbox/proxy/proxy-deploy.sh
+ sandbox/proxy/proxy-deploy.sh
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
fbe0c31a680f51db9c91367df65493ea1cf874cc900466dd943824dc19bd9e74
➜ Documents
how to resolve the error
I have tried performing some tweaks
adding the below line
--platform linux/amd64
still i encounter the same error
what is the work around to fix this issue ?
Tweaks i have tried
before running the script i have tried adjusting the script by adding
# CAN EDIT THESE VALUES
registry="hortonworks"
name="sandbox-hdp"
version="3.0.1"
proxyName="sandbox-proxy"
proxyVersion="1.0"
flavor="hdp"
arch = arm64
platform = linux/arm64
Deploy the sandbox into the cda docker network
docker run --privileged --name $name -h $hostname --network=cda --network-alias=$hostname -d "$registry/$name:$version" $platform
option 1
arch = linux/arm64
option 2
platform = linux/arm64
First validate if Rosetta2 is configured correctly. You can test with alpine - very small linux docker container. Run:
docker run --rm -ti --platform linux/amd64 alpine:latest uname -a
The output should be something like this:
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
Digest: sha256:d6d0a0eb4d40ef96f2310ead734848b9c819bb97c9d846385c4aca1767186cd4
Status: Downloaded newer image for alpine:latest
Linux 35d1145cff16 5.11.0-49-generic #55-Ubuntu SMP Wed Jan 12 17:36:34 UTC 2022 x86_64 Linux
Now if that works you can modify scripts to run HDP. There are actually 2 scripts to modify.
First
Open docker-deploy-hdp30.sh in a editor and modify following line:
docker run --privileged --name $name -h $hostname --network=cda --network-alias=$hostname -d "$registry/$name:$version"
To:
docker run --platform linux/amd64 --privileged --name $name -h $hostname --network=cda --network-alias=$hostname -d "$registry/$name:$version"
Second
Open assets/generate-proxy-deploy-script.sh in an editor & scroll to the bottom. There's docker run in line 204, add platform switch after it like this (the backslashes at the end of line are crucial!):
docker run --name sandbox-proxy --network=cda \\
--platform linux/amd64 \\
-v $absPath/assets/nginx.conf:/etc/nginx/nginx.conf \\
Now you are ready to run the first script (it triggers the other one).
After it executes docker ps should return 2 containers: sandbox-proxy and sandbox-hdp.
I'm running Docker on my Synology DS918+ and I'm looking to run the jetbrains/teamcity-server container.
When I run the following command:
sudo docker run teamcity-server -v /volume2/docker/teamcity-server/datadir/:/data/teamcity_server/datadir -v /volume2/docker/teamcity-server/logs/:/opt/teamcity/logs -p 8001:8001 jetbrains/teamcity-server:latest
I get a result back that it can't find the image locally:
Unable to find image 'teamcity-server:latest' locally
docker: Error response from daemon: pull access denied for teamcity-server, repository does not exist or may require 'docker login'.
See 'docker run --help'.
I believe that I have confirmed that I have the image installed locally because when I run the command:
sudo docker images
I get the following result:
REPOSITORY TAG IMAGE ID CREATED SIZE
jetbrains/teamcity-server latest bfe4a2f841c1 2 weeks ago 2.2GB
The correct way is to use --name before the container name (teamcity-server). Otherwise, docker will think that it is the image that you are trying to pull and error out.
sudo docker run --name teamcity-server -v /volume2/docker/teamcity-server/datadir/:/data/teamcity_server/datadir -v /volume2/docker/teamcity-server/logs/:/opt/teamcity/logs -p 8001:8001 jetbrains/teamcity-server:latest
I run the following docker command. I don't understand why it is failed. Could you show me how to debug this? Thanks.
$ docker run -v `pwd`:/share -ti --name aerospike-aql --rm aerospike/aerospike-tools aql --host 192.168.1.191 --no-config-file
Unable to find image 'aerospike/aerospike-tools:latest' locally
latest: Pulling from aerospike/aerospike-tools
6ec7b7d162b2: Pull complete
177617b11d13: Pull complete
10273812b9e3: Pull complete
ac553cdb1df6: Pull complete
d633ea8cb425: Pull complete
1f91817a9ef3: Pull complete
403620a9a728: Pull complete
3a80741c7bbc: Pull complete
Digest: sha256:109801d7e8440dcf53461b13b55eaa96c1f86482209691285a34c6bb2fee4e1d
Status: Downloaded newer image for aerospike/aerospike-tools:latest
Seed: 192.168.1.191
User: None
Config File: None
2021-01-16 02:46:44 WARN Failed to connect to seed 192.168.1.191 3000. AEROSPIKE_ERR_CONNECTION Socket write error: 111, 192.168.1.191:3000
Error -10: Failed to connect
To add to Micah's comment I can suggest running again the server image, and then the command below that is not using the hardcoded host IP 192.168.1.191 but rather pulls it from the settings:
$ docker run --rm -tid --name aerospike -p 3000:3000 -p 3001:3001 -p 3002:3002 -p 3003:3003 aerospike/aerospike-server
$ docker run -v `pwd`:/share -ti --name aerospike-aql --rm aerospike/aerospike-tools aql --host $(docker inspect -f '{{.NetworkSettings.IPAddress }}' aerospike) --no-config-file
I'm unable to run the eclipse/che local image. i.e., from the eclipse/che source code in my pc.
Here are the steps that i tried:
Clone the eclipse/che src code into //d/checmd3/che.
git clone https://github.com/eclipse/che.git &
git checkout tags/7.0.0-beta-2.0
Build it
cd assembly/assembly-main
mvn clean install
...A new assembly is placed in:
cd che/assembly/assembly-main/target/eclipse-che-/eclipse-che-
Run it in docker
docker run -it --rm -v //var/run/docker.sock://var/run/docker.sock -v //d/checmd3/che/assembly/assembly-main/target/eclipse-che-7.0.0-beta-2.0/eclipse-che-7.0.0-beta-2.0
:/che -e CHE_ASSEMBLY=//d/checmd3/che/assembly/assembly-main/target/eclipse-che-7.0.0-beta-2.0/eclipse-che-7.0.0-beta-2.0 -v //d/checmd3/che/tmp:/data eclipse/che start
After step #3 above, the following message was shown:
Unable to find image 'eclipse/che:7.0.0-beta-2.0' locally
7.0.0-beta-2.0: Pulling from eclipse/che
I believe that docker is not trying to run the image from my local pc ?
I'm not sure if step #3 above is the issue or not. Please help me in running the image from the src code cloned in my pc.
(reference : https://github.com/eclipse/che/wiki/Development-Workflow)
I'm unfamiliar with Eclipse Che but, it appears you can simply run their image(s) on your machine assuming that you've Docker installed.
Start by creating a local data directory, perhaps:
mkdir -p ${PWD}/che/data
Then:
docker run
--interactive \
--tty \
--rm \
--net=host \
--volume=/var/run/docker.sock:/var/run/docker.sock \
--volume=${PWD}/che/data:/data \
eclipse/che:nightly start
https://www.eclipse.org/che/docs/che-6/docker-single-user.html
You may not need the --net=host flag
You should then be able to then access the tool:
http://localhost:8080
NB
Your steps 1&2 (git clone... and mvn clean install) are probably redundant. These are likely the commands to build the Docker image. But, since the image already exists in dockerhub, you need not follow these steps.
Try this docker command:
docker run -it --rm \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /d/checmd3/che/assembly/assembly-main/target/eclipse-che-7.0.0-beta-2.0/eclipse-che-7.0.0-beta-2.0:/che \
-e CHE_ASSEMBLY='/d/checmd3/che/assembly/assembly-main/target/eclipse-che-7.0.0-beta-2.0/eclipse-che-7.0.0-beta-2.0' \
-v /d/checmd3/che/tmp:/data \
eclipse/che start
ABOVE COMMAND IS WORKING
INFO: (che init): CHE_VERSION=7.0.0-beta-2.0
INFO: (che init): CHE_CONFIG=/d/checmd3/che/tmp
INFO: (che init): CHE_INSTANCE=/d/checmd3/che/tmp/instance
INFO: (che config): Generating che configuration...
INFO: (che config): Customizing docker-compose for running in a container
INFO: (che start): Preflight checks
mem (1.5 GiB): [OK]
disk (100 MB): [OK]
port 8080 (http): [AVAILABLE]
conn (browser => ws): [OK]
conn (server => ws): [OK]
INFO: (che start): Starting containers...
INFO: (che start): Services booting...
INFO: (che start): Server logs at "docker logs -f che"
INFO: (che start): Booted and reachable
INFO: (che start): Ver: 7.0.0-beta-2.0
INFO: (che start): Use: http://172.26.10.112:8080
INFO: (che start): API: http://172.26.10.112:8080/swagger
Thank you. I tried the following docker command:
docker run -it --rm -v //var/run/docker.sock://var/run/docker.sock -v //d/checmd3/che/assembly/assembly-main/target/eclipse-che-7.0.0-beta-2.0/eclipse-che-7.0.0-beta-2.0
:/che -e CHE_ASSEMBLY='//d/checmd3/che/assembly/assembly-main/target/eclipse-che-7.0.0-beta-2.0/eclipse-che-7.0.0-beta-2.0' -v //d/checmd3/che/tmp:/data eclipse/che start
but, still the message shown is:
Unable to find image 'eclipse/che:latest' locally
latest: Pulling from eclipse/che
(docker is still not using the source code built locally in my pc)
If you want to run your custom Che binaries, the syntax you use is the correct one. Che CLI will pull default image anyway, but your binaries will be mounted into a container. Will that work for you?
If you want to run your own image for some reason, you may simply pass the following env to CLI.
-e IMAGE_CHE=myRegistry/myRepo:myTag
You could try first to prevent Eclipse Che from pulling the image from the Docker hub by setting: CHE_DOCKER_ALWAYS__PULL__IMAGE=false in your che.env config file.
If it doesn't help then I think you need to install and run a local Docker registry, then push your Eclipse Che image that you have built locally to that registry.
docker run -d -p 5000:5000 --name registry registry:2
docker image tag che:7.0.0-beta-2.0 eclipse/che:7.0.0-beta-2.0
docker push eclipse/che:7.0.0-beta-2.0
Then you can pull and run your image using your Docker run command.
You can stop the registry by:
docker container stop registry && docker container rm -v registry
There was an error when i ran the command you suggested:
$ docker run --interactive --tty --rm --net=host --volume=//var/run/docker.sock://var/run/docker.sock --volume=/${PWD}/che/data:/data eclipse/che:nightly start
The following is the log:
Unable to find image 'eclipse/che:latest' locally
latest: Pulling from eclipse/che
d6a5679aa3cf: Pull complete
cc87d3e420c3: Pull complete
afef80a99ec8: Pull complete
d4be2f254bed: Pull complete
3e449e5a7821: Pull complete
5b621c46cfe0: Pull complete
ecdf06277042: Pull complete
dcbe7590a8ca: Pull complete
Digest: sha256:bd853bd40a4fafe73153dda478f1191d3d29447f3d110584933a5fb22e8cb199
Status: Downloaded newer image for eclipse/che:latest
Error: No such image or container: linuxkit-00155d19290d
I didnt get the linuxkit error yesterday :-(