I want to pass my GOOGLE_APPLICATION_CREDENTIALS environmental variable when I run mlflow run using a Docker container:
This is my current docker run when using mlflow run:
Running command 'docker run --rm -e MLFLOW_RUN_ID=f18667e37ecb486cac4631cbaf279903 -e MLFLOW_TRACKING_URI=http://3.1.1.11:5000 -e MLFLOW_EXPERIMENT_ID=0 mlflow_gcp:33156ee python -m trainer.task --job-dir /tmp/ \
--num-epochs 10 \
--train-steps 1000 \
--eval-steps 1 \
--train-files gs://cloud-samples-data/ml-engine/census/data/adult.data.csv \
--eval-files gs://cloud-samples-data/ml-engine/census/data/adult.test.csv \
--batch-size 128
This is how I would normally pass it:
docker run \
-p 9090:${PORT} \
-e PORT=${PORT} \
-e GOOGLE_APPLICATION_CREDENTIALS=/tmp/keys/[FILE_NAME].json
What is the best way to option to pass this value to mlflow? I'm writing files in GCS and Docker requires access to GCP.
MLproject contents
name: mlflow_gcp
docker_env:
image: mlflow-gcp-example
entry_points:
main:
parameters:
job_dir:
type: string
default: '/tmp/'
num_epochs:
type: int
default: 10
train_steps:
type: int
default: 1000
eval_steps:
type: int
default: 1
batch_size:
type: int
default: 64
train_files:
type: string
default: 'gs://cloud-samples-data/ml-engine/census/data/adult.data.csv'
eval_files:
type: string
default: 'gs://cloud-samples-data/ml-engine/census/data/adult.test.csv'
mlflow_tracking_uri:
type: uri
default: ''
command: |
python -m trainer.task --job-dir {job_dir} \
--num-epochs {num_epochs} \
--train-steps {train_steps} \
--eval-steps {eval_steps} \
--train-files {train_files} \
--eval-files {eval_files} \
--batch-size {batch_size} \
--mlflow-tracking-uri {mlflow_tracking_uri}
I already tried in Python file and fails since Docker has no access to local file system:
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/Users/user/key.json"
Related
I want to run a temporary docker and execute several command in it:
.PHONY : Test
Test:
#echo Starting Docker container
/home/pagl_home/bin/SI.docker_17cy/bin/pagl-build-env run -b '$(PRJ_BUILD_TREE)' -w '$(PRJ_WORK_DIR)' --objdir '' -c 'rm -rf build_results; \
make clean_brutal; \
$(PRJ_MISRA_CMD)/cov-build --dir build_results --encoding UTF-8 make SUB_PRODUCT=$(SUB_PRODUCT); \
$(if $(COVERITY_MISRA_2012), \
$(PRJ_test_CMD)/cov-analyze --dir build_results --disable-default --coding-standard-config /home/XX/XX-linux64-2018.06/config/coding-standards/XX/XX-all.config --paths 100000 --tu-pattern "file('.*\.c$\')"; \
$(PRJ_test_CMD)/cov-format-errors --dir build_results --html-output build_results/results/HTML_2012, );'
but I het the following error:
Illegal character in pattern: .
[ERROR] No results found.
I want to know how to run command seperately in a running container?
like:
.PHONY : Test
Test:
#echo Starting Docker container
/home/pagl_home/bin/SI.docker_17cy/bin/pagl-build-env run -b '$(PRJ_BUILD_TREE)' -w '$(PRJ_WORK_DIR)' --objdir '' -c 'rm -rf build_results; \
/home/pagl_home/bin/SI.docker_17cy/bin/pagl-build-env run -c 'make clean_brutal';
/home/pagl_home/bin/SI.docker_17cy/bin/pagl-build-env run -c '$(PRJ_MISRA_CMD)/cov-build --dir build_results --encoding UTF-8 make SUB_PRODUCT=$(SUB_PRODUCT)'
/home/pagl_home/bin/SI.docker_17cy/bin/pagl-build-env run -c '$(if $(FFFF), \
/home/pagl_home/bin/SI.docker_17cy/bin/pagl-build-env run $(PRJ_test_CMD)/cov-analyze --dir build_results --disable-default --coding-standard-config /home/XX/XX-linux64-2018.06/config/coding-standards/XX/XX-all.config --paths 100000 --tu-pattern "file('.*\.c$\')"'
/home/pagl_home/bin/SI.docker_17cy/bin/pagl-build-env run -c '$(PRJ_test_CMD)/cov-format-errors --dir build_results --html-output'build_results/results/HTML_2012, );'
I know this doesn't work but to give an idea what I want to do
I'm trying to set up Couchbase as part of a collection of servers using Docker-Compose. The sole purpose of this is for local application development.
The problem is that, once set up, I'm unable to write to the database. Insert and Upsert operations give me a DurabilityImpossibleError.
Docker compose file:
version: '3.4'
services:
...
couchbase-db:
image: couchbase/server
volumes:
- ./docker-data/couchbase/node3:/opt/couchbase/var
- ./provision/couchbase:/opt/startup/
ports:
- 8091:8091
- 8092:8092
- 8093:8093
- 8094:8094
- 11210:11210
The startup bash script, run after building, is an attempt to perform database setup without requiring a manual step:
#!/bin/bash
# Enables job control
set -m
# Enables error propagation
set -e
# Run the server and send it to the background
/entrypoint.sh couchbase-server &
# Check if couchbase server is up
check_db() {
curl --silent http://${1}:8091/pools > /dev/null
echo $?
}
# Variable used in echo
i=1
# Echo with
log() {
echo "[$i] [$(date +"%T")] $#"
i=`expr $i + 1`
}
# Wait until main server is ready
until [[ $(check_db 127.0.0.1) = 0 ]]; do
>&2 log "Waiting for Couchbase Server to be available ..."
sleep 1
done
couchbase-cli cluster-init -c localhost:8091 \
--cluster-username Administrator --cluster-password password \
--cluster-password password --services data,index,query --cluster-ramsize 512 \
--cluster-index-ramsize 256 || true
couchbase-cli setting-cluster -c localhost:8091 -u Administrator -p password \
--cluster-username Administrator --cluster-password password \
--cluster-password password --cluster-ramsize 512 \
--cluster-index-ramsize 256;
couchbase-cli setting-cluster -c localhost:8091 \
-u Administrator -p password --cluster-username Administrator \
--cluster-password password --cluster-ramsize 512 --cluster-index-ramsize 256;
curl -v POST http://localhost:8091/pools/default/buckets \
-u Administrator:password \
-d name=organisations \
-d bucketType=couchbase \
-d ramQuotaMB=512 \
-d durabilityMinLevel=majorityAndPersistActive
curl -v -X POST -u Administrator:password \
http://localhost:8091/settings/indexes \
-d indexerThreads=4 \
-d logLevel=verbose \
-d maxRollbackPoints=10 \
-d storageMode=plasma \
-d memorySnapshotInterval=150 \
-d stableSnapshotInterval=40000
# Need to wait until query service is ready to process N1QL queries
echo "$(date +"%T") Waiting ........."
sleep 20
# Create bucket1 indexes
echo "$(date +"%T") Create bucket1 indexes ........."
cbq -u Administrator -p password -s "CREATE PRIMARY INDEX idx_primary ON \`organisations\`;"
cbq -u Administrator -p password -s "CREATE INDEX idx_type ON \`organisations\`(_type);"
If I try to add a document via the web interface, I get:
Errors from server: ["Unexpected server error, request logged."]
If I try to add a document via the JavaScript SDK, I get:
DurabilityImpossibleError durability impossible
details:
{
name: 'DurabilityImpossibleError',
cause: LibcouchbaseError: libcouchbase error 308
at Object.translateCppError
(/app/node_modules/couchbase/dist/bindingutilities.js:174:21)
at /app/node_modules/couchbase/dist/connection.js:245:54 {
code: 308
},
context: KeyValueErrorContext {
status_code: 0,
opaque: 0,
cas: CbCas {
'0': <Buffer 00 00 00 00 00 00 00 00>
},
key: '22738bd4-7972-4370-85a3-71399d96ef05',
bucket: '',
collection: '',
scope: '',
context: '',
ref: ''
}
}
I've also attempted to send the following settings with the insert/upsert, to no effect:
insertOptions: {
durabilityLevel: 0,
durabilityPersistTo: 1,
durabilityReplicateTo: 0,
timeout: 5000,
},
My most recent attempt at a fix was to build a cluster of 3 nodes within docker-compose, and call the API commands to "add server" as part of a build script. However, "add server" takes a static IP, so the second time I run the servers, the IPs change and the database becomes unresponsive. I do get a functioning database on that first run though.
I'm looking for either a fix for a single node system (or an idea of where I'm going wrong), or a way of getting a cluster working in Docker-Compose after a down/up cycle. Anything that will give me a stable environment to develop in.
Thanks!
The bucket is created with -d durabilityMinLevel=majorityAndPersistActive, By default bucket enables replica with 1.
For single node cluster you will not have enough data nodes to satisfy durability (https://docs.couchbase.com/server/current/learn/data/durability.html). You can disable replica via UI and rebalance to take affect or change the bucket setting not include Minimum durability.
I have no idea about the 3 node docker compose error.
Hi I use Kafka connect docker container image confluentinc/cp-kafka-connect 5.5.3 and everything was running fine when using follwing Parameters
...
-e "CONNECT_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter" \
-e "CONNECT_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter" \
-e "CONNECT_INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter" \
-e "CONNECT_INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter" \
...
Now we introduced Schema Registry and decided to go with JsonSchemaConverter for now and not avro. I changed follwoing (INTERNAL stays as it is for now)
...
-e "CONNECT_KEY_CONVERTER=io.confluent.connect.json.JsonSchemaConverter" \
-e "CONNECT_VALUE_CONVERTER=io.confluent.connect.json.JsonSchemaConverter" \
-e "CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL=http://<schemaregsirty_url>:8081" \
-e "CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL=http://<schemaregsirty_url>:8081" \
...
Following Error appeared:
[2021-02-04 09:24:14,637] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed)org.apache.kafka.common.config.ConfigException: Invalid value io.confluent.connect.json.JsonSchemaConverter for configuration key.converter: Class io.confluent.connect.json.JsonSchemaConverter could not be found.
at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:727)
at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:473)
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:466)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:108)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:129)
at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:374)
at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:316)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:93)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
It seems the converter is not available here by default. Do I have to install JsonSchemaConverter? I thought it comes by default?
i am trying to download sentinel 2 data by running a docker container. I just want to download a single imagery for testing and so i am passing the filename as an environment variable but when i execute the docker run, it finds all relevant imageries and starts downloading all of them which is not what i want.
here is the command for executing the docker run statement
sudo docker run --rm -v $(pwd):/out_data \
> -e scihub_username=test \
> -e scihub_password=test \
> -e producttype=S2MSI2A \
> -e platformname=Sentinel-2 \
> -e files=S2A_MSIL2A_20190612T104031_N0212_R008_T31UGT_20190612T133140 \
> -e days_back=7 \
> -e footprint="POLYGON((5.8664000 50.3276000,9.4623000 50.3276000,9.4623000 52.5325000,5.8664000 52.5325000,5.8664000 50.3276000))" \
> -e max_cloud_cover_percentage=10 \
> -e start_date=2018-01-01T00:00:00.000Z \
> -e end_date=2019-01-01T00:00:00.000Z \
> -e BASE_URL=localohost:8081/swagger-ui.html \
> -e JOB_ID=8c04ee18-92e3-4739-b460-a78b0822a497 \
> ingestion
i have also defined the vairables in an ingestion.py file which is executing at docker build. it looks like this :
import os
import shutil
import logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(name)s - %(message)s')
logger = logging.getLogger("ingestion")
import requests
import datahub
scihub_username = os.environ["scihub_username"]
scihub_password = os.environ["scihub_password"]
result_url = "http://" + os.environ["BASE_URL"] + "/jobs/" + os.environ["JOB_ID"] + "/results"
logger.info("Searching the Copernicus Open Access Hub")
scenes = datahub.search(username=scihub_username,
password=scihub_password,
producttype=os.getenv("producttype"),
platformname=os.getenv("platformname"),
files=os.getenv("filename")
days_back=os.getenv("days_back", 2),
footprint=os.getenv("footprint"),
max_cloud_cover_percentage=os.getenv("max_cloud_cover_percentage"),
start_date = os.getenv("start_date"),
end_date = os.getenv("end_date"))
logger.info("Found {} relevant scenes".format(len(scenes)))
job_results = []
for scene in scenes:
# do not donwload a scene that has already been ingested
if os.path.exists(os.path.join("/out_data", scene["title"]+".SAFE")):
logger.info("The scene {} already exists in /out_data and will not be downloaded again.".format(scene["title"]))
filename = scene["title"]+".SAFE"
else:
logger.info("Starting the download of scene {}".format(scene["title"]))
filename = datahub.download(scene, "/tmp", scihub_username, scihub_password, unpack=True)
logger.info("The download was successful.")
shutil.move(filename, "/out_data")
result_message = {"description": "test",
"type": "Raster",
"format": "SAFE",
"filename": os.path.basename(filename)}
job_results.append(result_message)
res = requests.put(result_url, json=job_results, timeout=60)
res.raise_for_status()
that's how search looks like in datahub below:
def search(username, password, producttype=None, platformname=None, days_back=2, footprint=None, max_cloud_cover_percentage=None, start_date=None, end_date=None):
I'm trying to execute this normal tf_serving command (which work correctly) with docker version of tf_serving. I'm not sure why it's not working.. Any suggestion? I'm new to Docker!
Normal tf_serving command:
tensorflow_model_server \
--model_config_file=/opt/tf_serving/model_config.conf \
--port=6006
here is what my model_config.conf looks like:
model_config_list: {
config: {
name: "model_1",
base_path: "/opt/tf_serving/model_1",
model_platform: "tensorflow",
},
config: {
name: "model_2",
base_path: "/opt/tf_serving/model_2",
model_platform: "tensorflow",
},
}
Docker version of command that I'm trying but not working:
docker run --runtime=nvidia \
-p 6006:6006 \
--mount type=bind,source=/opt/tf_serving/model_1,target=/models/model_1/ \
--mount type=bind,source=/opt/tf_serving/model_2,target=/models/model_2/ \
--mount type=bind,source=/opt/tf_serving/model_config.conf,target=/config/model_config.conf \
-t tensorflow/serving:latest-gpu --model_config_file=/config/model_config.conf
Error:
2019-04-13 19:41:00.838340: E tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:369] FileSystemStoragePathSource encountered a file-system access error: Could not find base path /opt/tf_serving/model_1 for servable model_1
Found the issue! You have to change the models path in model_config.conf as follow, and the above docker command will work and load both models!
model_config_list: {
config: {
name: "model_1",
base_path: "/models/model_1",
model_platform: "tensorflow",
},
config: {
name: "model_2",
base_path: "/models/model_2",
model_platform: "tensorflow",
},
}
EDIT: corrected typo on base_path for model_2.