i am trying to download sentinel 2 data by running a docker container. I just want to download a single imagery for testing and so i am passing the filename as an environment variable but when i execute the docker run, it finds all relevant imageries and starts downloading all of them which is not what i want.
here is the command for executing the docker run statement
sudo docker run --rm -v $(pwd):/out_data \
> -e scihub_username=test \
> -e scihub_password=test \
> -e producttype=S2MSI2A \
> -e platformname=Sentinel-2 \
> -e files=S2A_MSIL2A_20190612T104031_N0212_R008_T31UGT_20190612T133140 \
> -e days_back=7 \
> -e footprint="POLYGON((5.8664000 50.3276000,9.4623000 50.3276000,9.4623000 52.5325000,5.8664000 52.5325000,5.8664000 50.3276000))" \
> -e max_cloud_cover_percentage=10 \
> -e start_date=2018-01-01T00:00:00.000Z \
> -e end_date=2019-01-01T00:00:00.000Z \
> -e BASE_URL=localohost:8081/swagger-ui.html \
> -e JOB_ID=8c04ee18-92e3-4739-b460-a78b0822a497 \
> ingestion
i have also defined the vairables in an ingestion.py file which is executing at docker build. it looks like this :
import os
import shutil
import logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(name)s - %(message)s')
logger = logging.getLogger("ingestion")
import requests
import datahub
scihub_username = os.environ["scihub_username"]
scihub_password = os.environ["scihub_password"]
result_url = "http://" + os.environ["BASE_URL"] + "/jobs/" + os.environ["JOB_ID"] + "/results"
logger.info("Searching the Copernicus Open Access Hub")
scenes = datahub.search(username=scihub_username,
password=scihub_password,
producttype=os.getenv("producttype"),
platformname=os.getenv("platformname"),
files=os.getenv("filename")
days_back=os.getenv("days_back", 2),
footprint=os.getenv("footprint"),
max_cloud_cover_percentage=os.getenv("max_cloud_cover_percentage"),
start_date = os.getenv("start_date"),
end_date = os.getenv("end_date"))
logger.info("Found {} relevant scenes".format(len(scenes)))
job_results = []
for scene in scenes:
# do not donwload a scene that has already been ingested
if os.path.exists(os.path.join("/out_data", scene["title"]+".SAFE")):
logger.info("The scene {} already exists in /out_data and will not be downloaded again.".format(scene["title"]))
filename = scene["title"]+".SAFE"
else:
logger.info("Starting the download of scene {}".format(scene["title"]))
filename = datahub.download(scene, "/tmp", scihub_username, scihub_password, unpack=True)
logger.info("The download was successful.")
shutil.move(filename, "/out_data")
result_message = {"description": "test",
"type": "Raster",
"format": "SAFE",
"filename": os.path.basename(filename)}
job_results.append(result_message)
res = requests.put(result_url, json=job_results, timeout=60)
res.raise_for_status()
that's how search looks like in datahub below:
def search(username, password, producttype=None, platformname=None, days_back=2, footprint=None, max_cloud_cover_percentage=None, start_date=None, end_date=None):
Related
Ansible v2.11.x
I have a Jenkins pipeline that does this. All the $VARIABLES are passed-in from the job's parameters.
withCredentials([string(credentialsId: "VAULT_PASSWORD", variable: "VAULT_PASSWORD")]) {
stage("Configure $env.IP_ADDRESS") {
sh """
ansible-playbook -i \\"$env.IP_ADDRESS,\\" \
-e var_host=$env.IP_ADDRESS \
-e web_branch=$env.WEB_BRANCH \
-e web_version=$env.WEB_VERSION \
site.yml
"""
}
}
My playbook is this
---
- hosts: "{{ var_host | default('site') }}"
roles:
- some_role
I have a groups_vars/all.yml file meant to be used by ad-hoc inventories like this. When I run the pipeline, I simply get the following, and the run does nothing
22:52:29 + ansible-playbook -i "10.x.x.x," -e var_host=10.x.x.x -e web_branch=development -e web_version=81cdedd6fe-20210811_2031 site.yml
22:52:31 [WARNING]: Could not match supplied host pattern, ignoring: 10.x.x.x
If I go on the build node and execute exactly the same command, it works. I can also execute the same command on my Mac, and it works too.
So why does the ad-hoc inventory not work when executed in the pipeline?
This post gave me a clue
The correct syntax that worked for me is
withCredentials([string(credentialsId: "VAULT_PASSWORD", variable: "VAULT_PASSWORD")]) {
stage("Configure $env.IP_ADDRESS") {
sh """
ansible-playbook -i $env.IP_ADDRESS, \
-e var_host=$env.IP_ADDRESS \
-e web_branch=$env.WEB_BRANCH \
-e web_version=$env.WEB_VERSION \
site.yml
"""
}
}
Hi I use Kafka connect docker container image confluentinc/cp-kafka-connect 5.5.3 and everything was running fine when using follwing Parameters
...
-e "CONNECT_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter" \
-e "CONNECT_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter" \
-e "CONNECT_INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter" \
-e "CONNECT_INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter" \
...
Now we introduced Schema Registry and decided to go with JsonSchemaConverter for now and not avro. I changed follwoing (INTERNAL stays as it is for now)
...
-e "CONNECT_KEY_CONVERTER=io.confluent.connect.json.JsonSchemaConverter" \
-e "CONNECT_VALUE_CONVERTER=io.confluent.connect.json.JsonSchemaConverter" \
-e "CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL=http://<schemaregsirty_url>:8081" \
-e "CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL=http://<schemaregsirty_url>:8081" \
...
Following Error appeared:
[2021-02-04 09:24:14,637] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed)org.apache.kafka.common.config.ConfigException: Invalid value io.confluent.connect.json.JsonSchemaConverter for configuration key.converter: Class io.confluent.connect.json.JsonSchemaConverter could not be found.
at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:727)
at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:473)
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:466)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:108)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:129)
at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:374)
at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:316)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:93)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
It seems the converter is not available here by default. Do I have to install JsonSchemaConverter? I thought it comes by default?
I want to pass my GOOGLE_APPLICATION_CREDENTIALS environmental variable when I run mlflow run using a Docker container:
This is my current docker run when using mlflow run:
Running command 'docker run --rm -e MLFLOW_RUN_ID=f18667e37ecb486cac4631cbaf279903 -e MLFLOW_TRACKING_URI=http://3.1.1.11:5000 -e MLFLOW_EXPERIMENT_ID=0 mlflow_gcp:33156ee python -m trainer.task --job-dir /tmp/ \
--num-epochs 10 \
--train-steps 1000 \
--eval-steps 1 \
--train-files gs://cloud-samples-data/ml-engine/census/data/adult.data.csv \
--eval-files gs://cloud-samples-data/ml-engine/census/data/adult.test.csv \
--batch-size 128
This is how I would normally pass it:
docker run \
-p 9090:${PORT} \
-e PORT=${PORT} \
-e GOOGLE_APPLICATION_CREDENTIALS=/tmp/keys/[FILE_NAME].json
What is the best way to option to pass this value to mlflow? I'm writing files in GCS and Docker requires access to GCP.
MLproject contents
name: mlflow_gcp
docker_env:
image: mlflow-gcp-example
entry_points:
main:
parameters:
job_dir:
type: string
default: '/tmp/'
num_epochs:
type: int
default: 10
train_steps:
type: int
default: 1000
eval_steps:
type: int
default: 1
batch_size:
type: int
default: 64
train_files:
type: string
default: 'gs://cloud-samples-data/ml-engine/census/data/adult.data.csv'
eval_files:
type: string
default: 'gs://cloud-samples-data/ml-engine/census/data/adult.test.csv'
mlflow_tracking_uri:
type: uri
default: ''
command: |
python -m trainer.task --job-dir {job_dir} \
--num-epochs {num_epochs} \
--train-steps {train_steps} \
--eval-steps {eval_steps} \
--train-files {train_files} \
--eval-files {eval_files} \
--batch-size {batch_size} \
--mlflow-tracking-uri {mlflow_tracking_uri}
I already tried in Python file and fails since Docker has no access to local file system:
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/Users/user/key.json"
This is my setup:
docker pull riot/riotbuild
wget https://github.com/RIOT-OS/RIOT/archive/2019.04.zip
unzip 2019.04.zip
cd RIOT-2019.04/examples/hello-world/
make BUILD_IN_DOCKER=1 BOARD=stm32f4discovery all
All of this is OK.
make BUILD_IN_DOCKER=1 BOARD=stm32f4discovery flash
This step failed. Error is :
> Building application "hello-world" for "stm32f4discovery" with MCU
> "stm32f4". "make" -C /data/riotbuild/riotbase/boards/stm32f4discovery
> "make" -C /data/riotbuild/riotbase/core "make" -C
> /data/riotbuild/riotbase/cpu/stm32f4 "make" -C
> /data/riotbuild/riotbase/cpu/cortexm_common "make" -C
> /data/riotbuild/riotbase/cpu/cortexm_common/periph "make" -C
> /data/riotbuild/riotbase/cpu/stm32_common "make" -C
> /data/riotbuild/riotbase/cpu/stm32_common/periph "make" -C
> /data/riotbuild/riotbase/cpu/stm32f4/periph "make" -C
> /data/riotbuild/riotbase/drivers "make" -C
> /data/riotbuild/riotbase/drivers/periph_common "make" -C
> /data/riotbuild/riotbase/sys "make" -C
> /data/riotbuild/riotbase/sys/auto_init "make" -C
> /data/riotbuild/riotbase/sys/isrpipe "make" -C
> /data/riotbuild/riotbase/sys/newlib_syscalls_default "make" -C
> /data/riotbuild/riotbase/sys/pm_layered "make" -C
> /data/riotbuild/riotbase/sys/stdio_uart "make" -C
> /data/riotbuild/riotbase/sys/tsrb text data bss dec
> hex filename 8756 140 2620 11516 2cfc
> /data/riotbuild/riotbase/examples/hello-world/bin/stm32f4discovery/hello-world.elf
> /home/huaxing/oproj/riot/RIOT-2019.04/dist/tools/openocd/openocd.sh
> flash
> /home/huaxing/oproj/riot/RIOT-2019.04/examples/hello-world/bin/stm32f4discovery/hello-world.elf
> Flashing Target sh: 1: openocd: not found
> /home/huaxing/oproj/riot/RIOT-2019.04/examples/hello-world/../../Makefile.include:538:
> recipe for target 'flash' failed make: *** [flash] Error 127
I check the script, in fact it will finally call docker as next:
a) make BUILD_IN_DOCKER=1 BOARD=stm32f4discovery all
docker run --rm -t -u "$(id -u)" \
-v '/usr/share/zoneinfo/PRC:/etc/localtime:ro' -v '/home/shubuntu1/g/RIOT-2019.04:/data/riotbuild/riotbase' -e 'RIOTBASE=/data/riotbuild/riotbase' -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' -e 'BUILD_DIR=/data/riotbuild/riotbase/build' -e 'RIOTPROJECT=/data/riotbuild/riotbase/examples/hello-world' -e 'RIOTCPU=/data/riotbuild/riotbase/cpu' -e 'RIOTBOARD=/data/riotbuild/riotbase/boards' -e 'RIOTMAKE=/data/riotbuild/riotbase/makefiles' \
-e 'BOARD=stm32f4discovery' \
-w '/data/riotbuild/riotbase/examples/hello-world/' \
'riot/riotbuild:latest' make all 'BOARD=stm32f4discovery'
b) make BUILD_IN_DOCKER=1 BOARD=stm32f4discovery flash
docker run --rm -t -u "$(id -u)" \
-v '/usr/share/zoneinfo/PRC:/etc/localtime:ro' -v '/home/shubuntu1/g/RIOT-2019.04:/data/riotbuild/riotbase' -e 'RIOTBASE=/data/riotbuild/riotbase' -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' -e 'BUILD_DIR=/data/riotbuild/riotbase/build' -e 'RIOTPROJECT=/data/riotbuild/riotbase/examples/hello-world' -e 'RIOTCPU=/data/riotbuild/riotbase/cpu' -e 'RIOTBOARD=/data/riotbuild/riotbase/boards' -e 'RIOTMAKE=/data/riotbuild/riotbase/makefiles' \
-e 'BOARD=stm32f4discovery' \
-w '/data/riotbuild/riotbase/examples/hello-world/' \
'riot/riotbuild:latest' make 'BOARD=stm32f4discovery'
They are nearly same, and the main use for container is just to build the source to stm32 binary as it's not add any permissions to control hardware when start container.
Compare to a), the b) also have next output:
### Flashing Target ###
sh: 1: openocd: not found
/home/shubuntu1/g/RIOT-2019.04/examples/hello-world/../../Makefile.include:538: recipe for target 'flash' failed
make: *** [flash] Error 127
This in fact not related to container, it runs on host, the root cause is you do not install openocd on host which used to flash the stm32 binary to device.
See this, search OpenOCD, it clear requires you to install openocd, then after the docker container finish the build, the binary will also be in host as it's in docker volume, and finally the openocd on host could flash your binary.
I'm trying to run a foreach function that only should run when a condition is fulfilled.
The foreach function works fine when I delete the surrounding if. When I write my makefile like the followring, the foreach loop stops working before the first command is executed and the build process never ends (I don't think that I built an infinite loop, because none of the commands inside is executed).
if [ $(BUILD_SPEC) = mySpec ]; \
then ( \
if [ ! -d $(PRJ_ROOT_DIR)/TARGET ]; then mkdir $(PRJ_ROOT_DIR)/TARGET; fi; \
$(foreach target,$(basename $(PROJECT_TARGETS)), \
if [ -e $(PRJ_ROOT_DIR)/$(BUILD_SPEC)/$(target).crc ]; \
then ( \
echo Deleting $(PRJ_ROOT_DIR)/$(BUILD_SPEC)/$(target).crc; \
rm -f $(PRJ_ROOT_DIR)/$(BUILD_SPEC)/$(target).crc; \
) \
fi; \
) \
) \
fi;
I solved this by my own:
For some reason the command "if [ $(BUILD_SPEC) = mySpec ];" does not work. I replaced it by "ifeq ($(BUILD_SPEC),mySpec)". Now the Script works the way I wanted it to.