I added several docs to an index in a single type in my elasticsearch database by running the following lines :
curl -XPUT localhost:9200/dt/cc/1 -d '{"cn": "ibm","name": "cat1", "lines": 80 }'
curl -XPUT localhost:9200/dt/cc/2 -d '{"cn": "zf", "name": "fox1", "lines": 90 }'
curl -XPUT localhost:9200/dt/cc/3 -d '{"cn": "zf", "name": "fox2", "lines": 42 }'
curl -XPUT localhost:9200/dt/cc/4 -d '{"cn": "ibm","name": "cat2", "lines": 50 }'
curl -XPUT localhost:9200/dt/cc/5 -d '{"cn": "ibm","name": "cat3", "lines": 20 }'
curl -XPUT localhost:9200/dt/cc/6 -d '{"cn": "zf", "name": "fox3", "lines": 20 }'
curl -XPUT localhost:9200/dt/cc/7 -d '{"cn": "ibm","name": "cat4"}'
curl -XPUT localhost:9200/dt/cc/8 -d '{"cn": "zf", "name": "fox4"}'
I know how to calculate the average of field 'lines' among docs whose cn is "zf", which is actually 51.
Now, I am looking for a way to add another field named difference_to_zf_mean_lines to docs whose 'cn' is 'ibm', indicating the difference between the 'lines' values and the previously calculated mean (51).
This is the kind of structure I want to get:
'{"cn": "ibm","name": "cat1", "lines": 80, difference_to_zf_mean_lines: 29}'
'{"cn": "ibm","name": "cat2", "lines": 50, difference_to_zf_mean_lines: -1}'
'{"cn": "ibm","name": "cat3", "lines": 20, difference_to_zf_mean_lines: -31}'
But I would like to know if elasticsearch provides a way to compute basic operations like this one to add new fields to documents inside an index.
Related
I'd like to execute the following call in rails:
curl -i -X POST \
https://graph.facebook.com/v12.0/FROM_PHONE_NUMBER_ID/messages \
-H 'Authorization: Bearer ACCESS_TOKEN' \
-H 'Content-Type: application/json' \
-d '{ "messaging_product": "whatsapp", "to": "TO_PHONE_NUMBER", "type": "template", "template": { "name": "hello_world", "language": { "code": "en_US" } } }'
How can I do this using an http-client? I can do it in postman but formatting it for an http client usually has some syntax issue.
You can look at the HTTParty gem, which helps make API-call easier.
I'm trying to execute curl to add a value to the API within the same container where API is starting.
Dockerfile command which I'm using to get that to work looks like
CMD java -jar ./my-api.jar ; wait 30; curl -X POST 'http://localhost:8080/user/' -H 'Content-Type: application/json' --data-raw '{ "username: "admin", "data": {"email": "admin#test.com", "password" : "somepassword"}}'
but the admin used was not added after 30 seconds when API got started.
The same curl command works fine when I will execute it within the container itself.
Any thought on that?
Got it solved with moving all to the scrip and then execute it with the CMD
Script looks like:
#!/bin/bash
set -m
java -java -jar ./my-api.jar &
sleep 20
curl -X POST 'http://localhost:8080/user/' -H 'Content-Type: application/json' --data-raw '{ "username: "admin", "data": {"email": "admin#test.com", "password" : "somepassword"}}'
fg %1
I'm trying to upload images to my private docker registry using the HTTP API, unfortunately without success.
Apparently I haven't understood the upload process yet and would like to ask if anyone can explain this in detail or push me in the right direction.
So far I have tried the following with curl. To experiment I only use the empty alpine image.
For that i download it to my workstation with docker pull alpine and then I create tar archive with docker save -o alpine.tar alpine
I then unpack this archive into an alpine directory. This is what the content looks like:
ls -R alpine
alpine:
39cb81dcd06e3d4e2b813f56b72da567696fa9a59b652bd477615b31af969239
e7d92cdc71feacf90708cb59182d0df1b911f8ae022d29e8e95d75ca6a99776a.json
manifest.json
alpine/39cb81dcd06e3d4e2b813f56b72da567696fa9a59b652bd477615b31af969239:
json
layer.tar
VERSION
According to the documentation I should first initiate the upload by sending a POST to the /v2//blobs/uploads/ URL, for this I execute the following command:
curl -X POST -L -D headers $DOCKER_HOST/v2/alpine/blobs/uploads
The answer is the following:
cat headers
HTTP/1.1 301 Moved Permanently
Docker distribution api version: registry/2.0
location: /v2/alpine/blobs/uploads/
Date: Tu, 21 Jan 2020 12:42:55 GMT
content length: 0
HTTP/1.1 202 Accepted
content length: 0
Docker distribution api version: registry/2.0
Docker upload guide: ed595b3c-1236-46c8-a759-14187fc60e7d
Location: http://<IPADDRESS>/v2/alpine/blobs/uploads/ed595b3c-1236-46c8-a759-14187fc60e7d?_state=GjzU_y-YDQherf4xXO57KyEonSSSwNEM8FiF8rmNfuN7Ik5hbWUiOiJhbHBpbmUiLCJVVUlEIjoiZWQ1OTViM2MtMTIzNi00NmM4LWE3NTktMTQxODdmYzYwZTdkIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIwLTAxLTIxVDEyOjQyOjU1LjMxNDYzNTg0NVoifQ%3D%3D
Range: 0-0
X Content Type Options: nosniff
Date: Tu, 21 Jan 2020 12:42:55 GMT
now i would like to do a monolithic upload, as described in the docker-registry documentation.
For this I make the following curl request:
url=http://<IPADDRESS>/v2/alpine/blobs/uploads/ed595b3c-1236-46c8-a759-14187fc60e7d?_state=GjzU_y-YDQherf4xXO57KyEonSSSwNEM8FiF8rmNfuN7Ik5hbWUiOiJhbHBpbmUiLCJVVUlEIjoiZWQ1OTViM2MtMTIzNi00NmM4LWE3NTktMTQxODdmYzYwZTdkIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIwLTAxLTIxVDEyOjQyOjU1LjMxNDYzNTg0NVoifQ%3D%3D?digest=sha256:39cb81dcd06e3d4e2b813f56b72da567696fa9a59b652bd477615b31af969239
layerpath=/home/user/alpine/39cb81dcd06e3d4e2b813f56b72da567696fa9a59b652bd477615b31af969239/layer.tar
curl -X PUT -H "Content-Type=application/octet-stream" --data-binary #"$layerpath" $url
For this I receive an answer that means nothing to me:
{"errors":[{"code": "BLOB_UPLOAD_INVALID", "message": "blob upload invalid", "detail":212}]}
I don't know what I'm doing wrong. For any help I am very grateful.
Okay, I figured it out.
For all the others who are also dealing with this question here is a little script which should clarify the handling of the HTTP-Api:
#!/bin/bash
# Global Variables
MANIFEST="./Manifest.json"
DOCKER_HOST=$1
REPOSITORY=$2
LAYERPATH=$3
CONFIGPATH=$4
SIZE=
DIGEST=
LOCATION=
CONFIGSIZE=
CONFIGDIGEST=
LAYERSIZE=
LAYERDIGEST=
# Functions
function initiateUpload(){
LOCATION=$(curl -X POST -siL -v -H "Connection: close" $DOCKER_HOST/v2/$REPOSITORY/blobs/uploads | grep Location | sed '2q;d' | cut -d: -f2- | tr -d ' ' | tr -d '\r')
}
function patchLayer(){
layersize=$(stat -c%s "$1")
LOCATION=$(curl -X PATCH -v -H "Content-Type: application/octet-stream" \
-H "Content-Length: $layersize" -H "Connection: close" --data-binary #"$1" \
$LOCATION 2>&1 | grep 'Location' | cut -d: -f2- | tr -d ' ' | tr -d '\r')
SIZE=$layersize
}
function putLayer(){
DIGEST=sha256:$(sha256sum $1 | cut -d ' ' -f1)
url="$LOCATION&digest=$DIGEST"
curl -X PUT -v -H "Content-Length: 0" -H "Connection: close" $url
}
function uploadManifest(){
((size=$(stat -c%s "$MANIFEST")-1))
curl -X PUT -vvv -H "Content-Type: application/vnd.docker.distribution.manifest.v2+json"\
-H "Content-Length: $size" -H "Connection: close" \
-d "$(cat "$MANIFEST")" $DOCKER_HOST/v2/$REPOSITORY/manifests/latest
}
# Check Parameters
if [ $# -lt 4 ]
then
echo "Error: No arguments supplied."
echo "Usage: upload.sh <DOCKER_HOST> <REPOSITORY> <LAYER> <CONFIG>"
exit 1
fi
#upload Layer
initiateUpload
patchLayer $LAYERPATH
LAYERSIZE=$SIZE
putLayer $LAYERPATH
LAYERDIGEST=$DIGEST
#upload Config
initiateUpload
patchLayer $CONFIGPATH
CONFIGSIZE=$SIZE
putLayer $CONFIGPATH
CONFIGDIGEST=$DIGEST
cat > $MANIFEST << EOF
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": $CONFIGSIZE,
"digest": "$CONFIGDIGEST"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size":$LAYERSIZE,
"digest": "$LAYERDIGEST"
}
]
}
EOF
uploadManifest
It's not pretty, but it works for that.
If you need more information, I will gladly extend this answer
I'm failing to find this information within the existing documentation - either permanent or non-permanent tokens.
Using Keyrock 7.8, Ultralight 1.11.0 (though any current agent will do)
The following Docker parameters are set:
- IOTA_AUTH_ENABLED=true
- IOTA_AUTH_TYPE=oauth2
- IOTA_AUTH_HEADER=Authorization
- IOTA_AUTH_HOST=keyrock
- IOTA_AUTH_PORT=3000
- IOTA_AUTH_URL=http://keyrock:3000
- IOTA_AUTH_CLIENT_ID=tutorial-dckr-site-0000-xpresswebapp
# - IOTA_AUTH_PERMANENT_TOKEN=true
The default Docker configuration is used in the image, so no provisioning group types are created.
I am able to provision a trusted group as shown:
curl -X POST \
http://iot-agent:4041/iot/services \
-H 'fiware-service: openiot' \
-H 'fiware-servicepath: /' \
-d '{
"services": [
{
"apikey": "4jggokgpepnvsb2uv4s40d59ov",
"cbroker": "http://orion:1026",
"entity_type": "Motion",
"resource": "/iot/d",
"trust": "<motn-auth-token>"
}
]
}'
Question 1 - how do I generate the trust token within Keyrock.
When I provision the device
curl -X POST \
http://iot-agent:4041/iot/devices \
-H 'Content-Type: application/json' \
-H 'fiware-service: openiot' \
-H 'fiware-servicepath: /' \
-d '{
"devices": [
{
"device_id": "motion001",
"entity_name": "urn:ngsi-ld:Motion:001",
"entity_type": "Motion",
"timezone": "Europe/Berlin",
"attributes": [
{ "object_id": "c", "name":"count", "type":"Integer"}
],
"static_attributes": [
{"name":"refStore", "type": "Relationship","value": "urn:ngsi-ld:Store:001"}
]
}
]
}
'
I receive the following error in the IoT Agent:
{
"name": "SECURITY_INFORMATION_MISSING",
"message": "Some security information was missing for device type:Motion"
}
And the following in the Keyrock logs:
Fri, 06 Dec 2019 14:13:52 GMT idm:oauth2-model_oauth_server -------getClient-------
Executing (default): SELECT `id`, `redirect_uri`, `token_types`, `jwt_secret`, `scope`, `grant_type` FROM `oauth_client` AS `OauthClient` WHERE `OauthClient`.`id` = 'tutorial-dckr-site-0000-xpresswebapp' AND `OauthClient`.`secret` = 'tutorial-lcal-host-0000-clientsecret';
Fri, 06 Dec 2019 14:13:52 GMT idm:oauth_controller Error { invalid_client: Invalid client: client is invalid
Question 2: What additional information needs to be supplied?
How do I generate the trust token within Keyrock.
Trust tokens are described as access tokens within in the Keyrock documentation, Firstly set up the client application to generate permanent tokens:
This can also be done programmatically by using the /v1/applications endpoint.
Request
curl -iX POST \
'http://keyrock:3005/v1/applications' \
-H 'Content-Type: application/json' \
-H 'X-Auth-token: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa' \
-d '{
"application": {
"name": "Tutorial Application",
"description": "FIWARE Application protected by OAuth2 and Keyrock",
"redirect_uri": "http://tutorial/login",
"url": "http://tutorial",
"grant_type": [
"authorization_code",
"implicit",
"password"
],
"token_types": ["permanent"]
}
}'
To generate a permanent trust token, ensure the Keyrock application has been configured to offer permanent tokens and log in as an authorised user using the standard Authorization: Basic header holding a base 64 concatenation of the client id and secret. The parameter
scope=permanent is added to retrieve permanent tokens when available. The response contains an access_token (a.k.a. Trust Token) which
can be used for device provisioning.
Request
curl -X POST \
http://keyrock:3005/oauth2/token \
-H 'Accept: application/json' \
-H 'Authorization: Basic dHV0b3JpYWwtZGNrci1zaXRlLTAwMDAteHByZXNzd2ViYXBwOnR1dG9yaWFsLWRja3Itc2l0ZS0wMDAwLWNsaWVudHNlY3JldA==' \
-d 'username=alice-the-admin#test.com&password=test&grant_type=password&scope=permanent'
Response
{
"access_token": "e37aeef5d48c9c1a3d4adf72626a8745918d4355",
"token_type": "Bearer",
"scope": ["permanent"]
}
Setting up the IoT Agent to use Keyrock and a PEP proxy
The following additional Docker parameters are required
- IOTA_CB_HOST=orion-proxy
- IOTA_CB_PORT=${ORION_PROXY_PORT}
- IOTA_AUTH_ENABLED=true
- IOTA_AUTH_TYPE=oauth2
- IOTA_AUTH_HEADER=Authorization
- IOTA_AUTH_HOST=keyrock
- IOTA_AUTH_PORT=${KEYROCK_PORT}
- IOTA_AUTH_URL=http://keyrock:${KEYROCK_PORT}
- IOTA_AUTH_CLIENT_ID=tutorial-dckr-site-0000-xpresswebapp
- IOTA_AUTH_CLIENT_SECRET=tutorial-dckr-host-0000-clientsecret
- IOTA_AUTH_PERMANENT_TOKEN=true
- IOTA_AUTH_TOKEN_PATH=/oauth2/token
IoT Agent - provisioning a trusted service group
The Access token (also known as a Trust Token) must be added to the service group.
This is held in the trust attribute and repeats the token retrieved in the step above
Request
curl -iX POST \
'http://iot-agent:4041/iot/services' \
-H 'Content-Type: application/json' \
-H 'fiware-service: openiot' \
-H 'fiware-servicepath: /' \
-d '{
"services": [
{
"apikey": "4jggokgpepnvsb2uv4s40d59ov",
"cbroker": "http://orion:1026",
"entity_type": "Motion",
"resource": "/iot/d",
"trust": "e37aeef5d48c9c1a3d4adf72626a8745918d4355"
}
]
}'
Once a trusted service group has been created, a device can be provisioned in the usual manner
Request
curl -iX POST \
'http://iot-agent:4041/iot/devices' \
-H 'Content-Type: application/json' \
-H 'fiware-service: openiot' \
-H 'fiware-servicepath: /' \
-d '{
"devices": [
{
"device_id": "motion001",
"entity_name": "urn:ngsi-ld:Motion:001",
"entity_type": "Motion",
"timezone": "Europe/Berlin",
}
]
}
'
I create an installation object using a REST API call like this :
curl -X POST \
-H "X-Parse-Application-Id: ${APPLICATION_ID}" \
-H "X-Parse-REST-API-Key: ${REST_API_KEY}" \
-H "Content-Type: application/json" \
-d '{
"deviceType": "ios",
"deviceToken": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
"channels": [
""
]
}' \
https://<your.parseprovider.here>/1/installations
The installation object is created and is indicated by the response :
{
"objectId": "EmqGmZXGEm",
"createdAt": "2017-02-15T10:13:18.647Z"
}
Now let's say I want to update the channels field to include the "foo" channel in the installation object, I could simply issue a call like :
curl -X PUT \
-H "X-Parse-Application-Id: ${APPLICATION_ID}" \
-H "X-Parse-REST-API-Key: ${REST_API_KEY}" \
-H "Content-Type: application/json" \
-d '{
"channels": [
"",
"foo"
]
}' \
https://<your.parseprovider.here>/1/installations/EmqGmZXGEm
Success is then indicated by the response :
{
"updatedAt": "2017-02-15T10:18:31.055Z"
}
However, when I execute the PUT call like this (as in the REST API docs, note the inclusion of the deviceType and deviceToken fields) :
curl -X PUT \
-H "X-Parse-Application-Id: ${APPLICATION_ID}" \
-H "X-Parse-REST-API-Key: ${REST_API_KEY}" \
-H "Content-Type: application/json" \
-d '{
"deviceType":"ios",
"deviceToken":"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
"channels": [
"",
"foo"
]
}' \
https://<your.parseprovider.here>/1/installations/EmqGmZXGEm
I now get the following response :
{
"code": 101,
"error": "Object not found."
}
The installation object has now suddenly been deleted from the Parse server database.
This seems to happen as soon as the deviceToken field is included in the PUT request.
Is this supposed to happen, or am I missing something? I am using a Parse API for Delphi which is breaking because of this "phenomenon". I would rather not hack the API if the error is due to a Parse bug that should be fixed on the server side.
Try PATCH instead of PUT. See table. Both PUT and PATCH can be used for update.