node-red upload attachment over http with token (multipart/form-data) - token

I am lost recreating curl POST request in node red.
My bank API documentation requires to form request like this:
multipart/form-data
curl -S -s --cacert GeoTrust_Primary_CA.pem -X POST -F "type=xml" -F "token=aGEMQB9Idh35fxxxxxxxxxxxxxQwyGlQ" -F "file=#C:\davka.xml" https://www.fio.cz/ib_api/rest/import/> result.xml 2>errorlog.txt type errorlog.txt
I really need just
curl -X POST -F "type=xml" -F "token=aGEMQB9Idh35fxxxxxxxxxxxxxQwyGlQ" -F "file=#C:\davka.xml" https://www.fio.cz/ib_api/rest/import/
However I am not able to pass the token to the form.
still get Required String parameter 'token' is not present
I failed with prefered http request or many node-red-contrib-send-form(...) variations.
My flow:
[{"id":"f0c84434.8907a8","type":"fileinject","z":"6012893b.689db8","name":"/share/result.xml","x":220,"y":300,"wires":[["690ae5f4.37b66c"]]},{"id":"690ae5f4.37b66c","type":"function","z":"6012893b.689db8","name":"","func":"msg.headers = {\n "content-type" : 'multipart/form-data'\n };\nlet databuffer = msg.payload;\n\nmsg.payload = {\n "KEY": {\n "value": databuffer,\n "options": {\n "filename": "data.xml",\n "type": "xml",\n "token": "aGEMQB9Idh35fxxxxxfakexxxxxxxxQwyGlQ"\n }\n }\n}\n\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":600,"y":220,"wires":[["6cca79ea.a45f28"]]},{"id":"6cca79ea.a45f28","type":"http request","z":"6012893b.689db8","name":"","method":"POST","ret":"txt","paytoqs":"ignore","url":"https://www.fio.cz/ib_api/rest/import/","tls":"","persist":false,"proxy":"","authType":"","x":760,"y":220,"wires":[["8ec9dd16.83be","3f347cc6.b9a114"]]},{"id":"8ec9dd16.83be","type":"debug","z":"6012893b.689db8","name":"File has been sent","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":980,"y":220,"wires":[]},{"id":"ba973765.11aa88","type":"inject","z":"6012893b.689db8","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":200,"y":220,"wires":[["e75f0fb7.fa71b"]]},{"id":"e75f0fb7.fa71b","type":"file in","z":"6012893b.689db8","name":"","filename":"/share/result.xml","format":"","chunk":false,"sendError":false,"encoding":"none","x":360,"y":220,"wires":[["690ae5f4.37b66c"]]},{"id":"3f347cc6.b9a114","type":"debug","z":"6012893b.689db8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"statusCode","targetType":"msg","statusVal":"","statusType":"auto","x":970,"y":280,"wires":[]}]

You should be able to do this with the core HTTP Request node. The issue is the format of the payload you are passing it.
If you change your Function node to structure the payload like this, I believe you should get further:
msg.payload = {
"file": {
"value": databuffer,
"options": {
"filename": "data.xml",
}
},
"type": "xml",
"token": "aGEMQB9Idh35fxxxxxfakexxxxxxxxQwyGlQ"
}

Related

Exception while adding ssh credentials to Jenkins programmatically

I have used a method mentioned here to add credential to Jenkins programmatically. It worked successfully for adding secret texts and secrets files. But it gives an exception while adding ssh private keys. Below is the curl command I used.
curl -X POST 'http://localhost:8080/jenkins/credentials/store/system/domain/_/createCredentials' \
--data-urlencode 'json={
"": "0",
"credentials": {
"scope": "GLOBAL",
"id": "temp",
"username": "temp",
"privateKeySource": {
"stapler-class": "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$FileOnMasterPrivateKeySource",
"privateKeyFile": "/home/udhan/private-key.pem",
},
"stapler-class": "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey"
}
}'
Here is the exception I get.
A problem occurred while processing the request.
Please check our bug tracker to see if a similar problem has already been reported.
If it is already reported, please vote and put a comment on it to let us gauge the impact of the problem.
If you think this is a new issue, please file a new issue.
When you file an issue, make sure to add the entire stack trace, along with the version of Jenkins and relevant plugins.
The users list might be also useful in understanding what has happened.</p><h2>Stack trace</h2><pre style="margin:2em; clear:both">org.kohsuke.stapler.NoStaplerConstructorException: There's no #DataBoundConstructor on any constructor of class com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$FileOnMasterPrivateKeySource
at org.kohsuke.stapler.ClassDescriptor.loadConstructorParamNames(ClassDescriptor.java:265)
at org.kohsuke.stapler.RequestImpl.instantiate(RequestImpl.java:765)
at org.kohsuke.stapler.RequestImpl.access$200(RequestImpl.java:83)
at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:678)
I've just bumped into this same problem right now. Rather than using a pem file, I ended up putting the SSH pem's value into a variable and passed it that way instead.
CRUMB=$(curl -s 'https://{{jenkins_admin_username}}:{{jenkins_admin_password}}#localhost:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)')
SSH_KEY="$(cat /your/ssh/pem)"
curl -H $CRUMB -X POST 'https://{{jenkins_admin_username}}:{{jenkins_admin_password}}#localhost:8080/credentials/store/system/domain/_/createCredentials' --data-urlencode 'json={
"": "0",
"credentials": {
"scope": "GLOBAL",
"id": "'test-jenkins-id'",
"username": "'test-username'",
"password": "",
"privateKeySource": {
"stapler-class": "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$DirectEntryPrivateKeySource",
"privateKey": "$SSH_KEY",
},
"description": "test-jenkins-ssh description",
"stapler-class": "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey"
}
}'
Not that I went with https instead of http here as we're passing things that should be secured.
I hope this helps.

502 Bad Gateway nginx/1.13.12 on localhost while creating a new policy in ditto

I am new to docker, I'm using docker on windows 10 and am using Eclipse Ditto and Eclipse Mosquitto, I have to create a twin device. Before that I have to create a new policy via Postman,
While executing the policy am getting the following error:
The code to create the new policy is ass follow
curl -X put 'http://localhost:8080/api/2/policies/my.test:policy' -u 'ditto:ditto' -H 'Content-Type: application/json' -d '{
"entries": {
"owner": {
"subjects": {
"nginx:ditto": {
"type": "nginx basic auth user"
}
},
"resources": {
"thing:/": {
"grant": [
"READ","WRITE"
],
"revoke": []
},
"policy:/": {
"grant": [
"READ","WRITE"
],
"revoke": []
},
"message:/": {
"grant": [
"READ","WRITE"
],
"revoke": []
}
}
}
}
}'
I'm following the procedure given in this link
is there any solution to solve this kind of problem?
Seems that the Eclipse Ditto gateway service was not running correctly.
I'm confident that you figured out to look at the output of docker-compose ps and restarted the services.

SCM-manager CLI to set branch write protection

I have downloaded SCM-manager-CLI
I need to set the branch write protection for GIT repository in SCM-manager through CLI
I have set the same through web-app
I need to do that through CLI as a part of Automation
How to do that?
Hi the cli has currently no support for properties, but you can use the rest api to set the properties:
curl -u scmadmin:scmadmin -v http://localhost:8080/scm/api/rest/repositories -XPOST -H 'Content-Type: application/json' -d '{
"name": "newrepository",
"description": "repository with branch protection",
"type": "git",
"properties": [
{ "key": "branchwp.enabled", "value": true },
{ "key": "branchwp.permissions", "value": "master,userone;develop,#devgroup;" }
],
"permissions": [
{ "name": "userone", "type": "WRITE", "groupPermission": false },
{ "name": "devgroup", "type": "WRITE", "groupPermission": true }
]
}'
The example above creates a new repository with enabled branch protection.

How to create jenkins credentials via the REST API?

I need to create a jenkins credential ( https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Plugin ) via a script. How can I do that using either the REST API or the cli ?
Note that I'm able to list the credentials using /credential-store/domain//api/json and /credential-store/domain//credential/8bd82461-e239-4db1-90bc-831ca3412e70/api/json etc.
This issue took me a while to figure, a lot of digging around, so I decided to let the solution here, if someone else needs it.
curl -X POST 'http://user:token#jenkins_server:8080/credentials/store/system/domain/_/createCredentials' \
--data-urlencode 'json={
"": "0",
"credentials": {
"scope": "GLOBAL",
"id": "identification",
"username": "manu",
"password": "bar",
"description": "linda",
"$class": "com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl"
}
}'
with latest jenkins you need a CRUMB to authenticate for this operation (ref https://stackoverflow.com/a/38314286)
CRUMB=$(curl -s 'http://user:token#jenkins_server:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)')
curl -H $CRUMB -X POST 'http://user:token#jenkins_server:8080/credentials/store/system/domain/_/createCredentials' \
--data-urlencode 'json={
"": "0",
"credentials": {
"scope": "GLOBAL",
"id": "identification",
"username": "manu",
"password": "bar",
"description": "linda",
"$class": "com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl"
}
}'
Otherwise you get
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /credentials/store/system/domain/_/createCredentials. Reason:
<pre> No valid crumb was included in the request</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
if you need to create credentials but with pem file path you can use this:
prerequisites: ssh-credentials plugin
CRUMB=$(curl -s 'http://{{jenkins_admin_username}}:{{jenkins_admin_password}}#localhost:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)')
curl -H $CRUMB -X POST 'http://{{jenkins_admin_username}}:{{jenkins_admin_password}}#localhost:8080/credentials/store/system/domain/_/createCredentials' \
--data-urlencode 'json={
"": "0",
"credentials": {
"scope": "GLOBAL",
"id": "'{{ii.ssh_user}}'",
"username": "'{{ii.ssh_user}}'",
"password": "",
"privateKeySource": {
"stapler-class": "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$FileOnMasterPrivateKeySource",
"privateKeyFile": "'{{jenkins_home}}/{{ii.key_name}}.pem'",
},
"description": "'{{ii.ssh_user}}'",
"stapler-class": "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey"
}
}'
this command used in ansible but you can replace the {{variables}} with your own variables
if you need to add all the pem file content you need to change the lines to:
....
"stapler-class": "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$DirectEntryPrivateKeySource",
"privateKey": "{{private_key_content}}",
},
"description": "{{user}}",
"stapler-class": "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey"
...
Just adding my 2 cents here: if you want to create the credentials for a specific folder, then use the following:
curl -H $CRUMB -X POST 'http://user:token#jenkins_server:8080/job/MY_FOLDER_NAME/credentials/store/folder/domain/_/createCredentials' \
...
So, you need to use /job/My_Folder at the beginning of the query part and replace the /store/system with /store/folder
There is no specific API call for this, but you can do it via cli commands to the jenkins jar.
echo 'jenkins.model.Jenkins.instance.securityRealm.createAccount("username", "password")' | java -jar jenkins-cli.jar -s http://localhost/ groovy =
For granting them permissions you can create a task in Jenkins which is running every N minutes and executing a groovy script as described here:
https://wiki.jenkins-ci.org/display/JENKINS/Grant+Cancel+Permission+for+user+and+group+that+have+Build+permission
I have a groovy script that also sets user permission using Matrix-based security. The script was posted at Creating user in Jenkins via API
Unable to point to ssh keys in ~/.ssh on Jenkins host
Means this no longer works,
"privateKeySource": {
"stapler-class": "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$FileOnMasterPrivateKeySource",
"privateKeyFile": "'{{jenkins_home}}/{{ii.key_name}}.pem'",
},
📌 Here is the official documentation for managing the Jenkins Credentials via REST API
https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials
Example of adding a deploy-key credential using the username wecoyote and the password secret123 in the testing domain of the /example-folder folder.
$ cat > credential.xml <<EOF
<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
<scope>GLOBAL</scope>
<id>deploy-key</id>
<description>Test User</description>
<usernameSecret>false</usernameSecret>
<username>wecoyote</username>
<password>secret123</password>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
EOF
$ curl -X POST -u <username>:<password_or_token> -H content-type:application/xml -d #credential.xml \
https://jenkins.example.com/job/example-folder/credentials/store/folder/\
domain/testing/createCredentials
The expected responses are:
HTTP/200 Success, the credentials has been created.
HTTP/409 Failure, a credential with that id already exists.
HTTP/50x Could not parse the supplied domain XML body.

Neo.ClientError.Statement.EntityNotFound when deleting and then creating nodes in parallel using transactional Neo4j REST API

When I delete and then create nodes in Neo4j database (v2.1.6, Ubuntu) using the transactional REST API and auto-commit mode in parallel I get errors like this one
{"results":[],"errors":[{"code":"Neo.ClientError.Statement.EntityNotFound","message":"Node with id 35275"}]}
For each node I run the query:
curl -X POST -d #data http://localhost:7474/db/data/transaction/commit -H "Content-Type:application/json; charset=UTF-8" -H Accept:application/json >out$i
with the following statements
{
"statements": [
{
"statement": "MATCH (k:Keyword {kwId: \"1\"}) DELETE k",
"parameters": {}
},
{
"statement": "CREATE (k:Keyword {props}) RETURN k",
"parameters": {
"props": {
"kwId": "1"
}
}
}
]
}
The kwId values are different for each query.
When I execute queries sequentially everything is fine
When I run the queries in parallel I get the error responses for some of the queries (~5 error responses for 20 queries)
{"results":[],"errors":[{"code":"Neo.ClientError.Statement.EntityNotFound","message":"Node with id 35275"}]}
Is this a Neo4j issue or I am using it wrong?
The script which reproduces the issue:
#!/bin/bash
NEO4J_URL=http://localhost:7474
TOTAL_QUERIES=20
# Create temp JSON files tmp1.json, tmp2.json, ....
for (( i=1; i<$TOTAL_QUERIES; i++ ))
do
JSON=tmp$i.json
cat > $JSON << END_OF_JSON
{
"statements": [
{
"statement": "MATCH (k:Keyword {kwId: \"$i\"}) DELETE k",
"parameters": {}
},
{
"statement": "CREATE (k:Keyword {props}) RETURN k",
"parameters": {
"props": {
"kwId": "$i"
}
}
}
]
}
END_OF_JSON
done
for (( i=1; i<$TOTAL_QUERIES; i++ ))
do
JSON=tmp$i.json
if [ "$1" = "sync" ]; then
curl --silent -X POST -d #$JSON $NEO4J_URL/db/data/transaction/commit -H "Content-Type:application/json; charset=UTF-8" -H Accept:application/json >out$i
else
curl --silent -X POST -d #$JSON $NEO4J_URL/db/data/transaction/commit -H "Content-Type:application/json; charset=UTF-8" -H Accept:application/json >out$i &
fi
done
wait
awk 'FNR==1{print ""}1' out* > result
grep "\"errors\":\[.*\]" result
rm out* tmp*.json result
Update: when using legacy REST API the error messages are better
"message" : "Node with id 35970",
"exception" : "EntityNotFoundException",
"fullname" : "org.neo4j.cypher.EntityNotFoundException",
"stacktrace" : [ "org.neo4j.cypher.internal.spi.v2_1.TransactionBoundQueryContext$NodeOperations.getById(TransactionBoundQueryContext.scala:160)", "org.neo4j.cypher.internal.spi.v2_1.TransactionBoundQueryContext$$anonfun$getNodesByLabel$1.apply(TransactionBoundQueryContext.scala:135)", "org.neo4j.cypher.internal.spi.v2_1.TransactionBoundQueryContext$$anonfun$getNodesByLabel$1.apply(TransactionBoundQueryContext.scala:135)", "org.neo4j.cypher.internal.helpers.JavaConversionSupport$$anon$2.next(JavaConversionSupport.scala:33)", "scala.collection.Iterator$$anon$11.next(Iterator.scala:328)", "scala.collection.Iterator$$anon$13.next(Iterator.scala:372)", "scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:389)", "scala.collection.Iterator$class.foreach(Iterator.scala:727)", "scala.collection.AbstractIterator.foreach(Iterator.scala:1157)", "scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)", "scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:176)", "scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:45)", "scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)", "scala.collection.AbstractIterator.to(Iterator.scala:1157)", "scala.collection.TraversableOnce$class.toList(TraversableOnce.scala:257)", "scala.collection.AbstractIterator.toList(Iterator.scala:1157)", "org.neo4j.cypher.internal.compiler.v2_1.pipes.EagerPipe.internalCreateResults(EagerPipe.scala:32)", "org.neo4j.cypher.internal.compiler.v2_1.pipes.PipeWithSource.createResults(Pipe.scala:105)", "org.neo4j.cypher.internal.compiler.v2_1.pipes.PipeWithSource.createResults(Pipe.scala:102)", "org.neo4j.cypher.internal.compiler.v2_1.pipes.PipeWithSource.createResults(Pipe.scala:102)", "org.neo4j.cypher.internal.compiler.v2_1.pipes.PipeWithSource.createResults(Pipe.scala:102)", "org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionPlanBuilder$$anonfun$getExecutionPlanFunction$1$$anonfun$apply$2.apply(ExecutionPlanBuilder.scala:120)", "org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionPlanBuilder$$anonfun$getExecutionPlanFunction$1$$anonfun$apply$2.apply(ExecutionPlanBuilder.scala:119)", "org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionWorkflowBuilder.runWithQueryState(ExecutionPlanBuilder.scala:168)", "org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionPlanBuilder$$anonfun$getExecutionPlanFunction$1.apply(ExecutionPlanBuilder.scala:118)", "org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionPlanBuilder$$anonfun$getExecutionPlanFunction$1.apply(ExecutionPlanBuilder.scala:103)", "org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionPlanBuilder$$anon$1.execute(ExecutionPlanBuilder.scala:68)", "org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionPlanBuilder$$anon$1.execute(ExecutionPlanBuilder.scala:67)", "org.neo4j.cypher.internal.ExecutionPlanWrapperForV2_1.execute(CypherCompiler.scala:159)", "org.neo4j.cypher.ExecutionEngine.execute(ExecutionEngine.scala:76)", "org.neo4j.cypher.ExecutionEngine.execute(ExecutionEngine.scala:71)", "org.neo4j.cypher.javacompat.ExecutionEngine.execute(ExecutionEngine.java:84)", "org.neo4j.server.rest.web.CypherService.cypher(CypherService.java:114)", "java.lang.reflect.Method.invoke(Method.java:606)", "org.neo4j.server.rest.transactional.TransactionalRequestDispatcher.dispatch(TransactionalRequestDispatcher.java:139)", "java.lang.Thread.run(Thread.java:745)" ],
Not sure if this helps, but just a random thing I noticed: You're using parameters for the CREATE but not the DELETE. I don't know if/why that might cause race conditions, but it's worth checking
It also seems a bit weird to be doing these two queries. Maybe there's some confidential stuff that you stripped out, but I think you could do:
MATCH (k:Keyword) WHERE k.kwId IN ({kw_ids}) DELETE k WITH k.kwId AS kwId CREATE (:Keyword: {kwId: kwId})
Creating new index on the property kwId seems to fix the errors for my case
CREATE INDEX ON :Keyword(kwId)

Resources