As part of an automated tests suite I have to use OpenShift's REST APIs to send commands and get OpenShift's status. To authenticate these API calls I need to embed an authorization token in every call.
Currently, I get this token by executing the following commands with ssh on the machine where OpenShift is installed:
oc login --username=<uname> --password=<password>
oc whoami --show-token
I would like to stop using the oc tool completely and get this token using HTTP calls to the APIs but am not really able to find a document that explains how to use it. If I use the option --loglevel=10 when calling oc commands I can see the HTTP calls made by oc when logging in but it is quite difficult for me to reverse-engineer the process from these logs.
Theoretically this is not something specific to OpenShift but rather to the OAuth protocol, I have found some documentation like the one posted here but I still find it difficult to implement without specific examples.
If that helps, I am developing this tool using ruby (not rails).
P.S. I know that normally for this type of job one should use Service Account Tokens but since this is a testing environment the OpenShift installation gets removed and reinstalled fairly often. This would force me to re-create the service account every time with the oc command line tool and again prevent me from automatizing the process.
I have found the answer in this GitHub issue.
Surprisingly, one curl command is enough to get the token:
curl -u joe:password -kv -H "X-CSRF-Token: xxx" 'https://master.cluster.local:8443/oauth/authorize?client_id=openshift-challenging-client&response_type=token'
The response is going to be an HTTP 302 trying to redirect to another URL. The redirection URL will contain the token, for example:
Location: https://master.cluster.local:8443/oauth/token/display#access_token=VO4dAgNGLnX5MGYu_wXau8au2Rw0QAqnwq8AtrLkMfU&expires_in=86400&token_type=bearer
You can use token or combination user/password.
To use username:password in header, you can use Authorizartion: Basic. The oc client commands are doing simple authentication with your user and password in header. Like this
curl -H "Authorization: Basic <SOMEHASH>"
where the hash is exactly base64 encoded username:password. (try it with echo -n "username:password" | base64).
To use token, you can obtain the token here with curl:
curl -H Authorization: Basic $(echo -n username:password | base64)" https://openshift.example.com:8443/oauth/authorize\?response_type\=token\&client_id\=openshift-challenging-client
But the token is replied in the ugly format format. You can try to grep it
... | grep -oP "access_token=\K[ˆ&]*"
You need to use the correct url for your oauth server. In my case, I use openshift 4.7 and this is the url:
https://oauth-openshift.apps.<clustername><domain>/oauth/authorize\?response_type\=token\&client_id\=openshift-challenging-client
oc get route oauth-openshift -n openshift-authentication -o json | jq .spec.host
In case you are using OpenShift CRC:
Then the URL is: https://oauth-openshift.apps-crc.testing/oauth/authorize
Command to get the Token:
curl -v --insecure --user developer:developer --header "X-CSRF-Token: xxx" --url "https://oauth-openshift.apps-crc.testing/oauth/authorize?response_type=token&client_id=openshift-challenging-client" 2>&1 | grep -oP "access_token=\K[^&]*"
Note:
2>&1 is required, because curl writes to standard error
--insecure: because I have not set up TLS certificate
Adjust the user and password developer as needed (crc developer/developer is standard user in crc, therefore good for testing.)
Token is per default 24h vaild
Export the Token to an environment Variable
export TOKEN=$(curl -v --insecure --user developer:developer --header "X-CSRF-Token: xxx" --url "https://oauth-openshift.apps-crc.testing/oauth/authorize?response_type=token&client_id=openshift-challenging-client" 2>&1 | grep -oP "access_token=\K[^&]*")
And Use the token then in, e.g., oc login:
oc login --token=$TOKEN --server=https://api.crc.testing:6443
Related
I want to test that a PUT to an endpoint (products/:id) works, but when I try
curl -X PUT -d listing_id_created=True localhost:3000/products/27
it gives ActionController::InvalidAuthenticityToken, which I now realise is the expected result (since there's no authenticity token provided since the PUT is coming from curl and curl doesn't know anything about it).
So my question is how do I run some simple curl PUTs (or any other verbs) to check that endpoints work correctly? Is the only solution to simply disable/skip the authenticity token?
I am trying to trigger a Jenkins build by post request from my Windows computer, using the following commands.
First, I obtain the crumb using...
curl http://JENKINS-URL/crumbIssuer/api/xml?xpath=//crumb
The response is this...
<crumb>string-of-digits<\crumb>
I then use the command
curl -u USERNAME:PASSWORD -X POST -H "Jenkins-Crumb:string-of-digits" http://JENKINS-URL/job/my-job/buildWithParameters?token=my-token
The username is correct, the password is correct, the crumb is exactly the string of digits that I got from the first command (everything between the crumb> at the beginning and <\crumb> at the end), the token matches the token I have specified in Jenkins. But still, I am getting the response
Error 403 No valid crumb was included in the request
In Configure Global Security -> CSRF Protection, I also have "Enable proxy compatibility" checked and am using "Default Crumb Issuer".
Does it look like I am missing anything here? Forgetting any steps? Improperly formatted commands? Anything else you might be able to think of?
I have followed the steps and formatted my command like the solution here as well, and still no luck..
https://linuxacademy.com/community/posts/show/topic/28964-no-valid-crumb-was-included-in-the-request
In addition to this, I have also tried saving the cookies from the first request to get the crumb, and then I pass the file I saved the cookies to into the second curl command, see below for the command. Still I am getting the 403 No valid crumb...
Here are the commands using cookies...
wget --keep-session-cookies --save-cookies cookies.txt --auth-no-challenge --user admin --password my_password -q --output-document - http://JENKINS-URL/crumbIssuer/api/xml?xpath=//crumb
curl --cookie cookies.txt -u admin:my_password -H "JenkinsCrumb: string-of-digits-from-stdout" -X POST http://JENKINS-URL/job/my-job/buildWithParameters?token=my-token
I have finally discovered the answer. I thought that I could specify my actual account password, but it turns out that I needed to use an API token to authenticate. Using the API token instead of my password on the above commands with cookies allowed me to trigger my build remotely.
enter image description here
I finally resolved this issue by selecting the jenkins own user database in security realm
An API I am using is telling me to make a GET request as follows:
curl -s \
-X GET \
--user "$MJ_APIKEY_PUBLIC:$MJ_APIKEY_PRIVATE" \
https://api.mailjet.com/v3/REST/template/$template_ID/detailcontent
and I am trying to convert this to Rails' NET::HTTP. I've tried adding a req.body, adding ?user=TOKEN and keep getting a 401 Unauthorized response. I've tested it in curl and my credentials are valid.
How do I include the --user "$MJ_APIKEY_PUBLIC:$MJ_APIKEY_PRIVATE" portion in my GET request?
--user in curl is used for server authentication. In your case, after --user you provide username and password separated with colon. This will be used for the http basic authentication.
Since you know that, you can check how to do the basic authentication with NET::HTTP here or here.
I'm currently using Codefresh's free private registry to store my images. I am using CircleCI for my Docker builds, so from there I use docker login so I can push and pull to the Codefresh registry. This works fine. However, Codefresh only lists images in its web app for which it did the building, so I cannot easily see a catalogue of them. I suspect this is by design, so that users stick with Codefresh for CI. However, if possible, I would like to list the images I have in the registry.
I assume that this registry is a basic v2 version of the standard Docker Registry. So, I have had a hunt around, and found this console utility to manage images. However, it does not seem to work for me. I don't know whether this is because registry management tools are still immature generally (web searches indicate they were added much later, and there are lot of folks for whom this simple task has become a substantial undertaking) or whether there is something unusual with Codefresh.
Here is what I am trying:
reg -d -r r.cfcr.io -u myusername -p cfaccesstoken
(The -d is for debug).
This results in:
2017/10/18 11:24:43 registry.ping url=https://r.cfcr.io/v2/
2017/10/18 11:24:44 registry.catalog url=https://r.cfcr.io/v2/_catalog
2017/10/18 11:25:53 registry.catalog url=https://r.cfcr.iohttps://r.cfcr.io/v2/_catalog?n=1000&last=davigsantos/davigsantos/codeflix
FATA[0075] Get https://r.cfcr.iohttps//r.cfcr.io/v2/_catalog?n=1000&last=davigsantos/davigsantos/codeflix: dial tcp: lookup r.cfcr.iohttps on 127.0.1.1:53: no such host
The penultimate line contains a container name I don't recognise - I hope it is a public one, and not one I should not be seeing!
The last line indicates some sort of fatal error, with all sorts of URL fragments mashed together in ways that definitely won't work.
I have also discovered that dropping the access token makes no difference; the output is exactly the same:
reg -d -r r.cfcr.io -u myusername
What else can I try? I am running Mint Linux and would be fine with swapping to another utility if necessary. Is there something wrong about the way I am issuing this command, or is Codefresh running a non-standard registry that is incompatible with standard API calls?
Update
It looks like Codefresh also has their own API, though it does not seem to be documented as far as I can tell. Running the get operation produces this error:
Failed to authenticate request because no token was provided
That's encouraging, so I will try to find out how to provide a token in a curl call; the Swagger web interface does not seem to permit it. However I am conscious that, if I can get the API working, it may not list my Docker images anyway, since they were not built by Codefresh.
Update 2
I have found some articles on the Codefresh blog that hint how to authenticate on the API, the format is a header thus:
--header "x-access-token: (token)"
However I get this error:
{"status":401,"code":"2401","name":"UNAUTHORIZED_ERROR","message":"Failed to authenticate request because token is not valid","context":{}}
The token I am using is the same one as I use for docker login, which works. I notice I was not specifying my username, so I am also adding this curl option:
-u (user):(token)
As you can see, I am close to trying random things now, as there does not seem to be official help online.
Update 3
Prompted by a comment below, it seems that Docker maintains an access token after login is used, in ~/.docker/config.json.
I therefore tried this:
reg -d -r r.cfcr.io -u myusername -p dockeraccesstoken
(Note how cfaccesstoken has been changed to dockeraccesstoken).
This returns much more quickly now (as opposed to appearing to hang), but returns nothing:
2017/10/24 10:56:16 registry.ping url=https://r.cfcr.io/v2/
2017/10/24 10:56:18 registry.catalog url=https://r.cfcr.io/v2/_catalog
Repositories for r.cfcr.io
You need to first generate a token on CodeFresh User Settings
Next I will show you how to login from terminal
curl -H 'Host: r.cfcr.io' -H 'User-Agent: ItsMe' -H 'Authorization: Basic <AUTH>' --compressed 'https://r.cfcr.io/v2/token?account=xxxx&scope=repository%3A<user>/<name>%3Apush%2Cpull&service=r.cfcr.io'
You can get the <AUTH> by running
echo <account>:<token> | base64
This will return you a big token
{"expires_in":43200,"issued_at":"2017-10-24T03:34:54.118390368-07:00","token":"APMm...P91"}%
Now you can use that token to make a docker API call
$ curl -IH 'Host: r.cfcr.io' -H 'User-Agent: ItsMe' -H 'Authorization: Bearer APMm+...aRL' -X HEAD 'https://r.cfcr.io/v2/<user>/<user>/testci/blobs/sha256:c7fefcc4c54c63f32b5422ede095793eb5ff0f45ad7a22861974ec9b61e7749b'
HTTP/1.1 200 OK
Docker-Distribution-API-Version: registry/2.0
Content-Length: 22488057
Date: Tue, 24 Oct 2017 10:42:23 GMT
Content-Type: text/html
Server: Docker Registry
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
For registry search access below should have worked
curl -H 'Host: r.cfcr.io' -H 'User-Agent: ItsMe' -H 'Authorization: Basic ' --compressed 'https://r.cfcr.io/v2/token?account=xxxx&scope=registry%3Acatalog%3Asearch&service=r.cfcr.io'
But it doesn't and neither does registry:catalog:* for scopes. This is like hitting a target with blind folds and not even knowing in which direction we stand. You best bet is to get them to disclose some information to you
Edit-1: Getting the catalog
So I finally managed to crack it, but the result is a bit unfortunate. I ended up getting catalog/repositories of every single user. I checked, you still can't pull those repo. So their docker images are safe as such. These look like so:
Edit-2: Fetching all repos
After we notified Codefresh of this situation, here is how to do the fetch. One needs to first generate a token using below:
curl -H 'Host: r.cfcr.io' -H 'User-Agent: MyAgent' -H 'Authorization: Basic .....' --compressed 'https://r.cfcr.io/v2/token?account=<account>&service=r.cfcr.io'
And then using the same we can query the complete catalog:
curl -H "User-Agent: ItsMe" -H 'Authorization: Bearer <TOKEN>' 'https://r.cfcr.io/v2/_catalog?n=10' --compressed
I have a partial answer, and I think it is useful enough on its own for others approaching the same difficulty. I got some kind assistance via the chat widget on Codefresh's support page.
Proprietary API
Regarding the Codefresh API, I had not spotted that the domain g.codefresh.io is the same as their control panel. So, to authenticate, all I had to do was log into the control panel - oops! That reveals the call to https://g.codefresh.io/api/images and a much more complicated access token than I've been using - perhaps it is OAuth. It looks like this:
curl \
-X GET \
--header "Accept: application/json" \
--header "x-access-token: (36chars).(143chars).(22chars)-(4chars)-(15chars)" \
"https://g.codefresh.io/api/images"
As I considered in the question, it does look like the /api/images endpoint is for Codefresh builds only. So, this is out for me - I want everything in the registry.
Open API
So, turning to Docker's open standard for accessing registry, my support contact said this:
if you want to connect to cfcr.io through docker CLI, you can.
The username is your username at Codefresh. The password is a token you can generate at user settings -> "GENERATE" button at the Codefresh registry section.
That's what I've been doing so far, and it works for login, push and pull. It does not seem to work for the reg utility in ls mode though. Either I am still doing something wrong, or there are restrictions at Codefresh as to how the private registry may be used.
Unfortunately, it's not possible to use search and catalog commands for Codefresh Registry. The Codefresh registry (r.cfcr.io) is based on Google Container Registry (aka GCR) and Google Registry does not implement v1 Docker registry API and _catalog function.
Once they do, things will work for Codefresh Registry too.
The version I use is neo4j-enterprise-2.2.0-M02
My question is :
How can I configure a user (like add a new user, change the password ,etc) in backend or browser, instead of REST API? Can I do it via neo4j-shell? imagine that I am a DBA, it is not very convenient to do this by REST API.
Any help will be greatly appreciated!
You can use the browser instead of the API. Just go to http://localhost:7474 (or whatever IP to which the web console is bound) and you will be prompted to change the password. Once authenticated, use the command :server change-password to change the password again.
It is not yet possible to create multiple user accounts within the system.
You can use the command :help server to see available authentication commands.
Although still utilizing the REST API, I'll throw the cURL option out there to anyone who doesn't have access to a web browser (AWS instance, for example):
$ curl -H "Content-Type: application/json" -X POST -d '{"password":"WHATEVER THE PASSWORD IS"}' -u neo4j:neo4j http://localhost:7474/user/neo4j/password
Another option is to modify the auth file directly and restart neo. Doing this, you can even change the username!
Run
find / -name dbms
For me this gave one hit:
/var/lib/neo4j/data/dbms/auth
Save this code as build_auth_string.sh:
#!/bin/bash
DEFAULT_IFS="$IFS"
SALT_LEN=32
# either read from stdin or use the argument
if [ -z "$1" ]; then
read INPUT
else
INPUT="$1"
fi
if [ -z "$INPUT" ]; then
echo "correct format <uname:pass>"
exit
fi
IFS=':'
read -a UNAME_PASS <<< "$INPUT"
UNAME="${UNAME_PASS[0]}"
PASS="${UNAME_PASS[1]}"
# representing the password in hex format like \xAB\x0C etc
# HEX_PASS=$(echo -n $PASS | xxd -p | awk '{print toupper($1);}' | sed -r 's/(.{2})/\\x\1/g')
HEX_PASS=$(echo -n $PASS | hexdump -v -e '"\\\x" 1/1 "%02X"')
# echo $HEX_PASS
# create the salt and store it in hex format
SALT=$(cat /dev/urandom | tr -dc 'a-f0-9' | fold -w $SALT_LEN | head -n 1)
# SALT="28FD26AD92D6D2D8820E969F3F3732B4"
HEX_SALT=$(echo -n $SALT | sed -r 's/(.{2})/\\x\1/g')
# calculate the sha256 sum of the salt and password value
# need to split the output because the output ends with a hyphen
IFS=' '
read -a PASSWORD_HASH_ARRAY <<< $(printf $HEX_SALT$HEX_PASS | sha256sum)
PASSWORD_HASH="${PASSWORD_HASH_ARRAY[0]}"
# echo "$UNAME;$PASS;$SALT"
# echo "$PASSWORD_HASH"
# and print out the auth string
COMBINED=$(echo -n "$PASSWORD_HASH,$SALT" | awk '{print toupper($1);}')
echo "$UNAME:SHA-256,$COMBINED:"
IFS="$DEFAULT_IFS"
The code for the above came from https://github.com/artsince/docker-neo4j-auth/blob/master/build_auth_string.sh - im posting it here just encase..
And then just run the above script like
build_auth_string.sh myUsername:myP#ssw0rd
Copy/paste that into your auth file replacing whatever was there before, and restart neo4j :)
A fresh install of Neo4j 2.2.x has a user 'neo4j', with an initial password 'neo4j'. You are required to change the password before you can do anything.
It's easy to do this from the command line, by calling httpie to interact with the REST API. For example, to set a new password of 'foobar', run this command:
http -a neo4j:neo4j POST http://localhost:7474/user/neo4j/password password=foobar
If you want to reset the password and you dont know the old password :
then for Windows user
go to this path:
C:\Users\xyz\Documents\Neo4j\default.graphdb\dbms
and delete that auth file.
Restart the neo4j they will again ask to set the username and password!!
by default
username:neo4j
password:neo4j
Currently it's not possible to configure authorization using neo4j-shell. As you've mentioned the REST API is the way to go. Using a convenient REST client this is very easy.
My tools of choice is either postman (a plugin for chrome browser) or httpie for the command line. E.g. with httpie changing the password for a user is as simple as:
http localhost:7474/user/neo4j/password password=neo4j new_password=mypass
Be aware that password (and other authorization settings) are not automatically distributed in a cluster, see the manual how to copy over settings between instances.
For Mac users, version 2.3.1 of Neo4J, best way to reset credentials is to remove the file with credential information and start the service again.
Steps to follow
Find where the file that contains credentials is located from the browser console (localhost:7474). Go to Star (Favourites)->System->Server configuration
Search for dbms.security.auth_store.location property to see where it points to. In my case it was /Users/felipe/Documents/Neo4j/default.graphdb/./dbms/auth
Delete that file.
Start the service again and go to the console again (localhost:7474).
By default you will be asked to set the password for the user neo4j.
I hope it helps.
To elaborate on felipe's response (since I do not have enough rep points to comment):
I stopped the server, I deleted the auth files in BOTH:
DBROOT\data\auth
DBROOT\dbms\auth
Restarted the server, and connected to it via the localhost:7474, used the default username/password (neo4j/neo4j) and then it prompted me for a new password.
On Neo4j 4.0+, you can run:
$ cypher-shell
If it's the first time you connect, you can enter neo4j as user and password and you will be prompted to set a new password.
If you want to change the password afterwards, you can write in the Cypher shell:
:server change-password