I'm trying to set up Spring Boot Config Server for a few microservices from the Bitbucket repo(earlier while learning tutorial I did the same using Github, it worked, but I think bitbucket is a little different).
Have added dependencies in Config-Server build.gradle
implementation 'org.springframework.cloud:spring-cloud-config-server'
Annotation added in the application class
#EnableConfigServer
Modified .properties file to .yml, and configured like this
server:
port: 8012
spring:
application:
name: config-server
cloud:
config:
server:
git:
uri: ssh://bitbucket.org/project/repo.git
ignore-local-ssh-settings: true
clone-on-start: true
passphrase: passphrase
private-key: |
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,***************************
*************************************************
*************************************************
-----END RSA PRIVATE KEY-----
Have gone through multiple other SO posts.
Tried with these ssh formats
git#bitbucket.org:project/repo.git >> failed to send channel request
ssh://git#bitbucket.org:project/repo.git >> extra / showed up because of jgit
ssh://bitbucket.org/project/repo.git >> giving auth fail
Now I'm confused.
Did try adding this
strict-host-key-checking: false
nothing changed
Any help what am I missing?
Related
I am using SCDF deployment on k8s and trying to add a new Task Application from our internal Maven repo. By default, SCDF seems to only lookup in the [springRepo] repository. I followed the documentation to add a new maven repo here .
Since the documentation only talks about CloudFoundry example, I added these lines to application.yaml section based on my understanding.
spring:
cloud:
dataflow:
task:
platform:
local:
accounts:
localDev:
********
datasource:
uri: xxx
*********
maven:
remote-repositories:
repo1:
url: https://repo1
auth:
username: user1
password: pass1
snapshot-policy:
update-policy: daily
checksum-policy: warn
release-policy:
update-policy: never
checksum-policy: fail
While adding the app I used the syntax : maven://:[:[:]]:. However, when I launch the task, it fails with error : Failed to resolve maven Resource XXX at configured Remote Repository : [springRepo]
How can I override it to search in my newly added repo.. why SCDF still only searching in default [springRepo]? Appreciate any help.
The property prefix is maven.remote-repositories but what you have is spring.maven.remote-repositories.
You need to specify:
spring:
cloud:
dataflow:
task:
platform:
local:
accounts:
localDev:
********
datasource:
uri: xxx
*********
maven:
remote-repositories:
repo1:
url: https://repo1
...
Please note that the Kubernetes deployment works with containers rather than maven jar artifacts and hence, you need to have your apps registered with the app's URI using docker: prefix.
I have some issues with the format of the provisioning file. I have some service account file that looks like this
{
"type": "service_account",
"project_id": "my-project",
"private_key_id": "XXXXX_my_private_key_id_XXXXXXX",
"private_key": "-----BEGIN PRIVATE KEY-----\nXXXXXXX_my_private_key___\nXXXXX_another_line_here_XXXXX\nXXXXXX_final_line_XXXXXX==\n-----END PRIVATE KEY-----\n",
"client_email": "my-project#company.iam.gserviceaccount.com",
"client_id": "123456",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/my-project%40company.iam.gserviceaccount.com"
}
And the provisioning file that I put in /etc/grafana/provisioning/datasources/all.yaml
Which looks like this
apiVersion: 1
# list of datasources to insert/update depending
# what's available in the database
datasources:
# <string, required> name of the datasource. Required
- name: bigquery-project
type: doitintl-bigquery-datasource
access: proxy
isDefault: true
jsonData:
authenticationType: jwt
clientEmail: my-project#company.iam.gserviceaccount.com
defaultProject: my-default-project
tokenUri: https://accounts.google.com/o/oauth2/token
secureJsonData:
privateKey: "-----BEGIN PRIVATE KEY-----\nXXXXXXX_my_private_key___\nXXXXX_another_line_here_XXXXX\nXXXXXX_final_line_XXXXXX==\n-----END PRIVATE KEY-----\n"
version: 2
readOnly: false
But when I clicked on save and test I got some errors. I think I miss-formated the provisioning file.
I tried to upload the service account file using the UI and the test passed, I was able to query BQ. However when I did this, I couldn't find any file in /etc/grafana/provisioning/datasources to get and example of it.
I'm executing a custom Grafana image in a docker container
### file system hierarchy of the project
.
├── Dockerfile
└── provisioning
├── dashborads
└── datasources
└── all.yaml
### Docker file
ARG GRAFANA_VERSION=6.5.3
FROM grafana/grafana:$GRAFANA_VERSION
ENV GF_AUTH_DISABLE_LOGIN_FORM "true"
ENV GF_AUTH_ANONYMOUS_ENABLED "true"
ENV GF_AUTH_ANONYMOUS_ORG_ROLE "Admin"
ENV GF_INSTALL_PLUGINS "doitintl-bigquery-datasource 1.0.4"
The way I'm running the container
docker run -d -p 3000:3000 -v $(pwd)/provisioning:/etc/grafana/provisioning massy/custom-grafana
I'm providing the provisioning file via a volume.
What's wrong with the provisioning file ?
When we add a datasource in Grafana, isn't the provisioning file updated automatically ? (if not created then it will be ?)
How could I get the logs when I test that bigquery plugin ?
Edit
When I tried to add a dummy SQL query in "new dashbord" section this is what I get
lvl=eror msg="Failed to get access token" logger=data-proxy-log error="private key should be a PEM or plain PKCS1 or PKCS8; parse error: asn1: structure error: tags don't match (16 vs {class:0 tag:28 length:110 isCompound:true}) {optional:false explicit:false application:false private:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} pkcs1PrivateKey #2"
t=2020-01-22T10:02:18+0000 lvl=info msg=Requesting logger=data-proxy-log url=https://www.googleapis.com/bigquery/v2/projects/undefined/queries
t=2020-01-22T10:02:18+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=1 uname= method=POST path=/api/datasources/proxy/1/bigquery/v2/projects/undefined/queries status=401 remote_addr=172.17.0.1 time_ms=55 size=304 referer="http://localhost:3000/dashboard/new?panelId=2&edit&fullscreen&orgId=1&gettingstarted"
I did a docker logs on the container
This is the correct format for the provisioning file
apiVersion: 1
datasources:
- name: bigquery-project
type: doitintl-bigquery-datasource
access: proxy
isDefault: true
jsonData:
authenticationType: jwt
clientEmail: my-project#company.iam.gserviceaccount.com
defaultProject: my-default-project
tokenUri: https://accounts.google.com/o/oauth2/token
secureJsonData:
privateKey: |
-----BEGIN PRIVATE KEY-----
XXXXXXX_my_private_key___
XXXXX_another_line_here_XXXXX
XXXXXX_final_line_XXXXXX
-----END PRIVATE KEY-----
version: 2
readOnly: false
There is a difference between the two provisioning files:
https://grafana.com/grafana/plugins/doitintl-bigquery-datasource?version=1.0.4
https://github.com/doitintl/bigquery-grafana#example-of-provisioning-a-file
The one on GitHub has the correct format
i've a jwt token from an aws cognito login process. this token needs to be sent from the application to some other apis (via cookie or bearer header, i've not yet decided).
the receiving apis has been proxied behind nginx/openresty, so i'm thinking to validate the jwt token before the upstream
i'm using this library (the seems the most updated)
https://github.com/cdbattags/lua-resty-jwt
then i followed these steps:
download the jwks file from my account
wget https://cognito-idp.eu-west-1.amazonaws.com/eu-west-1_5zCVSiMVH/.well-known/jwks.json
convert the jwks to pem with jwks2pem
cat jwks.json| jwks2pem > key.pem
then this code
local jwt = require "resty.jwt"
local key = [[ -----BEGIN PUBLIC KEY-----
(content of key.pem)
-----END PUBLIC KEY-----
]]
local jwt_token = ""
local jwt_obj = jwt:load_jwt(jwt_token)
local verified = jwt:verify_jwt_obj(key, jwt_obj)
ngx.say(cjson.encode(jwt_obj))```
the code fails:
$ resty jwt.lua
{"valid":false,"reason":"invalid algorithm: RS256","verified":false}
where i'm wrong?
ok, the problem is the key.
i've successfully obtained the pem key from jwks with this other tool https://www.npmjs.com/package/jwk-to-pem
the validation now works
Has methods secure Config Server by oauth2 token ?
I plan to implement spring cloud config-server by oauth2 token,so client-server can fetch property by :
cloud:
config:
uri: http://user:password#localhost:8888
Is it feasible ?
but ... i met some problems .
I start a demo https://github.com/keryhu/spring-oauth2-config-server.git
It contains four services :
1 : eureka : start first,and can implement service register and discovery,it has no oauth2 enviroment.
2 : auth-server : JWT OAuth2 server configuration ,start secondly.
#SessionAttributes("authorizationRequest")
#EnableResourceServer
#EnableDiscoveryClient
and inmemory user :
security:
user:
password: password
3: config-server : start thirdly
#EnableDiscoveryClient
#EnableConfigServer
#EnableResourceServer
and in application.yml :
spring:
cloud:
config:
server:
git:
uri: https://github.com/keryhu/cloud-config
security:
oauth2:
resource:
jwt:
keyValue: |
-----BEGIN PUBLIC KEY-----
....
-----END PUBLIC KEY-----
4: pc-gateway : is a client-server,also a ui server. start lastly
When i test the secured uri: http://localhost:8080/hello, the page was redirected to
http://localhost:9999/uua/login
After entering "user:password",it redirects back
http://localhost:8080/hello
So i think the oauth-server and oauth-client is fine.
but.. i also set the following configuration in bootstrap.yml
cloud:
config:
uri: http://user:password#localhost:8888
When starting pc-gateway service, Fetching config from server has 401 Unauthorized errors :
INFO 954 --- [main] c.c.c.ConfigServicePropertySourceLocator : Fetching config from server at: http://localhost:8888
WARN 954 --- [main] c.c.c.ConfigServicePropertySourceLocator : Could not locate PropertySource: 401 Unauthorized
Need help ! thanks !
I am using omniauth and twitter login for my site. However, whenever I try to login, it gave this error:
Started GET "/auth/failure?message=service_unavailable" for 98.83.218.118 at 2011-11-12 11:27:58 -0500
Processing by SessionsController#failure as HTML
Parameters: {"message"=>"service_unavailable"}
The only clue I have is that it might be an SSL error.
However, I have no idea on how to debug the cause of the error.
Here is my current configuration for twitter:
Rails.application.config.middleware.use OmniAuth::Builder do
provider :twitter, 'REDACTED', 'REDACTED'
end
It looks like ssl issue in my project.
For development mode you can off ssl. Then you can work without this issue.
add this in develoment.rb:
OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE
And about how fix SSL certificates:
Twitter API SSL Root CA Certificate
I don't really like the solution suggested in at twitter dev support (as linked by Yahor Zhuchkou), which suggest downloading just a bunch of certificates from an unsecure server.
And while turning of verify peer will work in production that isn't really a solution. What you need to do is to point omniauth to the correct PEM file, which should contain something like, the Verisign Root Certificate which backs Twitter's own certificate (i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority - G2/OU=(c) 1998 VeriSign, Inc. - For authorized use only/OU=VeriSign Trust Network):
-----BEGIN CERTIFICATE-----
MIIDAjCCAmsCEH3Z/gfPqB63EHln+6eJNMYwDQYJKoZIhvcNAQEFBQAwgcEx
CzAJBgNVBAYTAlVTMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjE8MDoGA1UE
CxMzQ2xhc3MgMyBQdWJsaWMgUHJpbWFyeSBDZXJ0aWZpY2F0aW9uIEF1dGhv
cml0eSAtIEcyMTowOAYDVQQLEzEoYykgMTk5OCBWZXJpU2lnbiwgSW5jLiAt
IEZvciBhdXRob3JpemVkIHVzZSBvbmx5MR8wHQYDVQQLExZWZXJpU2lnbiBU
cnVzdCBOZXR3b3JrMB4XDTk4MDUxODAwMDAwMFoXDTI4MDgwMTIzNTk1OVow
gcExCzAJBgNVBAYTAlVTMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjE8MDoG
A1UECxMzQ2xhc3MgMyBQdWJsaWMgUHJpbWFyeSBDZXJ0aWZpY2F0aW9uIEF1
dGhvcml0eSAtIEcyMTowOAYDVQQLEzEoYykgMTk5OCBWZXJpU2lnbiwgSW5j
LiAtIEZvciBhdXRob3JpemVkIHVzZSBvbmx5MR8wHQYDVQQLExZWZXJpU2ln
biBUcnVzdCBOZXR3b3JrMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDM
XtERXVxp0KvTuWpMmR9ZmDCOFoUgRm1HP9SFIIThbbP4pO0M8RcPO/mn+SXX
wc+EY/J8Y8+iR/LGWzOOZEAEaMGAuWQcRXfH2G71lSk8UOg013gfqLptQ5GV
j0VXXn7F+8qkBOvqlzdUMG+7AUcyM83cV5tkaWH4mx0ciU9cZwIDAQABMA0G
CSqGSIb3DQEBBQUAA4GBAFFNzb5cy5gZnBWyATl4Lk0PZ3BwmcYQWpSkU01U
bSuvDV1Ai2TT1+7eVmGSX6bEHRBhNtMsJzzoKQm5EWR0zLVznxxIqbxhAe7i
F6YM40AIOw7n60RzKprxaZLvcRTDOaxxp5EJb+RxBrO6WVcmeQD2+A2iMzAo
1KpYoJ2daZH9
-----END CERTIFICATE-----
(If you're using Ubuntu you may find it here: /etc/ssl/certs/Verisign_Class_3_Public_Primary_Certification_Authority_-_G2.pem )
Don't know why omniauth or ruby openssl implementation isn't finding this, but you can explicitly link to this pem file with the following option:
provider :twitter, 'REDACTED', 'REDACTED', {
:client_options => {:ca_file => '/etc/ssl/certs/Verisign_Class_3_Public_Primary_Certification_Authority_-_G2.pem'}
}