I am trying to use the spring-cloud-dataflow-rest-client v2.6.0 in an application to launch spring cloud tasks. I followed the instructions on this page https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#appendix-identity-provider-azure to secure Spring-cloud-dataflow-server using Azure AD. However, I am unable to get the setup that was provided for dataflow-shell to work with the SCDF rest client. I know shell internally uses SCDF-rest-client. So, not sure why it won't work for me.
Which properties should I use if my application which uses SCDF-rest-client is to launch tasks like the shell?
I tried with the following properties but I keep getting an invalid scope error.
-Dspring.cloud.dataflow.client.authentication.client-id=yhas7wqh-2a5d-4795-babb-b6213f896b52
-Dspring.cloud.dataflow.client.authentication.client-secret=asjajd8hhsasajdassakja
-Dspring.cloud.dataflow.client.authentication.oauth2.client-registration-id=Batch-Launcher
-Dspring.cloud.dataflow.client.authentication.token-uri=https://login.microsoftonline.com/d8bb2fd3-e835-4d68-b9db-7402a9bf39f1/oauth2/v2.0/token
-Dspring.cloud.dataflow.client.authentication.scope=api://dataflow-server/dataflow.deploy,api://dataflow-server/dataflow.view,offline_access
-Dspring.cloud.dataflow.client.authentication.oauth2.username=abcddemo#afdemo12.onmicrosoft.com
-Dspring.cloud.dataflow.client.authentication.oauth2.password=abcdPwd
-Dspring.cloud.dataflow.client.authentication.basic.username=abcddemo#afdemo12.onmicrosoft.com
-Dspring.cloud.dataflow.client.authentication.basic.password=abcdPwd
The exception that I get
Caused by: org.springframework.security.oauth2.core.OAuth2AuthorizationException: [invalid_scope] AADSTS70011: The provided request must include a 'scope' input parameter. The provided value for the input parameter 'scope' is not valid. The scope api://dataflow-server/dataflow.deploy offline_access is not valid.
Can someone from SCDF team update the Azure provider docs to include also how one can use SCDF rest client like shell to invoke SCDF API.
Related
I'm trying to connect to Microsoft Power BI using the Mule Power BI connector but the connector configuration is very confusing with respect to OAuth. Has anyone successfully connected to Power BI with Mule? Here is my config:
<power-bi-rest:config name="Power_Bi_Rest_Connector_Config" doc:name="Power Bi Rest Connector Config" doc:id="d9dbee0f-50fe-46ca-977e-2b3c6ce1ff3b" >
<power-bi-rest:oauth2-authorization-code-connection >
<power-bi-rest:oauth-authorization-code consumerKey="xxx" consumerSecret="yyy" authorizationUrl="https://login.microsoftonline.com/69f2560d-6d2a-4768-989e-d35ed85aeab2/oauth2/v2.0/authorize" accessTokenUrl="https://login.microsoftonline.com/69f2560d-6d2a-4768-989e-d35ed85aeab2/oauth2/v2.0/token" scopes="api://96eb9d1c-6dfb-4acf-9950-d4af04038693/Mule"/>
<power-bi-rest:oauth-callback-config listenerConfig="HTTP_Listener_config" callbackPath="https://login.microsoftonline.com/common/oauth2/nativeclient" authorizePath="https://login.microsoftonline.com/common/oauth2/nativeclient" />
</power-bi-rest:oauth2-authorization-code-connection>
</power-bi-rest:config>
I have not specifically used Power BI but I can tell the what the issue probably is.
The oAUthCallbackConfig is where you will need to make the changes.
External Callback URL: While configuring the OAuth credentials in your MS account, you must have provided a callback URL that should be pointing to your Mule application. You need to put that URL here.
Callback Path: Here you need to mention where the Mule App should be expecting to get the call back with the authorization code. For example of you have configured external path as https://localhost:8082/oauth/callbackthis parameter could be oauth/callback. (Note that you will need to consider the base path that you may have configured in the http listener)
Authorize path: This should not be a URL pointing to a Microsoft api, but it is the path that you will use to hit you mule app to start the oAuth dance. You can put something like /oauth/authorize. Then when you try https://localhost:8082/oauth/authorize the mule app will trigger the oAuth dance and will redirect you to the login screen of everything else is setup correctly.
I have an RDF-file stored on my server. The file or at least the file-content should be uploaded to a remote GrapbDB over API.
On the documentation there are two ways to do this. The first one is uploading it to server files and then loading it to GraphDB. Here the problem is, that I am not the owner of the server, GraphDB is running. So I can`t upload it to server files. Or is there maybe another API for that?
The other way is providing a public API on my server and then trigger GraphDB to download the file from my server. But my API must be protected with credantials or JWT. But I don´t know how to set the credantials in the API-Call.
Isn`t there a way to upload a simple graph to a repository?
There is a browser-based user interface in GraphDB that allows you to import from local files. If this is allowed on the server you are connecting to, and you only need to do this once then I think this would be the quickest route to go.
If you want to upload a local file to GraphDB using dotNetRDF, then I would advise you to use the SPARQL 1.1 graph store protocol API via the VDS.RDF.Storage.SparqlHttpProtocolConnector as described here. The base URL you need to use will depend on the configuration of the server and possibly also on the version of GraphDB that it is running, but for the latest version (9.4) the pattern is: <RDF4J_URL>/repositories/<repo_id>/rdf-graphs/service
The connector supports HTTP Basic Authentication (which is one of the options offered by GraphDB) so if you have a user name and password you could try the SetCredentials method on the connector to specify those credentals and if necessary force the use of HTTP Basic Authentication by setting the global options property VDS.RDF.Options.ForceHttpBasicAuth to true.
We have implemented an OAuth2 Authorization Server (and Identity Provider).
Now we want to perform load tests to gage system performance.
The concrete problem I have now, is that I want to load test an Authorization Code Flow.
I have been using JMeter so far. But I have no clue how to provide an endpoint for the required redirect_uri, to complete the flow.
Is there any way to do this at all, or am I out of luck? Google was no help.
If JMeter can't do it, is there a tool that can?
Provide whatever redirect_uri parameter you want, i.e. http://localhost:8081/oauth/login and fetch the required parameters from the response Location header using Regular Expression Extractor
Of course the sampler will fail as most probably you don't have anything running on your machine which could serve as a redirect target so either use HTTP Mirror Server to provide a stub or just mark the sampler as successful by adding Response Assertion to it and ticking Main sample and sub-samples and Ignore Status boxes
I am trying to create a Google Assistant for my Raspberry Pi in Kotlin. I implemented a OAuth flow using the so called "device flow" proposed in this IETF draft, since my Raspberry shall later just expose a web interface and does not have any input devices or graphical interfaces.
Google does support this flow (of course) and I obtain a valid access token with user consent in the end. For testing purpose I also tried a default authorization flow that will just forward the user to localhost, as it is normally done but it did not solve the problem.
I tested the access token using this tool and it confirmed validity of scope and token. So the token itself should work.
Scope is: https://www.googleapis.com/auth/assistant-sdk-prototype as documented here
This actually does not point to any valid web resource but is referenced in every documentation.
Then I tried to stream audio data to the assistant SDK endpoint using the gRPC provided java stubs. As took a third party reference implementation as a guide how to authenticate the rpc stub. But neither the reference implementation nor my own one works. They both report
io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
The stub is authenticated this way:
embeddedAssistantStub.withCallCredentials(
MoreCallCredentials.from(OAuth2Credentials
.newBuilder()
.setAccessToken(
myAccessToken,
myAccessTokenExpirationDate))
.build()))
and the authenticated request is performed like this:
val observer = authenticatedEmbeddedAssistantStub.converse(myStreamObserverImplementation)
observer.onNext(myConfigConverseRequest)
while(more audio data frames available) {
observer.onNext(myAudioFrameConverseRequest)
}
observer.onCompleted()
(I prefixed pseudo variables with "my" for clarity, they can consist of more code in the actual implementation.)
I even contacted the author of this demo implementation. He told me, last time he checked (several months ago) it was working perfectly fine. So I finally ran out of options.
Since the client implementation I took as reference used to work and I do actually authenticate the stub (although the error message suggests the opposite) Probably, either my valid access token with correct scope is not suitable chosen for the assistant API (though I followed the suggestions of google) or the API servers had a change not properly documented in the getting started articles by google.
So: Did anyone ran in the same problem and know how to fix it? I have the project on github. So if anyone needs the broken source code, I can do a temporary commit that produces the error.
Note, to save some works for mods: This issue referres to this and this question, both unresolved and using different languages but describing a similar problem.
Well, seems I was right about my second assumption: The error is server side. Here is the github issue, let's just wait for the fix.
https://github.com/googlesamples/assistant-sdk-python/issues/138
I'm using spring-cloud-dataflow with taskcloud module but I've some trouble to lunch a simple example in container.
tiny example 6.3 writing code then I've deploy it
but when I try to execute it throw me an
java.lang.IllegalArgumentException: Invalid TaskExecution, ID 1 not found
at org.springframework.util.Assert.notNull(Assert.java:134)
at org.springframework.cloud.task.listener.TaskLifecycleListener.doTaskStart(TaskLifecycleListener.java:200)
In my evaluation I've used Spring boot example
and for run in scd I've add #EnableTask and configured ad sqlserver datasource but it doesn't works.
I'm insisting on using spring cloud data flow cause I've read that spring batch admin is at end-of-life, but 2.0.0.BUILD-SNAPSHOT works
well and a tiny examples works as opposed to what happens in spring cloud data flow with #task annotation.
Probably is my misundestand but could you please provide me a tiny example where or address me some url ?
Referencing https://docs.spring.io/spring-cloud-dataflow/docs/current-SNAPSHOT/reference/htmlsingle/#configuration-rdbms, datasource arguments has to be passed to the data flow server and data flow shell(if using) in-order for the cloud data flow to persist the execution/task/step related data in the required datasource.
Ex from the link for a MySQL datasource(similar can be configured for SQL Server):
java -jar spring-cloud-dataflow-server-local/target/spring-cloud-dataflow-server-local-1.0.0.BUILD-SNAPSHOT.jar \
--spring.datasource.url=jdbc:mysql:<db-info> \
--spring.datasource.username=<user> \
--spring.datasource.password=<password> \
--spring.datasource.driver-class-name=org.mariadb.jdbc.Driver
This error:
Invalid TaskExecution, ID 1 not found
Can be about the SCDF's datasource, in general, SCDF cannot find the Task Execution table in its own database, not application database
You might fix it by adding database driver or fixing url connection string, point to SCDF's database
This issue below might help
How to properly compile/package a Task for Spring Cloud Data Flow