I'm using SCDF 2.5.1 deployed locally via docker-compose and my software sends commands to the SCDF server via the java DSL.
Let's say I create a stream such that
file > :queue
:queue > ftp
where file and ftp are docker deployed apps.
My question is, how can I get the logs for file and ftp?
So far the closest thing I've come up with is
Map<String, String> attributes = scdf.runtimeOperations().streamStatus(streamName).getContent()
.stream().flatMap(stream -> stream.getApplications().getContent().stream()
.filter(app -> app.getName().equals(appName))
.flatMap(appStatus -> appStatus.getInstances().getContent().stream()
.map(AppInstanceStatusResource::getAttributes)))
.findFirst().orElse(Collections.emptyMap());
String logLocation = attributes.get("stdout")
and then mounting logLocation and reading it as a file.
Is there a more elegant solution?
The Java DSL (and subsequently the SCDF REST client) doesn't have log retrieval operation as part of it's REST operations. There is an REST endpoint you can hit the SCDF server to get the logs of stream.
If you would like to contribute, you can submit a proposal/PR here: https://github.com/spring-cloud/spring-cloud-dataflow/pulls
Related
I am trying to set up mass ingestion in IICS. This will be a streaming ingestion task. The source is REST V2.
According to the documentation, I can't provide an absolute path to the Swagger file for this connection. Instead, I need a hosted URL.
I tried hosting the Swagger file on a server that has the Informatica Cloud secure agent installed. When I create a connection everything works.
But when I try to add a connection for mass ingestion I get following error:
What is interesting, I also tried hosting this file on a VM in Azure, and when I try to access this file from a server using an internet browser it works. I can also see requests on the web server, but when I create mass ingestion and define the source I still get an error and I can't see any requests for the Swagger file on the web server.
What is wrong?
My requirement is to get a file from SFTP and make it available client to download. I am forcefully asked to do that using spring cloud data flow.
In the documentation, I saw that there is an SFTP to the JDBC File Ingest tutorial (https://dataflow.spring.io/docs/recipes/batch/sftp-to-jdbc/).
So my question is can we transfer a file through spring cloud data flow rather than reading the file and inserting it into the databae?
Thanks,
Dasun.
Yes you can. It’s similar to the sftp to jdbc example which downloads the file to a shared file system from which the batch job reads it. You can create a simple pipeline like sftp | s3 or sftp l file or sftp l sftp, depending on your specific use case.
I am new to spring cloud data flow. I am trying to build a simple http source and rabbitmq sink stream using SCDF stream app.The stream should be deployed on OSCF (Cloud Foundry). Once deployed, the stream should be able to receive HTTP POST Request and send the request data to RabbitMQ.
So far, I have downloaded Data Flow Server using below link and push to cloud foundry. I am using Shall application from my local.
https://dataflow.spring.io/docs/installation/cloudfoundry/cf-cli/.
I also have HTTP Source and RabbitMQ Sink application which is deployed in CF. RabbitMQ service is also bound to sink application.
My question - how can I create a stream using application deployed in CF? Registering app requires HTTP/File/Maven URI but I am not sure how can an app deployed on CF be registered?
Appreciate your help. Please let me know if more details are needed?
Thanks
If you're using the out-of-the-box apps that we ship, the relevant Maven repo configuration is already set within SCDF, so you can freely already deploy the http app, and SCDF would resolve and pull it from the Spring Maven repository and then deploy that application to CF.
However, if you're building custom apps, you can configure your internal/private Maven repositories in SCDF/Skipper and then register your apps using the coordinates from your internal repo.
If Maven is not a viable solution for you on CF, I have seen customers resolve artifacts from s3 buckets and persistent-volume services in CF.
I trying to manage my streams on spring cloud data flow with skipper server.
I followed the instruction here:
https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#getting-started-deploying-spring-cloud-dataflow
The app registration and stream definition/deployment goes quite well, but after I undeploy the deployed stream, I can't see any stream on the dashboard any more.
The dashboard shows an error instead:
Could not parse Skipper Platform Status JSON:null
I have to restart the scdf server and skipper server in order to see my stream definition again.
The version of the components are:
scdf local server: 1.6.0.RELEASE
skipper server: 1.0.8.RELEASE
metrics collector: kafka-10-1.0.0.RELEASE
Some operation details:
I registered my app using scdf shell in skipper mode.
I defined and depolyed my stream on the scdf dashboard. I undeployed the stream via the stop button on the dashboard, too.
How should I solve this problem?
We have recently observed this on our side, too, and it has been fixed! [see spring-cloud/spring-cloud-dataflow#2361]
We are preparing for a 1.6.1 GA release, but in the meantime, please feel free to pull the 1.6.1.BUILD-SNAPSHOT from Spring repo and give it a go.
If you see any other anomaly, it'd be great to have a bug report on the matter.
I am able to configure an agent for window but i have a confusion regarding connectivity between web servers logs with agent.
1: How to connect web server with agent ?
2: while starting flume.bat file. It is generating flume.log file in which i am getting below mentioned Exception.
org.apache.flume.conf.ConfigurationException: No channel configured for sink: hdfssink
at org.apache.flume.conf.sink.SinkConfiguration.configure(SinkConfiguration.java:51)
atorg.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:661)
1.The data flow is as below
your application (or web server) --> source --> channel --> sink
Now, the data can flow from your webserver to the source either by "pull" mechanism or "push" mechanism. In your case, you can either tail the webserver logs or use a spooling source.
2.This looks like a misconfiguration issue. You need to post your config file to figure out the issue