Yesterday I upgraded my development environment to Spring Cloud Dataflow 1.2.0 and all of my sink/source apps dependencies.
I have two main issues:
javaOpts: -Xmx128m is not longer being picked up, so locally deployed apps have the default Xmx value.
Here is the format of my previously working Dataflow yaml config.
See full here: https://pastebin.com/p1JmLnLJ
spring:
cloud:
dataflow:
applicationProperties:
stream:
spring:
cloud:
deployer:
local:
javaOpts: -Xmx128m
Kafka config options like ssl.truststore.location etc. are not being read correctly. Another stackoverflow post indicated these must be marked like this "[ssl.truststore.location]". Is there some documented working yaml config or list of breaking changes with 1.2.0? The file based authentication block was also moved, and I was able to figure that one out.
Yes, It looks like a bug in Spring Cloud Local Deployer to consider the common application properties passed via args. Created https://github.com/spring-cloud/spring-cloud-deployer-local/issues/48 to track this.
Related
Are there examples of configuring SpecificAvroSerdes (or any schema registry-based serdes - JsonSchema and Protobuf) in spring-kafka that allow leveraging some of the autoconfiguration (based on yaml or properties files).
There are a few similar questions in SO like How to use Spring-Kafka to read AVRO message with Confluent Schema registry?
But I want to be a bit specific on Kafka Streams serdes and declarative configuration of the Serdes.
Thank you
Looks like Kafka Streams can be configured with the default Serde.
See StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG and StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG.
According Spring Boot convention we can provide any arbitrary properties from YAML file: https://docs.spring.io/spring-boot/docs/2.4.0/reference/html/spring-boot-features.html#boot-features-kafka-extra-props
So, probably your SpecificAvroSerdes can be configured in the application.properties like this:
spring.kafka.streams.properties[default.value.serde]=io.confluent.kafka.streams.serdes.avro.SpecificAvroSerdes
Recently, I upgraded from SCDF 1.7.3 to SCDF 2.1.2 for cloud foundry. Also, I am using skipper (I have to with 2.x). There are two main problems I am facing:-
Buildpack given as a property in the SCDF server environment is not being used to deploy stream applications. Following is the env key that I am using:-
SPRING_CLOUD_DATAFLOW_STREAM_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[xxx]_DEPLOYMENT_BUILDPACK. This has no effect at all.
Even though I set SPRING_CLOUD_DATAFLOW_STREAM_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[xxx]_DEPLOYMENT_ENABLE_RANDOM_APP_NAME_PREFIX to false skipper still generates random prefix for these applications.
I am not sure what I am doing wrong. Any advice will be of great help.
There are no stream platform properties with the prefix SPRING_CLOUD_DATAFLOW_STREAM_PLATFORM_CLOUDFOUNDRY in Spring Cloud Data Flow as the stream deployments are managed by Spring Cloud Skipper. Hence, you need to use the Skipper properties for stream deployment-related configurations.
The correct properties to use in this case are:
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[xxx]_DEPLOYMENT_ENABLERANDOMAPPNAMEPREFIX: false
SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_CLOUDFOUNDRY_ACCOUNTS[xxx]_DEPLOYMENT_BUILDPACK:
I'm trying to migrate my SCDF local server deployments to the k8s-based solution. But I've got some problems when handling the server configuration of the kafka broker-list for the apps.
I followed the instructions here: https://docs.spring.io/spring-cloud-dataflow-server-kubernetes/docs/1.7.2.RELEASE/reference/htmlsingle
and downloaded the sample configuration from : https://github.com/spring-cloud/spring-cloud-dataflow-server-kubernetes at branch v1.7.2.RELEASE
Because we've already deployed a kafka cluster, I'd like to configure the broker- and zk-nodes in the server-config-kafka.yaml file so that we could use the same kafka cluster.
I configured my environmentVaribales like this:
deployer:
kubernetes:
environmentVariables: >
SPRING_CLOUD_STREAM_KAFKA_BINDER_BROKERS='172.16.3.192:9092,172.16.3.193:9092,172.16.3.194:9092',
SPRING_CLOUD_STREAM_KAFKA_BINDER_ZK_NODES='172.16.3.192:2181,172.16.3.193:2181,172.16.3.194:2181'
but got an error when trying to deploy my SCDF stream:
Invalid environment variable declared: 172.16.3.193:9092
How should I configure it to make it work?
Thanks in advance.
Remove the > in your YAML
That's creating a block string, not a map of environment variables. In YAML, how do I break a string over multiple lines?
Also, if using CoreDNS in kubernetes, you should probably be using something like kafka.default.cluster.local for the value, rather than IP addresses, and similar for Zookeeper
Is there any way to enable a grafana plugin within its configuration files?
I am using grafana v5 or v4
It looks like you have to login and then click then enable button
I found a workaround for my problem running docker with a volume that would map the default sqlite db for grafana in /var/lib/grafana/grafana.db
this would keep any configuration, dashboard, datasource firstly set up in the web interface
You can use the Zabbix Plugin for Grafana.
You need to install and enable the plugin, then configure the Zabbix datasource:
URL: http://yourserver/zabbix/api_jsonrpc.php
access: proxy or direct, depends on the reachability of Zabbix and grafana servers
username: use a dedicated one, with the required read permissions
And you're ready to create dashboards by referencing groups, applications, hosts and items.
Here you can find the getting starded guide, it's quite complete.
The regexp & templating features are really powerful, I advise to read it carefully.
Trying Grails 3.0.0.M1, a lot has changed!
How would I run-app --https and how would I configure it that way also for the production-environment-war?
Many thanks!
First of all, Grails 3.0.0.M1 is not production ready, if thinking about making the move.
Major changes for Grails 3 include:
Built on top of Spring Boot
Introduction of Gradle as the build system
And many more (which is not relevant to this question)
In order to setup SSL for Spring Boot app, server.ssl.* setting has to be added to application.properties or application.yml file.
In Grails 3 it would correspond to application.yml under grails-app/conf. Below changes (as an example) would be required:
server:
port: 8443
ssl:
key-store: classpath:keystore.jks
key-store-password: secret
key-password: another-secret
which is equivalent to Configuring SSL in Spring Boot.
However, documentation is limited right now and is being worked on. So let Grails 3 brew more until its GA where it can be used in production environment.
Grails community will be more than happy if any issues/problems/enhancement is reported in Grails JIRA