I am using the Spring Cloud Dataflow local server/deployer and upgrading from 1.1.4 to 1.2.3. I've found my stream configs such as value.deserializer are not being passed into the sink process correctly. I have tried both:
spring.cloud.stream.kafka.bindings.input.consumer.configuration.value.deserializer
spring.cloud.stream.kafka.binder.configuration.value.deserializer
Neither are working. There seem to be some significant changes with the local deployer. The entire argument list from both common configs and stream configs are visible from the process level now (e.g. using ps). However the stream configs not actually working. Configs work fine when running the Spring Cloud stream app locally.
Any idea what is happening or how I can troubleshoot this? Since Dataflow server is deploying the app it is difficult to debug.
What you've listed above aren't SCDF or Deployer specific overrides. They are Spring Cloud Stream properties, and from your description, it is unclear how you're passing it to the apps from SCDF.
Once after the applications are deployed (via SCDF), you can access the /env or /configprops endpoints to verify whether those overrides are applied. If they aren't applied, you can start with the application and its dependencies - compare it with the old stack. Hopefully, that will provide some insights.
If it is still unclear, you can share a sample that reproduces the problem.
Related
I have a c++ application that uses log4cxx for logging. log4cxx configuration will be done via XML file where the logging level and different loggers can be enabled and disabled. with installation running in VM it was easy to make the necessary modifications as needed by getting into VM and changing the XML file manually. but now we are going to run the application as a docker image which will run in the cloud, so the question is how to make modifications around the logger level as and when needed. I did try to search for this before asking here, but the solutions which are mentioned are java based, like spring boot admin, etc. which is not suitable here.
The issue
I am using the same container (similar resources) on 2 projects -- production and staging. Both have custom domains setup with cloud flare DNS and are on the same region. Container build is done in a completely different project and IAM is used to handle the access to these containers. Both project services have 80 concurrency and 300 seconds time out for all 5 services.
All was working good 3 days back but from yesterday almost all cloud run services on staging (thankfully) started throwing 503 randomly and for most requests. Some services were not even deployed for a week. The same containers are running fine on production project, no issues.
Ruled out causes
anything to do with Cloudflare (I tried the URL cloud run gives it has the issue of 503)
anything with build or containers (I tried the demo hello world container with go - it has the issue too)
Resources: I tried giving it 1 GB ram and 2 cpus but the problem persisted
issues on deployment (deploy multiple branches - didn't work)
issue in code (just routed traffic to old 2-3 days old revision but still issue was there)
Issue on service level ( I used the same container to create a completely new service, it also had the issue)
Possible causes
something on cloud run or cloud run load balancer
may some env vars but that also doesn't seem to be the issue
Response Codes
I just ran a quick check with vegeta (30 secs with 10 rps) same container on staging and production for a static file path and below are the responses:
Staging
Production
If anyone has any insights on this it would help greatly.
Based on your explanation, I cannot understand what's going on. You explained what doesn't work but didn't point out what works (does your app run locally? are you able to run a hello world sample application?)
So I'll recommend some debugging tips.
If you're getting a HTTP 5xx status code, first, check your application's logs. Is it printing ANY logs? Is there logs of a request? Does your application have and deployed with "verbose" logging setting?
Try hitting your *.run.app domain directly. If it's not working, then it's not a domain or dns or cloudflare issue. Try debugging and/or redeploying your app. Deploy something that works first. If *.run.app domain works, then the issue is not in Cloud Run.
Make sure you aren't using Cloudflare in proxy mode (e.g. your DNS points to Cloud Run; not Cloudflare) as there's a known issue about certificate issuance/renewals when domains are behind Cloudflare, right now.
Beyond these, if a redeploy seems to solve your problem, maybe try redeploying. It could be very likely some configuration recently became different two different projects.
See Cloud Run Troubleshooting
https://cloud.google.com/run/docs/troubleshooting
Do you see 503 errors under high load?
The Cloud Run (fully managed) load balancer strives to distribute incoming requests over the necessary amount of container instances. However, if your container instances are using a lot of CPU to process requests, the container instances will not be able to process all of the requests, and some requests will be returned with a 503 error code.
To mitigate this, try lowering the concurrency. Start from concurrency = 1 and gradually increase it to find an acceptable value. Refer to Setting concurrency for more details.
I have a project in which I use the following technologies
mysql
spring boot (including spring jdbc template)
docker
kubernetes
gitlab (ci/cd).
As per the spring boot standard, I pass database credentials in as environment variables. These credentials obviously need to be secure, out of source code, and vary based on region, so I would like to keep them externalized as far away from the application as possible (I would like to keep them stored as gitlab secrets, if possible). As of right now, I can successfully pass them as environment variables from my yaml file defining both my kubernetes deployment and service, which gets picked up by my Dockerfile and passed to my spring boot app. Is there any way to pass from gitlab to kubernetes (avoiding kubectl run, which is bad practice and going to be deprecated)?
Thanks,
Connor
In case anyone was wondering, I solved this using envsubstr. If anyone is curious, I can paste the full example below
We deployed a rails app in Google Cloud Run using their managed platform. The app is working fine and it is able to serve requests.
Now we want to get access to the rails console of the deployed app. Can anyone suggest a way to achieve this?
I'm aware that currently, Cloud Run supports only HTTP requests. If no other way is possible I'll have to consider something like rails web console
I think you cannot.
I'm familiar with Cloud Run but I'm not familiar with rails.
I assume you'd need to be able to shell into a container in order to be able to run IRB. Generally, you'd do this by asking the runtime (Docker Engine, Kubernetes, Cloud Run) to connect you to the container so that you could do this.
Cloud Run does not (appear) to permit this. I think it's a potentially useful feature request for the service. For those containers that contain shells, this would be the equivalent of GCE's gcloud compute ssh.
Importantly, your app may be serviced by multiple, load-balanced containers and so you'd want to be able to console into any of these.
However, you may wish to consider alternatives mechanisms for managing your app: monitoring, logging, trace etc. These mechanisms should provide you with sufficient insight into your app's state. Errant container instances should be terminated.
This follows the concept of "pets vs. cattle" whereby, instead of nurturing individual containers (is one failing?), you nurture the containers holistically (is the service comprising many containers failing?)
For completeness, if you think that there's an issue with a container image that you're unable to resolve through other means, you could run the image elsewhere (e.g. locally) where you can use IRB. Since the same container image will behave consistently wherever it's run, you should be able to observe the issue using IRB locally too.
I sucessfully managed to complete with my Grail's new website and now i want to run it on my machine to acess it everywhere. I would like to know what is the best approach to do that, i'm a bit noob in what concerns servers and connections. Any help would be aprecciated.
Jm
You need a Java servlet container. The quickest and easiest approach is probably to use Jetty on a Linux system -- most will allow you to install it directly. This allows you run a deployed system from a "war" file. Basically, you would do something like:
grails war
To build the war file, and then copy that (it'll be created in the target directory, with a name based on your application name and version) to the webapps directory for Jetty. On my (Ubuntu) system, that's /var/lib/jetty/webapps/. If you name it root.war, it will serve URLs from the root of the server configuration (typically on port 8080), so http://myhost.mydomain:8080/ will be passed to your Grails application. You'll need to restart Jetty to pick up the new root.war file, e.g., sudo /etc/init.d/jetty restart
Jetty is not the most flexible or powerful servlet container, but it is probably the quickest way to get something up and running, and you can explore more powerful alternatives (Tomcat being an obvious one) as you have time.
Unless you need persistent storage in your app (e.g. end users uploading pictures, etc), you can use Cloud Foundry. It's free for now and makes deployment very easy.
http://blog.springsource.com/2011/04/12/one-step-deployment-with-grails-and-cloud-foundry/
I've been using is via STS and it works really well. There's just one really annoying thing and that's the fact that STS ends up doing a build war with every change.
There's a cloud foundry plugin that gives you command line deployment.
http://www.grails.org/plugin/cloud-foundry
Or there's a Cloud Foundry UI plugin that "provides a simple monitoring UI for your applications deployed to Cloud Foundry. It uses the same API as the Cloud Foundry plugin but is more convenient since runs in a web browser."
http://www.grails.org/plugin/cloud-foundry-ui