netty dubbojson read timeout - docker

i has dev and test two env that delpoy with k8s cluster.
dev:
rpc frame : dubbo 2.7.0
protocol:dubbo
JVM version:open jdk1.8
operate system : redhat 8.5
kernel.sysrq=1
vm.swappiness=10
net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-iptables=1
net.ipv4.neigh.default.gc_thresh1=4096
net.ipv4.neigh.default.gc_thresh2=6144
net.ipv4.neigh.default.gc_thresh3=8192
test:
rpc frame : dubbo 2.7.0
protocol:dubbo
JVM version:open jdk1.8
operate system : redhat 7.9
kernel config:
kernel.sysrq=1
net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-iptables=1
net.ipv4.neigh.default.gc_thresh1=4096
net.ipv4.neigh.default.gc_thresh2=6144
net.ipv4.neigh.default.gc_thresh3=8192
with my project updata lib, but netty was still in 4.1.25version,when i deploy to test on k8s pod , it will throw an timeout exception via dubbojson.
however it was work well on dev.
or update netty version to 4.1.71 on redhat 7.9 , the timeout exception was gone.
the tcpdump that i get of netty4.1.25 on test env show everthing was normal
ie:
docker container (psh ack)---provider
provider (psh ack)---docker container
docker (ack) --- provider
tcpdump show data had been reccived and ack to provider,but netty was timeout to read data.
when i use two server ,one os is redhat7.9, anther os is redhat8.5,both installed docker
i pull image(problematic) delpoy on both server , the timeout not exists anymore.
has anyone can help me.
begin, i was think that ,may be it was netty version too lower has compatiable operation system version.
after test, that was wrong.

Related

DTLSv1_listen unable to accept second client in a docker container

I'm experiencing an issue with OpenSSL/DTLS server.
Environment: docker container based on CentOs7
OpenSSL version: OpenSSL-1.1.1d
A DTLS server (non-blocking) using DTLSv1_Listen having a UDP socket with SO_REUSEADDR is unable to accept a second
client connection when it has already been accepted a client connection and serving it.
When the first client has finished, the second client connection is accepted.
I have used the dtls_udp_echo.c (taken from http://web.archive.org/web/20150617012520/http://sctp.fh-muenster.de/dtls-samples.html ) to carry out the test and reproduce the issue.
The test application has been compiled and executed within a docker container, having CentOS7 as base image, but the behaviour has been noticed with other base images OS too (e.g. Redhat, Ubuntu, Debian, SLES).
The same application running on a bare metal works without any issue.
Is there any known compatibility issue between Docker and OpenSSL/DTLS?
Is there any specific configuration to be done to overcome this issue?
Best Regards

Ssh.net Conncetion timeout after 30000 milliseconds in Docker

I am using Ssh.net nuget package in a .net core app to connect to remote Ubuntu servers on AWS.
I have dockerized my app and it works perfectly on local Windows envionment and a local ubuntu server.
Same docker image does not work if deployed on Aws Lightsail or other server providers like upcloud.com and vultr.com.
Below are the error details:
Connection failed to establish within 30000 milliseconds.
at Renci.SshNet.Abstractions.SocketAbstraction.Connect(IPEndPoint remoteEndpoint, TimeSpan connectTimeout) at Renci.SshNet.Session.SocketConnect(String host, Int32 port) at Renci.SshNet.Session.Connect() at Renci.SshNet.BaseClient.Connect()
I have tried turning off firewalls on all of the tested platforms but it doesn't make a difference.
Any ideas what I should do to resolve the issue?

IntelliJ unable to connect to SonarQube Server running in Docker

I'm trying to connect to a SonarQube server running on my machine in a Docker.
I installed the SonarLint plugin in IntelliJ, when I create the new connection to the server, given the token to the plugin, it shows:
Failed to connect the server. Please check the configuration. Error:
Fail to request http://localhost:9000/api/system/status
this URL works in the browser, giving back
{"id":"BF41A1F2-AWlcxJRKJ9-nWLU3WeGl","version":"7.6.0.21501","status":"UP"}
in the SonarLint log it says:
Caused by: java.net.ProtocolException: Unexpected status line:
I just pulled the last sonarqube image from docker repository and run it with only the port forwarding option (-p 9000:9000)
My environment:
SonarQube server version: 7.6.0.21501 ;
Java: 1.8.0_181
Docker: 2.0.0.3 (31259)
IntelliJ: Community 18.3.5
I found a workaround.
use the machine name instead localhost.

Event Sourcing in Microservices Using Spring Cloud and Reactor

I am currently doing a research on building Microservices using spring boot and Docker. Luckily I found this blog
Event Sourcing in Microservices Using Spring Cloud and Reactor by Kenny Bastani. You can find the github project source code here.
I really want to up and run this example because it has the functionalities/technologies that I was looking for.
I am running Ubuntu desktop 16.04 OS and already installed the following software mentioned on the said blog.
Maven 3
Java 8
Docker, Docker Machine
Docker Compose
Then executed the following scripts. This is also based on the blog. I just reduced the virtualbox-memory & virtualbox-disk-size to avoid computer freeze.
$ docker-machine create event-source-demo --driver virtualbox
--virtualbox-memory "1000" --virtualbox-disk-size "20000"
$ eval "$(docker-machine env event-source-demo)"
$ sh run.sh
After running the above scripts, The following snippet errors occurs.
2017-03-18 22:33:14.025 INFO 5136 --- [ main] com.netflix.discovery.DiscoveryClient : Getting all instance registry info from the eureka server
2017-03-18 22:33:14.151 ERROR 5136 --- [ main] c.n.d.s.t.d.RedirectingEurekaHttpClient : Request execution error
com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: Connection refused (Connection refused)
at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:187) ~[jersey-apache-client4-1.19.1.jar:1.19.1]
at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123) ~[jersey-client-1.19.1.jar:1.19.1]
at com.netflix.discovery.EurekaIdentityHeaderFilter.handle(EurekaIdentityHeaderFilter.java:27) ~[eureka-client-1.4.4.jar:1.4.4]
at com.sun.jersey.api.client.Client.handle(Client.java:652) ~[jersey-client-1.19.1.jar:1.19.1]
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) ~[jersey-client-1.19.1.jar:1.19.1]
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) ~[jersey-client-1.19.1.jar:1.19.1]
at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509) ~[jersey-client-1.19.1.jar:1.19.1]
at com.netflix.discovery.shared.transport.jersey.AbstractJerseyEurekaHttpClient.getApplicationsInternal(AbstractJerseyEurekaHttpClient.java:194) ~[eureka-client-1.4.4.jar:1.4.4]
a
It seems like created docker images not linked to each other or was not able to establish connections.
Please let me know if you need more log details.

VisualVM over Putty SSH-tunnel

I'm trying to profile remote java app, actually it is a gameserver. It works normally on my local machine (windows XP x64 with JDK1.7.0_02 x64), but behaves very wierd on the production server (CentOS with JDK1.7.0_03 i586).
I've done a lot of searching and found out that I should use VisualVM for this task. So VisualVM works great on local machine, but there is no hangs on local machine, i need profiling in production environment with real payload. I started jstatd on the remote machine with arguments
jstatd -J-Djava.security.policy=jstatd.all.policy -J-Djava.rmi.server.logCalls=false &
with the policy file
grant codebase "file:/usr/java/jdk1.7.0_02/lib/tools.jar" {
permission java.security.AllPermission;
};
then I started my java application like this
java -server -Dcom.sun.management.jmxremote\
-Dcom.sun.management.jmxremote.port=4000\
-Dcom.sun.management.jmxremote.ssl=false\
-Dcom.sun.management.jmxremote.authenticate=false\
-jar /home/pinballSocketServer/pinballSocketServer.jar
Both application and jstatd are launched with root priveledges.
and VisualVM didn't manage to connect to remote host. But on the remote host i see the following log, while VisualVM is running and remote host added:
Feb 16, 2012 7:11:52 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(3)-217.16.27.195: [217.16.27.195: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
Feb 16, 2012 7:11:56 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(3)-217.16.27.195: [217.16.27.195: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
Feb 16, 2012 7:12:00 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(3)-217.16.27.195: [217.16.27.195: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
Feb 16, 2012 7:12:04 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(3)-217.16.27.195: [217.16.27.195: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
After further googling, I found out that I need to use ssh tunneling. I configured putty in the following way
http://www.advancedigital.ru/show/putty_config.jpg
and VisualVM as this
http://www.advancedigital.ru/show/visualvm_config.jpg
Adter munipulations above VisualVM connects to remote host, but I can only see the threads summary chart and profiler is inactive.
I've seen some recommendations that jvms on both machines should be similar and have the same platform (x86 or x64) but i've already tried profiling from another machine (windows 7 x86 with JDK1.7.0_03 x86), and have the same result.
I've also tried this, but get the same result again.
VisualVM over ssh
How can I get this profiling to work?
Even though this has been answered, I managed to do it so I will add my approach:
I used the following source:
https://bowerstudios.com/node/731
I did this in windows using git bash that has ssh command. You can also do using cygwin or pure minggw.
1) Run ssh tunnel command in command prompt (I do this in git bash/MINGGW32).
ssh -D 9010 -p 22 root#IP -v
2) Run your application on server with JMX options (one liner)
java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -jar application.jar
3) Run visualVM through socks proxy to connect (one liner)
visualvm -J-Dnetbeans.system_socks_proxy=localhost:9010
-J-Djava.net.useSystemProxies=true
4) Actually add your JMX remote connection in visualVM
Also available: http://maythesource.com/2013/12/04/connecting-to-jmx-from-visualvm-using-ssh-tunnel/ (will be updated with more info over time).
Note that if the server has X11 libraries installed you can run JVisualVM remotely and just have Putty forward the X11 connection to a X11-server running on your local machine.
If you do not have an X11-server available, Xming - http://www.straightrunning.com/XmingNotes/ - works fine for this. If your computer is secure, you can run without access control making it much easier to get up and running.
Note that X11 communication is quite verbose. Use "blowfish"-cypher and ask for compression.
I would start with debugging just jstatd connection and leave JMX connection for later. From you wrote it is not clear why you should use ssh tunneling. Is there a firewall on your remote host or why do you think that you need to use ssh tunneling.
One additional note - it is not true that jvms on both machines should be similar and have the same platform (x86 or x64).

Resources