I'm trying to profile remote java app, actually it is a gameserver. It works normally on my local machine (windows XP x64 with JDK1.7.0_02 x64), but behaves very wierd on the production server (CentOS with JDK1.7.0_03 i586).
I've done a lot of searching and found out that I should use VisualVM for this task. So VisualVM works great on local machine, but there is no hangs on local machine, i need profiling in production environment with real payload. I started jstatd on the remote machine with arguments
jstatd -J-Djava.security.policy=jstatd.all.policy -J-Djava.rmi.server.logCalls=false &
with the policy file
grant codebase "file:/usr/java/jdk1.7.0_02/lib/tools.jar" {
permission java.security.AllPermission;
};
then I started my java application like this
java -server -Dcom.sun.management.jmxremote\
-Dcom.sun.management.jmxremote.port=4000\
-Dcom.sun.management.jmxremote.ssl=false\
-Dcom.sun.management.jmxremote.authenticate=false\
-jar /home/pinballSocketServer/pinballSocketServer.jar
Both application and jstatd are launched with root priveledges.
and VisualVM didn't manage to connect to remote host. But on the remote host i see the following log, while VisualVM is running and remote host added:
Feb 16, 2012 7:11:52 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(3)-217.16.27.195: [217.16.27.195: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
Feb 16, 2012 7:11:56 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(3)-217.16.27.195: [217.16.27.195: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
Feb 16, 2012 7:12:00 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(3)-217.16.27.195: [217.16.27.195: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
Feb 16, 2012 7:12:04 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(3)-217.16.27.195: [217.16.27.195: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
After further googling, I found out that I need to use ssh tunneling. I configured putty in the following way
http://www.advancedigital.ru/show/putty_config.jpg
and VisualVM as this
http://www.advancedigital.ru/show/visualvm_config.jpg
Adter munipulations above VisualVM connects to remote host, but I can only see the threads summary chart and profiler is inactive.
I've seen some recommendations that jvms on both machines should be similar and have the same platform (x86 or x64) but i've already tried profiling from another machine (windows 7 x86 with JDK1.7.0_03 x86), and have the same result.
I've also tried this, but get the same result again.
VisualVM over ssh
How can I get this profiling to work?
Even though this has been answered, I managed to do it so I will add my approach:
I used the following source:
https://bowerstudios.com/node/731
I did this in windows using git bash that has ssh command. You can also do using cygwin or pure minggw.
1) Run ssh tunnel command in command prompt (I do this in git bash/MINGGW32).
ssh -D 9010 -p 22 root#IP -v
2) Run your application on server with JMX options (one liner)
java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -jar application.jar
3) Run visualVM through socks proxy to connect (one liner)
visualvm -J-Dnetbeans.system_socks_proxy=localhost:9010
-J-Djava.net.useSystemProxies=true
4) Actually add your JMX remote connection in visualVM
Also available: http://maythesource.com/2013/12/04/connecting-to-jmx-from-visualvm-using-ssh-tunnel/ (will be updated with more info over time).
Note that if the server has X11 libraries installed you can run JVisualVM remotely and just have Putty forward the X11 connection to a X11-server running on your local machine.
If you do not have an X11-server available, Xming - http://www.straightrunning.com/XmingNotes/ - works fine for this. If your computer is secure, you can run without access control making it much easier to get up and running.
Note that X11 communication is quite verbose. Use "blowfish"-cypher and ask for compression.
I would start with debugging just jstatd connection and leave JMX connection for later. From you wrote it is not clear why you should use ssh tunneling. Is there a firewall on your remote host or why do you think that you need to use ssh tunneling.
One additional note - it is not true that jvms on both machines should be similar and have the same platform (x86 or x64).
Related
Morning,
I am in the process of moving our plastic server to a new machine. All going well so far, users, configs good, databases still transferring (will take a while).
However, I've tripped at a basic hurdle. Clients on our network are unable to see the server and it appears to be linked to good ol' windows firewall - as on disabling windows firewall on the server the client is then able to see the server.
Info:
Server OS: Windows Server 2019 Standard v. 1809
Client OS: Windows 10 Pro, Windows 11 Pro
Port used: 8087
Old server has been disconnected from the network
Troubleshooting so far
Using Telnet in command line on client computer confirms the port is open on the server's IP
Firewall rule created opening TCP 8087 over Domain, Private, Public - inbound and outbound
Tried reverting to the default port 8084
Any help much appreciated
I have a Windows 7 PC with Forefront TMG proxy. There is a VirtualBox virtual machine installed with Ubuntu 16.04 (64-bit) guest OS. I'm not a seasoned Unix-user, so I tried to configure Ubuntu's network through GUI window.
After several tries I wound up with these settings:
https://i.stack.imgur.com/in4yR.jpg
All works fine now. I can use browser and git and do various network stuff. But!
I have a Rails application, which uses a remote PostgreSQL database. When I start my Rails server, it throws this exception:
Puma caught this error: could not connect to server: Network is unreachable
Is the server running on host "this_is_the_host_url" and accepting
TCP/IP connections on port 5432?
(PG:ConnectionBad)
This applications works fine on the other PC with similar Windows 7 and VB Ubuntu, but without a proxy. Also I tried to reach the remote DB-server from my host Win7 with PGAdmin - all works fine, even with proxy. The issue is somewhere between my Ubuntu guest OS and my Windows host OS.
Should I configure proxy settings in Ubuntu in a different way?
Any other way to resolve this issue?
Traceroute shows this:
1 10.0.2.2 (10.0.2.2) 0.638 ms 0.577 ms 0.549 ms
2 10.0.2.2 (10.0.2.2) 29.369 ms !N 53.000 ms !N 75.644 !N
As I can get it, 10.0.2.2 is my host OS ip.
I searched through the web, but all I could get belongs to the opposite side: to reach to the proxied VB Ubuntu, not from it.
I have installed jenkins as windows service but every time after starting the machine i have to execute it through command line using $ java -jar jenkins.war
I am not able to directly open it into web browser at http://localhost:8080/
Am i missing something? Any help will be really appreciated.
Copy original 'jenkins.war' into 'jenkins' directory and restart service
Clear the Windows event viewer logs for Application and System
Please ensure your services are running and up
Firewall Issue:The easiest way to track down firewall issues is to use tcpdump. Just run the following command on the Jenkins server, which is trying to connect to the slave.
More Info on firewall part:
By default, Windows Firewall prevents the TCP connections necessary to make this mechanism work. The firewall on the slave must allow the following exceptions (see List of TCP&UDP port numbers):
TCP Port 135 (DCE/RPC Locator service)
TCP Port 139 (NetBIOS Session Service)
TCP Port 445 (Windows shares)
C:\WINDOWS\system32\dllhost.exe (dllhost.exe seems to use a random port number)
C:\WINDOWS\system32\javaw.exe (Jenkins also uses a random port number)
File and Printer sharing (TCP 139, TCP 445, UDP 137, UDP 138 (possibly only a subset of these is required))
The issue is fixed now. I have changed the service properties. In service properties -> Log On->Select Local system account and select the checkbox for allow the service to interact with desktop
I have a java process on a linux server, which runs with this option: -Dcom.sun.management.jmxremote
So I cannot just connect to this process via jconsole running on my local pc (because neither port nor -Dcom.sun.management.jmxremote.ssl=false options are set up).
But still, how can I connect to the application and run some operations over some of its MBeans? It this possible? I have a ssh access to the server and would be able to run it "locally" on the server (but not changing the options unfortunately)
According to JMX documentation the -Dcom.sun.management.jmxremote option
Enables the JMX remote agent and local monitoring via JMX connector published on a private
interface used by jconsole. The jconsole tool can use this connector if it is executed by
the same user ID as the user ID that started the agent. No password or access files are
checked for requests coming via this connector.
The naming is a bit unfortunate because it in fact enables the local monitoring only.
Since you can not change the options but can access the server via SSH the only option is to use X server forwarding (ssh -X ...) and run jconsole (or better yet jvisualvm which has specific optimisations for running remotely).
I am new to Java and I'm facing a problem in connecting a Remote Host to the JVisualVM.
I've searched the Internet and followed all the steps mentioned there but still am not able to resolve the issue. The steps I followed are:
I started the jstatd on the remote server by first creating a jstatd.all.policy file in the $JAVA_HOME/bin. The file contained: grant codebase "file:${java.home}/../lib/tools.jar" { permission java.security.AllPermission;};
I started the Jstatd as jstatd -J-Djava.security.policy=jstatd.all.policy
I started the Java application on the remote host as :
java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9000 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false application_name
I then started as instance of the JVisualVM on my local machine and as I added the remote host, it got connected but i wasn't able to see any of the Java processes.
Can anyone please help me with this.
Thanks.
I encountered similar problems when connecting to Glassfish application server. See solutions that worked for me as they can be same for You:
Try setting on your application:
-Djava.rmi.server.hostname=*Remote_Server_External_IP_Address*
The mentioned IP address should be server external IP (may sound silly but it worked for me). The main problem in my case was JMX pointing to the localhost and looping. In config files the exact IP address should be set to the remote host. I described it as 'Problem 2' In my blog: handling connection problems
If Firewall block is an issue then I recommend trying XMing with SSH tunnel (which is simple to set). Here is instruction, if You encounter problems setting it:
Remote use of VisualVM with Xming (my blog)
Biggest advantage of using XMing is that it will work almost always when SSH is enabled. You just have to place VisualVM files on the remote host and run it from command line. XWindow will show VisualVM Window on Your local computer.
There is a chance that it is VisualVM issue - try using some other tool just to verify what is wrong. I recommend JConsole. It works similar to VisualVM and I also described details on my blog
You need to start jstatd with the additional option that points to the server's external IP or hostname:
statd -J-Djava.security.policy=jstatd.all.policy -J-Djava.rmi.server.hostname=my_server_ip/hostname
Answer based on: https://java.net/projects/visualvm/lists/users/archive/2010-03/message/8
To connect to a remote VM you have to start that remote VM with specific options:
java
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9000
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
class
After the VM is started, go to your VisualVM and do the following:
File -> Add JMX Connection
Type: yourHostName:9000 and click OK
On the left you will have the added JMX connection, double click on it and that's it!
More details on the Java Monitoring and Management Platform can be found here.
Here are the steps to do this:
Launch an ejstatd in your remote host this way (in ejstatd folder): mvn exec:java -Djava.rmi.server.hostname=<remote_host_name> -Dexec.args="-pr 1099 -ph 1100 -pv 1101" (used for "jstatd" type connection) (only specify -Djava.rmi.server.hostname if the hostname of your remote host does not match with the one you are seeing from your local network)
Launch your Java application with those additional Java parameters: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=1102 -Dcom.sun.management.jmxremote.rmi.port=1102 -Djava.rmi.server.hostname=<remote_host_name> (used for "JMX" type connection) (same remark as the previous point for -Djava.rmi.server.hostname)
Open those 4 ports on your remote host and make them available to your local machine: 1099, 1100, 1101 and 1102
Launch JVisualVM
Right-click on "Remote" > "Add Remote Host..." and enter your remote host name in "Host name" (if you don't use the port 1099, you can change this in the "Advanced Settings")
Right-click on the remote host you've just created > "Add JMX Connection..." and enter "<remote_host_name>:1102" in "Connection" input, and check "Do not require SSL connection"
Your Java process will appear twice: one from the "jstatd" connection type, and one from the "JMX" connection type.
Disclaimer: I'm the author of the open source ejstatd tool.
Start jstatd in nohup on the server which needs to be monitored and connect VisualVM to the jstatd port, following below steps:
Step 1 : Create start-jstatd.sh and copy the below content:
nohup jstatd -p 1099 -J-Djava.security.policy=<(echo 'grant codebase "file:${java.home}/../lib/tools.jar" {permission java.security.AllPermission;};') &
Step 2: Give executable permission to the file:
$ chmod a+rwx start-jstatd.sh
Step 3: Start jstatd:
$ sh start-jstatd.sh
Step 4: Add Remote Host in VisualVM:
Step 5: Add JMX Connection to the Remote Host, as shown in the below image and Click OK button: