I have a really strange problem connecting to a Neo4j instance running in a docker instance. But first things first, here is my set up:
Everything runs on a MacOS. I use boot2docker, so the actual Docker instances run in a VirtualBox. I am able to reproduce this issue with Neo4j 2.1.5 and with Neo4j 2.2.0-M04.
Additionally, I installed Neo4j locally via homebrew and it is version 2.1.7. It is installed to have a neo4j-shell available.
The docker instances are accessible from the outside via 192.168.59.104. The docker instance itself has a 172.17.0.x IP address.
The problem:
I try to connect with neo4j-shell like this: neo4j-shell -v -host 192.168.59.104 -port 1337 -name shell. After a couple of minutes, it fails with:
ERROR (-v for expanded information):
Connection refused
java.rmi.ConnectException: Connection refused to host: 172.17.0.2; nested exception is:
java.net.ConnectException: Operation timed out
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:194)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:148)
at com.sun.proxy.$Proxy1.welcome(Unknown Source)
at org.neo4j.shell.impl.AbstractClient.sayHi(AbstractClient.java:215)
at org.neo4j.shell.impl.RemoteClient.findRemoteServer(RemoteClient.java:63)
at org.neo4j.shell.impl.RemoteClient.<init>(RemoteClient.java:55)
at org.neo4j.shell.impl.RemoteClient.<init>(RemoteClient.java:43)
at org.neo4j.shell.ShellLobby.newClient(ShellLobby.java:165)
at org.neo4j.shell.StartClient.startRemote(StartClient.java:297)
at org.neo4j.shell.StartClient.start(StartClient.java:175)
at org.neo4j.shell.StartClient.main(StartClient.java:120)
Caused by: java.net.ConnectException: Operation timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 14 more
The interesting part is, that it fails by trying to connect 172.17.0.2 instead of the provided IP address.
What I did so far:
Double check neo4j.properties are correct
remote_shell_host=0.0.0.0 is set
remote_shell_port=1337 is set
check the ports are exposed properly with docker ps:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
099cfd1b8018 daneroo/neo4j:latest "/bin/bash -c /launc 3 minutes ago Up 3 minutes 0.0.0.0:1337->1337/tcp, 0.0.0.0:7474->7474/tcp neo4j
Some Google and SO searches, but without any success.
Different neo4j versions and docker images
Basic connection checks with telnet 192.168.59.104 1337. This is successful.
Catch up with Mark Needham at todays neo4j meetup
So, do I miss something? In theory this should happen every time you can access the Neo4j through another IP address than the Neo4j instance can see (e.g. if you run a Neo4j instance at home on a machine with the IP 192.168.x.y and you try to access it via your public IP you have at this time).
Where is the client, running outside the Neo4j environment, getting the internal IP address from? Is there a way to work around this issue?
I think it might be related to the RMI protocol that the shell uses to communicate with Neo4j.
Afaik RMI also uses some additional ports and it might redirect the initial request. there is a system property in Java that you might want to try to set in neo4j-wrapper.conf: -Djava.rmi.server.hostname=192.168.59.104
see:
http://courses.cs.washington.edu/courses/cse341/98au/java/jdk1.2beta4/docs/guide/rmi/rmiNetworkingFAQ.html
http://docs.oracle.com/javase/7/docs/technotes/guides/rmi/javarmiproperties.html
Related
I have the production cluster of Wazuh 4 with open-distro for elasticsearch, kibana and ssl security in docker and I am trying to connect logstash (a docker image of logstash) with elasticsearch and I am getting this:
Attempted to resurrect connection to dead ES instance, but got an error
I have generated ssl certificates for logstash, tried other ways (changed the output of logstash , through filebeat modules) to connect without success.
What is the solution for this problem for Wazuh 4?
Let me help you with this. Our current documentation is valid for distributed architectures where Logstash is installed on the same machine as Elasticsearch, so we should consider adding documentation for the proper configuration of separated Logstash instances.
Ok, now let’s see if we can fix your problem.
After installing Logstash, I assume that you configured it using the distributed configuration file, as seen on this step (Logstash.2.b). Keep in mind that you need to specify the Elasticsearch IP address at the bottom of the file:
output {
elasticsearch {
hosts => ["<PUT_HERE_ELASTICSEARCH_IP>:9200"]
index => "wazuh-alerts-3.x-%{+YYYY.MM.dd}"
document_type => "wazuh"
}
}
After saving the file and restarting the Logstash service, you may be getting this kind of log message on /var/log/logstash/logstash-plain.log:
Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://192.168.56.104:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://192.168.56.104:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
I discovered that we need to edit the Elasticsearch configuration file, and modify this setting: network.host. On my test environment, this setting appears commented like this:
#network.host: 192.168.0.1
And I changed it to this:
network.host: 0.0.0.0
(Notice that I removed the # at the beginning of the line). The 0.0.0.0 IP will make Elasticsearch listen on all network interfaces.
After that, I restarted the Elasticsearch service using systemctl restart elasticsearch, and then, I started to see the alerts being indexed on Elasticsearch. Please, try these steps, and let’s see if everything is properly working now.
Let me know if you need more help with this, I’ll be glad to assist you.
Regards,
I'm currently running Neo4j on Google Cloud with in a Compute Engine VM running Ubuntu. The 7474 port works as expected, however I'm receiving the following message when trying to connect to server:
WebSocket connection to 'ws://<ip>:7687/' failed: Error in connection establishment: net::ERR_CONNECTION_TIMED_OUT
I checked the conf/neo4j.conf for dbms.connector.bolt.address=0.0.0.0:7687 and it's not commented out.
I checked the firewall, and there is a rule for port 7687, so what else could cause this?
Thanks in advance for the help
Update:
I was able to use the cypher-shell from the VM's command line, which connects to bolt://localhost:7687
It turns out the issue was with neither GCP nor neo4j. The company where I work for has a firewall blocking the port, and that's why I wasn't able to connect to the database using the browser. Dataflow in Compute Engine had no problem connecting to neo4j.
I've just installed docker on my windows 7 machine. When I start Docker QuickStart, I get following error which seems to be while creating the machine:
Creating machine...
(default) Unable to get the latest Boot2Docker ISO release version: Get https:/
/api.github.com/repos/boot2docker/boot2docker/releases/latest: dial tcp 192.30.2
52.124:443: connectex: A connection attempt failed because the connected party d
id not properly respond after a period of time, or established connection failed
because connected host has failed to respond.
(default) Copying C:\Users\robot\.docker\machine\cache\boot2docker.iso to C:\Use
rs\robot\.docker\machine\machines\default\boot2docker.iso...
(default) Creating VirtualBox VM...
(default) Creating SSH key...
Error attempting heartbeat call to plugin server: read tcp 127.0.0.1:60733->127.
0.0.1:60732: wsarecv: An existing connection was forcibly closed by the remote h
ost.
Error attempting heartbeat call to plugin server: connection is shut down
Error attempting heartbeat call to plugin server: connection is shut down
Error attempting heartbeat call to plugin server: connection is shut down
Error attempting heartbeat call to plugin server: connection is shut down
Error creating machine: Error in driver during machine creation: read tcp 127.0.
0.1:60733->127.0.0.1:60732: wsarecv: An existing connection was forcibly closed
by the remote host.
Looks like something went wrong... Press any key to continue...
There is a similar issue in docker/machine/issues/2773.
Try and see if the issue persists when creating a machine yourself instead of using quick-start:
Find where docker-machine.exe has been installed (or copy the latest released one in your %PATH%) and use that, from a regular CMD session:
First test the existing machine:
# find the name of the machine created.
docker-machine ls
docker-machine env --shell cmd <nameOfTheMachine>
docker machine ssh <nameOfTheMachine>
Then try creating a new one:
docker-machine create -d virtualbox <aNewMachine>
docker-machine env --shell cmd <aNewMachine>
docker machine ssh <aNewMachine>
I do not have a solution but found the root cause.
I had installed boot2docker and has been using for months. I had been creating all
my vbox images on the same folder all the while.
One fine day I decided to archive my machines and changed the folder in which I was creating the vbox images. It started giving this wired error. I reverted back my archive and tested again. It started working fine.
The difference I found on both the set up was, in the archived folder it was skipping the ca cert creation step and was directly creating the machine. In the new folder it was creating a cert and then creating the machine. It looks like the server doesn't like the new certs!!!!
I've installed neo4j 2.0.0 via homebrew and started the server. I'm able to access the webadmin console at port 7474 but when I run neo4j-shell -host localhost -port 7474 I get:
non-JRMP server at remote endpoint
java.rmi.ConnectIOException: non-JRMP server at remote endpoint
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:248)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:341)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at java.rmi.Naming.lookup(Naming.java:101)
at org.neo4j.shell.impl.RmiLocation.getBoundObject(RmiLocation.java:253)
at org.neo4j.shell.impl.RemoteClient.findRemoteServer(RemoteClient.java:62)
at org.neo4j.shell.impl.RemoteClient.<init>(RemoteClient.java:55)
at org.neo4j.shell.impl.RemoteClient.<init>(RemoteClient.java:43)
at org.neo4j.shell.ShellLobby.newClient(ShellLobby.java:165)
at org.neo4j.shell.StartClient.startRemote(StartClient.java:295)
at org.neo4j.shell.StartClient.start(StartClient.java:173)
at org.neo4j.shell.StartClient.main(StartClient.java:125)
Help?
Two years later and #jjaderberg's answer is still relevant. I'm on Neo4J v3.1.1 right now and when installing via Homebrew, the neo4j.conf file makes no reference to the default port being 1337. In researching this further, Neo4js is deprecating use of the neo4j-shell in favor of cypher-shell, and by extension remove's the need for the 1337 port.
According to https://neo4j.com/docs/operations-manual/current/configuration/ports/
"The neo4j-shell tool is being deprecated and it is recommended to discontinue its use."
non-JRMP server at remote endpoint
means that it turns out turning on SSL was required.
The default port for neo4j-shell is 1337, but if you are running the shell on the host computer you don't need to use a remote connection at all, try running it either without arguments at all or with -path path/to/your/db.
I'm trying to setup backup for a Neo4j cluster with 3 instances. Neo4j is embedded.
If I run:
./neo4j-backup -from ha://10.106.4.80:5001,10.106.4.203:5001,10.106.14.164:5001 -to /tmp/neobak2/
from a host outside the 10.106.4.0 network, I get this error:
Could not find backup server in cluster neo4j.ha at 10.106.4.80:5001,10.106.4.203:5001,10.106.14.164:5001, operation timed out.
If I run it from a cluster member it works just fine. Also if I run the backup script with single instead of ha works fine from anywhere.
Below the basic cluster config I'm using:
ha.server_id: 1
ha.initial_hosts:10.106.4.80:5001,10.106.4.203:5001,10.106.14.164:5001
ha.tx_push_factor: 2
I already checked for firewall issues, there aren't any. Neo4j version used is 1.9.5.
The webadmin interface shows the cluster has online backup enabled and listening to the default port.
Any help will be appreciated.
According to RFC 5735 IP Adresses 10.0.0.0/8 are private. So I assume they're not routed from an external host.