I am trying to commit a small graph of three nodes (a_py2neo_subgraph) to my graphene Neo4j server. I am using py2neo 3.1.2.
g = py2neo.Graph(server)
tx = g.begin()
tx.create(a_py2neo_subgraph)
tx.commit()
tx.finished()
Where "server" is the exact value given by Graphene (i.e. something like http://nick:password#hobby-hash.dbs.graphenedb.com:port/db/data/). In order to debug, I ran with py2neo watch and this was the info given by watch for my httpstream:
> GET server
< 200 OK [1287]
Then I get the following traceback:
Traceback (most recent call last):
File "C:/Users/petr.svarny/PycharmProjects/untitled/test.py", line 116, in <module>
tx = g.begin()
File "C:\Users\petr.svarny\untitled\lib\site-packages\py2neo\database\__init__.py", line 370, in begin
return self.transaction_class(self, autocommit)
File "C:\Users\petr.svarny\untitled\lib\site-packages\py2neo\database\__init__.py", line 1249, in __init__
self.session = driver.session()
File "C:\Users\petr.svarny\untitled\lib\site-packages\py2neo\packages\neo4j\v1\session.py", line 126, in session
connection = connect(self.address, self.ssl_context, **self.config)
File "C:\Users\petr.svarny\untitled\lib\site-packages\py2neo\packages\neo4j\v1\bolt.py", line 419, in connect
s = create_connection(host_port)
File "C:\Python27\Lib\socket.py", line 571, in create_connection
raise err
socket.error: [Errno 10060]
I already tried to set the socket timeout to 9999, did not help. I attempted to connect via telnet to the server and managed to do so without any problem. Similarly I am able to access the server address when I enter it into my browser. I also managed to run my code for a local Neo4j database.
Thank you for any suggestions.
I'm Judit from GrapheneDB. Can you check which version of py2neo are you using? The problem you have described looks like a common issue when moving from Py2neo v2 to v3. Since py2neo v3 supports Bolt protocol, you have to specify the bolt port or just tell the driver you're not using it.
If you don't want to use Bolt connection, your code should look like the following:
graph = Graph("http://USER:PASS#hobby-hash.dbs.graphenedb.com:port/db/data/", bolt = False)
If it's not your case, it'd be useful to know which version of Python/Py2neo/Neo4j are you using.
Cheers!
I had a similar problem with py2neo 3.1.2 and I could not get it to work even with the "bolt=False" switch.
I have switched to neorestclient 2.1.1 and now that works.
Related
I am having some issues using the Neomodel and py2neo clients with Neo4j. I have installed Neomodel and py2neo in seperate anaconda virtual environments and tested each individually. Neo4j is installed/docked using docker.
Neomodel
The code
from neomodel import (config, StructuredNode, StringProperty, IntegerProperty,UniqueIdProperty, RelationshipTo, RelationshipFrom)
config.DATABASE_URL = 'bolt://neo4j:password#localhost:7687'
class Country(StructuredNode):
code = StringProperty(unique_index=True, required=True)
# traverse incoming IS_FROM relation, inflate to Person objects
inhabitant = RelationshipFrom('Person', 'IS_FROM')
class Person(StructuredNode):
uid = UniqueIdProperty()
name = StringProperty(unique_index=True)
age = IntegerProperty(index=True, default=0)
# traverse outgoing IS_FROM relations, inflate to Country objects
country = RelationshipTo(Country, 'IS_FROM')
jim = Person(name='Jim', age=3).save()
jim.age = 4
jim.save() # validation happens here
jim.delete()
jim.refresh() # reload properties from neo
jim.id # neo4j internal id
While Neomodel generates the node viewed on the neo4j webapp. The node created is Jim with age=3 i.e. It does not seem to have recorded the fact that Jims age changed from 3 -> 4. Also, I am assuming that jim.delete() would have deleted the node which it did not neither. Lastly, it prompts the following error (below is a snippet of the last lines of the error).
Error
...
File "/Users/sjamal/.conda/envs/tneo/lib/python3.6/site-
packages/neomodel/core.py", line 452, in inflate
if db_property in node.properties:
AttributeError: 'Node' object has no attribute 'properties'
Now I did find this post where the user "Jack Daniel" mentioned that neomodel does not support neo4j 3. So I tried docking the Neo4j v.2.3 image but then I receive the following error (note that its a snippet of the last few lines of the error)
Error when docking image Neo4j 2.3
File "/Users/sjamal/.conda/envs/tneo/lib/python3.6/ssl.py", line 817, in __init__
self.do_handshake()
File "/Users/sjamal/.conda/envs/tneo/lib/python3.6/ssl.py", line 1077, in do_handshake
self._sslobj.do_handshake()
File "/Users/sjamal/.conda/envs/tneo/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
OSError: [Errno 0] Error
Py2neo
I started looking into using p2neo due to the issues I had with Neomodel but I cannot seem to get my configurations right.
The code
from py2neo import Node, Relationship, Graph
graph = Graph("localhost", user='neo4j', password='password', bolt=None)
alice = Node("Person", name="Alice")
bob = Node("Person", name="Bob")
alice_knows_bob = Relationship(alice, "KNOWS", bob)
graph.create(alice_knows_bob)
Error
File "/Users/sjamal/.conda/envs/py2neo_test/lib/python3.6/site-packages/neo4j/bolt/connection.py", line 459, in acquire
connection = self.connector(address)
File "/Users/sjamal/.conda/envs/py2neo_test/lib/python3.6/site-packages/neo4j/v1/bolt.py", line 46, in <lambda>
pool = ConnectionPool(lambda a: connect(a, security_plan.ssl_context, **config))
File "/Users/sjamal/.conda/envs/py2neo_test/lib/python3.6/site-packages/neo4j/bolt/connection.py", line 601, in connect
raise ProtocolError("Connection to %r closed without handshake response" % (address,))
neo4j.bolt.connection.ProtocolError: Connection to ('localhost', 7687) closed without handshake response
Thanks to anyone looking into this. I would be happy to receive any suggestion or explanation on how to set up Py2neo irrespective if I get Neomodel to work or not.
So I have managed to solve my issue with Py2neo but not the issue I had with Neomodel. If I do find a way to get Neomodel working I will post it and either link to this post or post as a comment in this thread.
Py2neo solution with py2neo v4.0 and neo4j v3.o
I tried various combinations, starting with neo4j 2.3 together with different versions of py2neo such as 3.1.2 and then did the same with neo4j v3.0.
I am posting my script that I used to create the node and the graph connection as I was going mad when trying to figure out if I set up the configuration poorly or there was a bug in the package, driver etc.
Py2neo script
from py2neo import Node, Relationship, Graph
graph = Graph('http://localhost:7474/db/data',user='neo4j',pass word='password1234')
tx = graph.begin()
a = Node(label='hero',name='Sabri')
tx.create(a)
tx.commit()
Outdated driver py2neo v3.1.2 in tandem with Neo4j v3.4
As discussed in this Github issue report https://github.com/neo4j/neo4j-python-driver/issues/252 the user who reported the issue was using py2neo 3.1.2 together with Neo4jv3.4. The suspicion was that it was due to an outdated driver (v1.1) that came with py2neo 3.1.2. The new distribution of Neo4j v3.4 seems to come with the new driver 1.6.
Upgrading py2neo to v4.0 and sticking to latest version of Neo4j server i.e. v3.4
When doing this I ran into a different error
File "/Users/sjamal/.conda/envs/py2neo.v4/lib/python3.6/site-packages/py2neo/internal/http.py", line 26, in <module>
from neo4j.addressing import SocketAddress
ModuleNotFoundError: No module named 'neo4j.addressing'
It was discussed in this stackoverflow thread (ModuleNotFoundError: No module named 'neo4j.addressing' and ModuleNotFoundError: No module named 'neo4j') that the issue might be that the driver 1.6 driver might have to be manually installed through pip, which I did.
pip install neo4j-driver==1.6.2
I now received a new error where TypeError was caught when calling a map object.
File "/Users/sjamal/.conda/envs/py2neo.v4/lib/python3.6/site-packages/py2neo/internal/http.py", line 74, in fix_parameters
raise TypeError("Parameters of type {} are not supported".format(type(value).__name__))
TypeError: Parameters of type map are not supported
I found this github issue posted by speters-cmri https://github.com/technige/py2neo/issues/688 which contained the following github commit (https://github.com/technige/py2neo/compare/v4...CMRI-ProCan:v4) to resolve the issue by modifying a json.py script in the py2neo package
I ran my script again to add a test node and it ran without any issues.
If you are too lazy or simply too frustrated to go through a long explanation here is a summary
1. Make sure neo4j v3.0+ is installed. I suggest you look into docker to install neo4j using a docker image
2. pip install py2neo==v4.0
3. pip install neo4j-driver==1.6.2
4. Modify json.py file as described here https://github.com/technige/py2neo/compare/v4...CMRI-ProCan:v4
5. Run py2neo script outlined above
Recently a script which uses py2neo package and connects via the Bolt protocol began failing unexpectedly with a ProtocolError: Server closed connection, and I'm struggling to understand why. Note the script works when using HTTP as the protocol.
The script pulls the Neo4j graph, augments it in Python and then attempts to push the relevant changes back to the database. The whole process takes about an hour, and was working reliably for quite some time, however the push stage recently starting failing.
If I create a toy example, bypasses the pull and augmentation, then the push works and thus I'm thinking the server closed the connection due to a timeout, however I can't find any timeout related parameter for Bolt in py2neo. Note I have set the HTTP socket timeout to be 9999 seconds (~ 2.75 hours),
from py2neo.packages.httpstream import http
http.socket_timeout = 9999
though my understanding of this is i) it's unrelated to Bolt and ii) the timeout is at the connection level which greatly exceeds the time the script ran.
For reference I'm using Neo4j v3.0.3 and py2neo v3.1.2. The stack-trace was:
File "/usr/local/lib/python2.7/dist-packages/py2neo/database/__init__.py", line 1017, in __exit__
self.commit()
File "/usr/local/lib/python2.7/dist-packages/py2neo/database/__init__.py", line 1059, in commit
self._post(commit=True)
File "/usr/local/lib/python2.7/dist-packages/py2neo/database/__init__.py", line 1291, in _post
self.finish()
File "/usr/local/lib/python2.7/dist-packages/py2neo/database/__init__.py", line 1296, in finish
self._sync()
File "/usr/local/lib/python2.7/dist-packages/py2neo/database/__init__.py", line 1286, in _sync
connection.fetch()
File "/usr/local/lib/python2.7/dist-packages/py2neo/packages/neo4j/v1/bolt.py", line 323, in fetch
raw.writelines(self.channel.chunk_reader())
File "/usr/local/lib/python2.7/dist-packages/py2neo/packages/neo4j/v1/bolt.py", line 174, in chunk_reader
chunk_header = self._recv(2)
File "/usr/local/lib/python2.7/dist-packages/py2neo/packages/neo4j/v1/bolt.py", line 157, in _recv
raise ProtocolError("Server closed connection")
ProtocolError: Server closed connection
and the stripped down Python code is of the form,
import py2neo
from py2neo.packages.httpstream import http
http.socket_timeout = 3600
graph = py2neo.Graph(
host='localhost',
bolt=True,
bolt_port=4096,
http_port=4095,
)
# Pull the graph from the Neo4j database via graph.run(...) statements,
# augments the graph etc.
# ...
# Exception is thrown when the following push transaction is executed.
with graph.begin() as tx:
statement = """
UNWIND {rows} AS row
WITH row.source AS source, row.target AS target
MATCH (s:Entity)
USING INDEX s:Entity(uuid)
WHERE s.uuid = source
MATCH (t:Entity)
USING INDEX t:Entity(uuid)
WHERE t.uuid = target
MATCH (s)-[r:FAVORITED]->(t)
DELETE r
"""
rows = [{
'source': '8267d7d0-a837-11e6-b841-22000bcec6a9',
'target': 'c6296c97-a837-11e6-b841-22000bcec6a9',
}]
tx.run(statement, rows=rows)
Does anyone have any suggestions as to how I can further debug this or what causes the connection to close? I've looked through the _recv function but it's not apparent why no data was received by the socket.
Looking through the Neo4j debug.log file the only possible related error was
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:242)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
I've also checked the uptime of the service, and confirmed that it was running throughout the entire duration of the script (HH:MM:SS),
> ps -p "14765" -o etime=
16:55:31
I believe the issue is with:
tx.run(statement, rows=rows)
The second argument to CypherResoure.run() is a dictionary of parameters. You're passing them in as additional python params. See py2neo code.
Try:
tx.run(statement, {"rows": rows});
We have a aiohttp based web services which uses ZMQ to send jobs to workers and waits for the result. We are of course using the ZMQ eventloop, so we can wait for ZMQ sockets. "Sometimes" the process crashes and we get this stack trace:
...
await socket.send(z, flags=flags)
File "/usr/local/lib/python3.5/dist-packages/zmq/eventloop/future.py", line 165, in send
kwargs=dict(flags=flags, copy=copy, track=track),
File "/usr/local/lib/python3.5/dist-packages/zmq/eventloop/future.py", line 276, in _add_send_event
timeout_ms = self._shadow_sock.sndtimeo
File "/usr/local/lib/python3.5/dist-packages/zmq/sugar/attrsettr.py", line 45, in _getattr_
return self._get_attr_opt(upper_key, opt)
File "/usr/local/lib/python3.5/dist-packages/zmq/sugar/attrsettr.py", line 49, in _get_attr_opt
return self.get(opt)
File "zmq/backend/cython/socket.pyx", line 449, in zmq.backend.cython.socket.Socket.get (zmq/backend/cython/socket.c:4920)
File "zmq/backend/cython/socket.pyx", line 221, in zmq.backend.cython.socket._getsockopt (zmq/backend/cython/socket.c:2860)
"Sometimes" means, that the code works fine, if I just run it on my test machine. We encountered the problem in some rare cases when using docker containers, but were never able to reproduce it in an reliable way. Since we moved our containers into a Kubernetes cluster, it occurs much more often. Does anybody know, what could be the source of the above stack trace?
aiohttp is not intended to be used with vanilla pyzmq.
Use aiozmq loopless streams instead.
See also https://github.com/zeromq/pyzmq/issues/894 and https://github.com/aio-libs/aiozmq/blob/master/README.rst
First let me briefly describe our set up before I ask the question proper:
We have a web application server (virtual machine) running a django application. nginx at the front, uwsgi running under that, then a newrelic application wrapper followed by django et al., database is a separate postgresql server located via smartstack (synapse/nerve)
The issue we face is that occasionally (happened once 2 weeks ago, and twice in the last 2 days), one or two of the uwsgi worker processes will trip up and start producing "django.db.utils.InterfaceError: connection already closed" on most of their requests.
slightly redacted stack trace (user and application_name):
Traceback (most recent call last):
File "/home/user/webapps/application_name/local/lib/python2.7/site-packages/newrelic-2.8.0.7/newrelic/api/web_transaction.py", line 863, in __call__
File "/home/user/webapps/application_name/local/lib/python2.7/site-packages/newrelic-2.8.0.7/newrelic/api/function_trace.py", line 90, in literal_wrapper
File "/home/user/webapps/application_name/local/lib/python2.7/site-packages/newrelic-2.8.0.7/newrelic/api/web_transaction.py", line 752, in __call__
File "/home/user/webapps/application_name/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 194, in __call__
signals.request_started.send(sender=self.__class__)
File "/home/user/webapps/application_name/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 185, in send
response = receiver(signal=self, sender=sender, **named)
File "/home/user/webapps/application_name/local/lib/python2.7/site-packages/django/db/__init__.py", line 91, in close_old_connections
conn.abort()
File "/home/user/webapps/application_name/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 374, in abort
self.rollback()
File "/home/user/webapps/application_name/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 177, in rollback
self._rollback()
File "/home/user/webapps/application_name/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 141, in _rollback
return self.connection.rollback()
File "/home/user/webapps/application_name/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/user/webapps/application_name/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 141, in _rollback
return self.connection.rollback()
File "/home/user/webapps/application_name/local/lib/python2.7/site-packages/newrelic-2.8.0.7/newrelic/hooks/database_dbapi2.py", line 82, in rollback
django.db.utils.InterfaceError: connection already closed
The stack trace never gets in to our application, it only touches new relic and django. Once a worker trips, it doesn't recover and all further requests result in 500's in the uwsgi logs and 502's on the front side. I assume database connectivity is fine because the sibling workers continue to function normally, and restarting uwsgi instantly fixes the problem.
My question is how one would go about diagnosing this issue to pinpoint the root cause, I have checked everything I know how to check (memory, cpu, logs, database connectivity) and some things I don't fully understand but am trying to read up on (file descriptors mainly).
For now I updated new relic (stack trace is older version) as it's the only thing I felt I could do.
I would appreciate any feedback, many google searches have proved fruitless.
replies may be slightly delayed, my timezone says it's time to sleep. Also, apologies if this should be on serverfault or something, I just figured it's closer to an application debug issue than a server config issue.
I'm trying to use the Java API for Neo4j but I seem to be stuck at IndexHits. If I query the DB with Cypher using
START n=node:types(type="Process") RETURN n;
I get all 2087 nodes of type "Process".
In my application I have the following lines
Index<Node> nodeIndex = db.index().forNodes("types");
IndexHits<Node> hits = nodeIndex.get("type", "Process");
System.out.println("Node index size: " + hits.size());
which leads my console to spit out a value of 0. Here, db is of course an instance of GraphDatabaseService.
I expected an object that included all 2087 nodes. What am I doing wrong?
The .size() question is just the prelude to my iterator
for(Node process : hits) { ... }
but that does not much when hits.size() == 0. According to http://api.neo4j.org/1.9.2/org/neo4j/graphdb/index/IndexHits.html this should be possible, provided there is something in hits.
Thanks in advance for your help.
I figured it out. Man, I feel so embarrassed...
It so happens that I had set up the DB_PATH to my default data folder, whereas the default storage folder is the default data folder plus graph.db. When I tried to run the code from that corrected DB_PATH I got an error saying that a lock file was in place because the Neo4j server was running. After shutting it down it worked perfectly.
So, if you happen to see the following error, just stop the server and run the code again:
Caused by: org.neo4j.kernel.StoreLockException: Could not create lock file
at org.neo4j.kernel.StoreLocker.checkLock(StoreLocker.java:74)
at org.neo4j.kernel.StoreLockerLifecycleAdapter.start(StoreLockerLifecycleAdapter.java:40)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:491)
I found on several forums that you cannot run the Neo4j server and use the Java API to query it at the same time.