I modified the demo EmbeddedNeo4jWithIndexing.java(create node with index).
I Comment all shutdown() method,
after do it, I open the database by neo4j-shell -readonly,then I run
start n=node:nodes("*:*") return n;
I get 0 rows
if I open the database without -readonly ,and run the command above,it successed!
I am very confused about it
I tried 1.8.2 and 1.9
We disallow read-only access to running databases because of lucene ignoring the R/O aspect and still doing merges asynchronically behind the scenes, clashing with the merge threads of the live database.
Related
I am migrating a MariaDB database into a Linux docker container.
I am using mariadb:latest in Ubuntu 20 LTS via Windows 10 WSL2 via VSCode Remote WSL.
I have copied the sql dump into the container and imported it into the InnoDB database which has DEFAULT CHARACTER SET utf8. It does not report any errors:
> source /test.sql
That file does this (actual data truncated for this post):
USE `mydb`;
DROP TABLE IF EXISTS `opsitemtest`;
CREATE TABLE `opsitemtest` (
`opId` int(11) NOT NULL AUTO_INCREMENT,
`opKey` varchar(50) DEFAULT NULL,
`opName` varchar(200) DEFAULT NULL,
`opDetails` longtext,
PRIMARY KEY (`opId`),
KEY `token` (`opKey`)
) ENGINE=InnoDB AUTO_INCREMENT=4784 DEFAULT CHARSET=latin1;
insert into `opsitemtest`(`opId`,`opKey`,`opName`,`opDetails`) values
(4773,'8vlte0755dj','VTools addin for MSAccess','<p>There is a super helpful ...'),
(4774,'8vttlcr2fTA','BAS OLD QB','<ol>\n<li><a href=\"https://www.anz.com/inetbank/bankmain.asp\" ...'),
(4783,'9c7id5rmxGK','STP - Single Touch Payrol','<h1>Gather data</h1>\n<ol style=\"list-style-type: decimal;\"> ...');
If I source a subset of 12 records of the table in question all the columns are correctly populated.
If I source the full set of data for the same table ( 4700 rows ) where everything else is the same, many of the opDetails long text fields have a length showing in sqlYog but no data is visible. If I run a SELECT on that column there are no errors but some of the opDetails fields are "empty" (meaning: you can't see any data), and when I serialize that field, the opDetails column of some records (not all) has
"opDetails" : "\u0000\u0000\u0000\u0000\u0000\u0000\",
( and many more \u0000 ).
The opDetails field contains HTML fragments. I am guessing it is something to do with that content and possibly the CHARSET, although that doesn't explain why the error shows up only when there are a large number of rows imported. The same row imported via a set of 12 rows works correctly.
The same test of the full set of data on a Windows box with MariaDB running on that host (ie no Ubuntu or WSL etc) all works perfectly.
I tried setting the table charset to utf8 to match the database default but that had no effect. I assume it is some kind of Windows WSL issue but I am running the source command on the container all within the Ubuntu host.
The MariaDB data folder is mapped using a volume, again all inside the Ubuntu container:
volumes:
- ../flowt-docker-volumes/mariadb-data:/var/lib/mysql
Can anyone offer any suggestions while I go through and try manually removing content until it works? I am really in the dark here.
EDIT: I just ran the same import process on a Mac to a MariaDB container on the OSX host to check whether it was actually related to Windows WSL etc and the OSX database has the same issue. So maybe it is a MariaDB docker issue?
EDIT 2: It looks like it has nothing to do with the actual content of opDetails. For a given row that is showing the symptoms, whether or not the data gets imported correctly seems to depend on how many rows I am importing! For a small number of rows, all is well. For a large number there is missing data, but always the same rows and opDetails field. I will try importing in small chunks but overall the table isn't THAT big!
EDIT 3: I tried a docker-compose without a volume and imported the data directly into the MariaDB container. Same problem. I was wondering whether it was a file system incompatibility or some kind of speed issue. Yes, grasping at straws!
Thanks,
Murray
OK. I got it working. :-)
One piece of info I neglected to mention, and it might not be relevant anyway, is that I was importing from an sql dump from 10.1.48-MariaDB-0ubuntu0.18.04.1 because I was migrating a legacy app.
So, with my docker-compose:
Version
Result
mysql:latest
data imported correctly
mariadb:latest
failed as per this issue
mariadb:mariadb:10.7.4
failed as per this issue
mariadb:mariadb:10.7
failed as per this issue
mariadb:10.6
data imported correctly
mariadb:10.5
data imported correctly
mariadb:10.2
data imported correctly
Important: remember to completely remove the external volume mount folder content between tests!
So, now I am not sure whether the issue was some kind of sql incompatibility that I need to be aware of, or whether it is a bug that was introduced between v10.6 and 10.7. Therefore I have not logged a bug report. If others with more expertise think this is a bug, I am happy to make a report.
For now I am happy to use 10.6 so I can progress the migration- the deadline is looming!
So, this is sort of "solved".
Thanks for all your help. If I discover anything further I will post back here.
Murray
I run neo4j Version: 3.3.2 in a docker container and tried to configure a default style as described above.
Grass file is accessible via HTTP (http://somedomain.com/resources/style.grass)
CORS rules are set
Command added to config: browser.post_connect_cmd=style http://somedomain.com/resources/style.grass
When i start my container with this config and run match (n) return n i see nothing.
Then i run :style http://somedomain.com/resources/style.grass and it works... Why?
That leads me to conclude that the grass file ifself (content) and the accessible of this file are not the problem.
It seems that there is an error with the default style configuration.
Do you have any tips to debug this? The logs are clean.
I found the problem. We added this property at the end of the neo4j.conf. After startup neo4j reorder the properties, put it at the 2nd place and accidentally add the 3rd property at the end of the 2nd, so that it’s not valid. We put it manually at the 2nd place and now it works.
In Dataflow 1.x versions, we could use CloudBigtableIO.writeToTable(TABLE_ID) to create, update, and delete Bigtable rows. As long as a DoFn was configured to output a Mutation object, it could output either a Put or a Delete, and CloudBigtableIO.writeToTable() successfully created, updated, or deleted a row for the given RowID.
It seems that the new Beam 2.2.0 API uses BigtableIO.write() function, which works with KV<RowID, Iterable<Mutation>>, where the Iterable contains a set of row-level operations. I have found how to use that to work on Cell-level data, so it's OK to create new rows and create/delete columns, but how do we delete rows now, given an existing RowID?
Any help appreciated!
** Some further clarification:
From this document: https://cloud.google.com/bigtable/docs/dataflow-hbase I understand that changing the dependency ArtifactID from bigtable-hbase-dataflow to bigtable-hbase-beam should be compatible with Beam version 2.2.0 and the article suggests doing Bigtble writes (and hence Deletes) in the old way by using CloudBigtableIO.writeToTable(). However that requires imports from the com.google.cloud.bigtable.dataflow family of dependencies, which the Release Notes suggest is deprecated and shouldn't be used (and indeed it seems incompatible with the new Configuration classes/etc.)
** Further Update:
It looks like my pom.xml didn't refresh properly after the change from bigtable-hbase-dataflow to bigtable-hbase-beam ArtifactID. Once the project got updated, I am able to import from the
com.google.cloud.bigtable.beam.* branch, which seems to be working at least for the minimal test.
HOWEVER: It looks like now there are two different Mutation classes:
com.google.bigtable.v2.Mutation and
org.apache.hadoop.hbase.client.Mutation ?
And in order to get everything to work together, it has to be specified properly which Mutation is used for which operation?
Is there a better way to do this?
Unfortunately, Apache Beam 2.2.0 doesn't provide a native interface for deleting an entire row (including the row key) in Bigtable. The only full solution would be to continue using the CloudBigtableIO class as you already mentioned.
A different solution would be to just delete all the cells from the row. This way, you can fully move forward with using the BigtableIO class. However, this solution does NOT delete the row key itself, so the cost of storing the row key remains. If your application requires deleting many rows, this solution may not be ideal.
import com.google.bigtable.v2.Mutation
import com.google.bigtable.v2.Mutation.DeleteFromRow
// mutation to delete all cells from a row
Mutation.newBuilder().setDeleteFromRow(DeleteFromRow.getDefaultInstance()).build()
I would suggest that you should continue using CloudBigtableIO and bigtable-hbase-beam. It shouldn't be too different from CloudBigtableIO in bigtable-hbase-dataflow.
CloudBigtableIO uses the HBase org.apache.hadoop.hbase.client.Mutation and translates them into the Bigtable equivalent values under the covers
I have archive(tar.gz) which was dumped from production neo4j server. Now I want to get this db locally in my computer. I did it in several steps:
I was maked this command: neo4j-admin load --from=<archive-path> --database=<database> [--force](I did it correct, set path to my archive and gave name for database)
2.When I maked previous command then folder in data/databases appeared, that's cool, I thought
Next I was change active db I changed this value:
dbms.active_database=graph.db.test
And when all must work I typed "./neo4j console" it started in localhost:7474 I see that db is my,for example "test" db but all was empty. No node labels, no relations and when I used simple command "MATCH(n) RETURN n" there are no records. But I totally sure that it's must to be not empty.
Question:
What I did incorrect and what I need to do to make it works?
I believe that you are having problems with the param database and the value of dbms.active_database property.
Try using the same value for both:
bin/neo4j-admin load --from=/backups/graph.db/2016-10-02.dump --database=graph.test.db --force
and
dbms.active_database=graph.test.db
after it, restart Neo4j.
You may try refreshing and re-entering your credentials. Also, it may be the case that your file might not be unzipped properly.
Exception in thread "main" java.lang.IllegalStateException: Database locked.
at org.neo4j.kernel.InternalAbstractGraphDatabase.create(InternalAbstractGraphDatabase.java:289)
at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:227)
at org.neo4j.kernel.EmbeddedReadOnlyGraphDatabase.<init>(EmbeddedReadOnlyGraphDatabase.java:81)
at org.neo4j.kernel.EmbeddedReadOnlyGraphDatabase.<init>(EmbeddedReadOnlyGraphDatabase.java:72)
at org.neo4j.kernel.EmbeddedReadOnlyGraphDatabase.<init>(EmbeddedReadOnlyGraphDatabase.java:54)
at QueryNodeReadOnly.main(QueryNodeReadOnly.java:55)
This is using 1.8.2 version of neo4j. I've written a program that opens the db in readonly mode, querying and and make it sleep for a while before exiting.
Here is the relevant text
graphDb = new EmbeddedReadOnlyGraphDatabase( dbname); // Line 55 - the exception.
......
......
......
......
......
if(sleepVal > 0)
Thread.sleep(sleepVal);
I reckon I should not be getting this error. There are only 2 processes that open the db , both in read-only mode. In fact, it should work even if i open the db when another process has opened it to write to it.
We disallow two databases accessing the same files on disk at the same time - even in read-only mode.
The reason being is that while we do not allow you to modify the database in read-only mode, Lucene will still write to disk when servicing your read requests, and having two instances access those same index files leads to race conditions and index corruptions.
Why is it you want 2x instances accessing the same files at the same time anyway? What is your use case?
You can't make multiple connections to an embedded database. Maybe you should consider using the REST server.