When making a backup in Advantage, we want to only backup some tables that hold some structural data. These tables may change in a database update, but most of the tables will not. The other tables contain enormous amounts of blob data: potentially 10s of gigs.
If something does go awry in the database update, and we want to restore the backed up tables, what are our options to keep the original large tables, but restore the selected tables? Restore them to the existing database, keeping the old large tables in place? Free the large tables and add them to the newly restored database?
(Note: no RI rules between the backup up tables and the large tables, but the large tables do reference autoincs in the backed-up tables.)
You can use inclusion lists to limit the tables that are backed up or restored. For example, if you are using the adsbackup.exe command line utility, you could restrict the backup to two tables:
adsbackup -i"table1,table2" test.add c:\mypath\backup\
Then with a restore, you can also restrict it to specific files:
adsbackup -r -i"table1,table2" c:\mypath\backup\test.add c:\mypath\restore\test.add
Note that when specifying a restore operation with an inclusion list, the dictionary (.add) file will only be restored if it does not already exist. So if you restore to an existing dictionary with an inclusion list, it will only copy the specified tables.
Related
As the title of the question clearly describes, the size of my database is 28.16 MB despite I have removed all the data stored. I have added the detail of the database below:
We have a knowledge base article on understanding database growth which covers this in detail.
The summary here is that when data is deleted, we mark the records in the store as deleted, so they continue to exist and take up space, but after meeting some conditions they will be eligible for reuse when new data is added later.
In addition we keep track of the ids of deleted and eligible-for-reuse spaces in the store files which will grow during deletes, but shrink during later ingestion (as the ids are reused and removed from the .id files), so you may see a counterintuitive effect: growing of the db size during deletions (since we're adding the ids to the .id files) and shrinking during additions when filling in those reusable places in the store files (as the ids are used up and removed from the .id files).
Lastly, we also have transaction logs, which will be holding transactional history for the database and can often be quite large, and the rotation and retention properties in your neo4j.conf will determine how often these get flushed to the store files, and how/when they are pruned. Never move or delete these files yourself, change the rotation and/or retention properties in the conf file to manage these.
If wanting to start from scratch, the cleanest way is to shut down Neo4j and delete the graph.db folder, and it will be recreated on the next startup.
I'm going to use Xodus for storing time-series data (100-500 million rows are inserted daily.)
I saw that Xodus was creating and deleting a lot of .xd files in the background. I read about log-structured design, but I don't clearly understand whether file is created on each transaction commit. Is each file represents snapshot of whole database? Is there any way to disable transactions (i don't need it) ?
Can I get any performance benefits by sharding my data between different stores ? I can store every metric in separate store instead of using one store with multikey. For now I'm creating separate store for each day
The .xd files don't actually represent certain transactions. The files are ordered, so they can be thought as an infinite log of records. Each transaction writes the changes and some meta information for making it possible to retrieve/search for saved data. Any .xd file has its maximum size, and when it is reached the new file is created.
It is not possible to disable transactions.
Basically, sharding your data between different stores gives better performance, at least the smaller the stores are, the faster and smoother GC works in background. The way you shard your data defines the way you can retrieve it. If data in different shards is completely decoupled than it is even better to store shards in different environments, not stores of a single environment. This will also physically isolate data in different shards, not only logically.
I'm having trouble with a core data many-to-many relationship where both side's delete rules are set to Nullify. I'm finding when I inspect the SQL database there are records left in the join table that should be deleted.
The two tables represent Playlists and Tracks. Playlist deletes don't cascade to delete each track automatically because each track can be in multiple playlists.
The tracks in a playlists are in an ordered relationship.
The track is being removed from the playlist and separately the playlist and track are being deleted but there remains a record in the Z_6PLAYLIST_TRACKS table in the database.
How do I make sure these records are deleted too?
Most likely the full delete has not propagated into the sqlite file and is still sitting in one of the two cache files. This is common now that sqlite is being used with WAL mode by Core Data.
As an aside, I would not worry about the tables inside of Core Data like that. Access Core Data through the public API only. The internal workings of the SQLite file is not something you should be trying to keep in sync.
If you are getting the data you expect to be getting and your deletes are giving you the result you expected then that is sufficient. Only if you are having issues with your deletes should you be working about the internal structure of the SQLite file.
Update 1
Thanks for the test case, that is an actual issue in Core Data. There is nothing in there that you are doing incorrectly. You REALLY need to file a radar on this and I will file one as well.
In the interim you can either stop using ordered relationships (they are a bastardization anyway) or do the double delete.
Personally I would change it from ordered relationships as they have terrible performance and clearly have issues.
The idea is to have git or a git-like system (users, revision tracking, branches, forks, etc) store the 'master copy' of objects and relationships.
Since the master copy is on the filesystem, any changes can be checked in, tracked, and backed up. Neo4j could then import the files and serve queries. This also gives freedom since node and connection files can be imported to any other database.
Changes in Neo4j can be written to these files as part of the query
Nodes and connections can be added by other means (like copying from a seed dataset)
Nodes and connections are rarely created/updated/deleted by users
Most of the usage is where Neo4j shines: querying
Due to these two, the performance penalty on importing can be safely ignored
What's the best way to set this up?
If this isn't wise; how come?
It's possible to do that, but it will be lot of work which would not have a real value. IMHO.
With unmanaged extension for Transaction Event API you are able to store information about each transaction onto disk in your common file format.
Here is the some information about Transaction Event API - http://graphaware.com/neo4j/transactions/2014/07/11/neo4j-transaction-event-api.html
Could you please tell us more about the use case and how would design that system?
In general nothing keeps you from just keeping neo4j database files around (zipped).
Otherwise I would probably use a format which can be quickly exported / imported and diffed too.
So very probably csv files with node-file per label ordered by a sensible key
And then relationship-files between pairs of nodes, with neo4j-import you can recover that data quickly into a graph again.
If you want to write changes to the files you have to make sure they are replayable (appends + updates + deletes) , i.e. you have to chose a format which is more or less a transaction-log (which Neo4j already has).
If you want to do it yourself the TransactionHandler is what you want to look at. Alternatively you could dump the full database to a snapshot at times you request.
There are plans to add point-in-time recovery on the existing tx-logs, which I think would also address your question.
Actually, We need to delete and insert data in all tables when i am going to launch the app. So, please tell me the best solution
It's actually more efficient to drop the table and recreate them - and if you're going to do that then you might as well delete the database file and recreate it (and thus not subject yourself to unnecessary fragmentation or database expansion).
Yes #Dai explanation is correct.
If you going to delete table rows one by one for each table than it is memory consuming too.
It takes bit of time to remove all table's rows. Also you need to query separately for each table and need to check that constraint are not conflict with other table.
Suppose, Table A is reference to Table B than you must have to remove Table B records before going to remove Table A records.
There are many things needs to check before going to removing each tables records.
As #Dai said that if dropping table is no issue with your current implementation.
I would suggest you one step ahead of this. Why don't you remove your database any copy it from bundle.
As per my knowledge we are creating database and place it in a application bundle. So if the previous state of database is not required than you just remove database and copy new from application bundle.
This would be very fast to perform on application launch.