We have recently started using Duplicati for backup of some of our data systems. We run an ERP solution that uses Pervasive (v10).
When Duplicati begins its backup process, to the best of my understanding, it's using either the file date, and or the file byte size to determine what to back up.
The problem that I see with that solution is that some of the data is missing from the table. For example, the workorders module we are certain had new rows of data on the server (source machine) that were NOT copied over to the new file.
Last night we backed up our ERP platform then restored to a new location so as to do a compare of what was backed up during the evening against what the source machine had. We noticed that there are rows missing from one table in the restored backup, that are there in the source table.
The backup is being created from the data directory. We are NOT using the integrated backup that came with the ERP suite.
What I personally believe is happening is that the database isn't writing out the data to the table until the last client disconnects from the ERP software. Also, the byte size of the file missing data and the source machine are the same, even though the source file holds more data.
Last week we did the same test that we did last night and I noticed when I closed the ERP suite, the file updates its modified stamp and the new rows are added to the table, but not before the client disconnects.
Can someone shed some light on why this is happening?
Are the data files open according to Pervasive when the backup occurs? If so, you should be using some sort of agent to close the files or put them into Continuous Operation mode or Backup Agent.
From the docs:
Continuous Operations provides the ability to backup data files while
database applications are running and users are connected.
When Continuous Operation mode is started, a delta file (.^^^) is created and the original data file is 'closed' so backup programs can access the file and back it up.
Backup Agent puts a GUI front end to Continuous Operation mode but is only supported with PSQL v11 and newer.
With Duplicati, you can set --disable-filetime-check=true to ignore the timestamps and sizes, and scan each file for changes.
This option is not active by default, because it may take a long time to fully read the file contents. For normal file operations, the OS should set the timestamp, but some applications, like TrueCrypt, will revert the timestamp.
Related
I am really sorry to ask a simple question like this, but it is getting frustrating. I installed neo4j 4.0.4 on my Windows machine, created a new project as shown in the official tutorial video and set a password for my local graph. Funnily, the tutorial video ends after setting the password and opening the browser not showing how to perform Cypher queries on this newly created database. In neo4j Desktop my database is shown correctly and it seems to be up and running.
However, when I try to connect to this database via the browser, I do not see the database at all. It is so confusing when connecting to the server to specify a username and password, if you only need to set a password for your database?! The default neo4j user can see the system and default database but not my project database. In addition, I cannot link files from the project directory in Cypher queries. I tried to disable authentication, but it did not help at all.
When I issue SHOW DATABASES command, it does not list my database as well.
Update / Edit:
Seems I misunderstood the concept of projects. Every database is named neo4j - default, regardless of the name specified in the project ?!. However, I still cannot access project files. So far, I copied the files manually in the database directory under "imports". But I guess that is not the intended way.
After importing data to this default database, it still shows no data in the project itself.
Data files in the imports directory are not automatically imported into the DB. That is because neo4j has no idea how you want to store that data as nodes and relationships.
So, it is up to you to determine your desired data model, and then write the appropriate code to enforce that data model.
You can take a look at this page to learn about how to import CSV data (probably the most commonly used import data format).
perforce submit is atomic.
that means if the change list contains 3 files to be checked in,
and the the operation successfully checked in the first 2 files, but then something when wrong
when operating on the 3rd file, it has the ability to rollback the operation of the first 2 files so that everything went back to its original state.
I'm awed and really impressed with that, and tried to search for how that actually works, but couldn't seem to find it.
will appreciate a lot if someone could help me understand technical details on how all this works at the background.
thank you very much in advance.
The fact that Perforce uses a database as its source of truth makes this very simple:
The depot files are locked (in the database, as if you'd run p4 lock).
The new revision contents are uploaded to the depot archive.
The database tables are locked for the final set of checks that everything is okay.
The new revision records are written into the database and all the locks are released.
If the submit fails somewhere in part 2, nothing needs to be rolled back, because the new revision contents don't overwrite anything, and they aren't visible as part of the file history until step 4. (An unfortunate side effect of this is that Perforce can actually "leak" disk space on a failed submit, but this is generally pretty minor compared to the expected normal increase in disk usage over time.)
If you'd like to be able to watch this happen in real time, tailing the journal (P4JOURNAL) will show you the database writes as they happen, and tailing the log (P4LOG) will show you the individual phases of the submit operation from when the user initiates the operation to when the change is fully committed.
I try use neo4j HA cluster (neo4j 2.0.1), but I got error "No such log version: ..." after database copied from master.
I deleted all *.log files, but it was not help for me.
Can you help me with this problem?
TIA.
"Log" in Neo4j refers to the write-ahead-log that the database uses to ensure durability and consistency. It is stored in files named something like nioneo_logical_log.vX, where X is a number. You should never delete these files manually, even if the database is turned off, this may lead to data loss. If you wish to limit the amount of disk space used by neo for logs, use this configuration:
keep_logical_logs=128M
The error you are seeing means that the database copied cannot perform an incremental update to catch up with the master, because the log files have been deleted. This may happen normally, if the copy is very old, or if you have keep_logical_logs set to false.
What you need to do to resolve this is to set keep_logical_logs to some value that makes sense to you, perhaps "keep_logical_logs=2 days" to keep logs up to two days old around. Once you've done that, you need to do a new full store copy on your slave, which can be triggered by shutting the slave off, deleting the database folder (data/graph.db), and starting the slave back up again.
Newer versions of Neo will do this automatically.
We're considering using EF 4.3.1 code-based migrations, but aren't clear about how to integrate Migrations with our present dev/deployment methodology...
The app in-question is a desktop WPF app, with each desktop having its own SQL Server instance (each with 4 separate databases). It is deployed into a "field" environment with zero local IT support. Any database migration must be done using SQL scripts executed by the installer (probably InstallShield). There will not be anyone available who can run a command at a PMC prompt to upgrade the db when it is deployed/upgraded at a field location. Thus the ultimate "output" from EF Migrations must be a set of SQL scripts, which the installer will selectively apply.
Also, we have multiple developers making concurrent database changes.. there is NO DBA. Each developer simply checks-in their code (model) changes to TFS, and the next time they do get-latest, the changes to the model automatically cause a new database to be created on their dev system. So how can we now have each developer perform their own local migrations (rather than deleting/recreating their local databases), and then manage/consolidate/combine those migrations? And what about collisions?
During dev and unit-testing, each developer may delete their (entire) local database multiple times during a single checkout/checkin iteration. This works great with Code First, since the database gets automatically rebuilt when the app is restarted. But this means that the _MigrationHistory table in the database also gets deleted. How do we handle this? Don't we need the migration history of each dev system? If not, then where/how do we detect the aggregate changes which need to be applied to the delivered system?
I can see the value of using Migrations to deal with the mechanics of migrating a database, but what's not clear is how to take advantage of it without introducing a centralized database "change-control" bottleneck into the dev cycle, and thus losing one of the key benefits of Code First.
Any insight/advice would be greatly appreciated!
DadCat
I know this is an old question but I thought I'd post some of my experiences with EF Migrations (v6.1).
Each dev will be fine. Migrations are put into classes with a timestamp in the name, so no collisions will happen. The DB on the dev's machine will be updated after doing a get latest and running the app (or the update-database command).
Deleting the local db and recreating is fine. Just make sure the dev runs update-database before adding additional migrations or things will get out of sync. I'm a bit confused as to why they'd need to delete the local DB, but that's out of scope. You may find that your process needs to change to accommodate EF Migrations.
I can't help you with the installer question, as a similar question brought me here. The update-database command does have a -script option that will generate the proper change script, but I'm unclear how to automate that on a build server.
I have a Rails application on a Windows Server 2008 machine. The application uses a sqlite database, and I need to backup it once a day. The backup should be done by coping the database file to a mirrored folder, and a mirror-backup service will take it from there.
There is one problem. I would like to avoid copying the database file while the application uses it. If I copy the file in the middle of an action, I'll get a corrupted and\or locked database file in the backup - and restoring such a file is too much trouble, if it's even possible.
Now, is there a make the scheduled task wait until the database is not in use? Or to prevent the rest of the application from using the database while the backup task is running?
As for the scheduling, I've seen several solutions, but I haven't committed to a scheduling solution yet, so it's possible for me to depend on a scheduling-solution feature to solve the backup problem.
I would suggest using the SQLite Command Line Shell:
sqlite3 /path/to/live.db '.backup /backup/path.db'
This will allow you to backup without causing regular DB operations to fail. From the documentation:
The copy operation may be done incrementally, in which case the source
database does not need to be locked for the duration of the copy, only
for the brief periods of time when it is actually being read from.