I have a Rails application on a Windows Server 2008 machine. The application uses a sqlite database, and I need to backup it once a day. The backup should be done by coping the database file to a mirrored folder, and a mirror-backup service will take it from there.
There is one problem. I would like to avoid copying the database file while the application uses it. If I copy the file in the middle of an action, I'll get a corrupted and\or locked database file in the backup - and restoring such a file is too much trouble, if it's even possible.
Now, is there a make the scheduled task wait until the database is not in use? Or to prevent the rest of the application from using the database while the backup task is running?
As for the scheduling, I've seen several solutions, but I haven't committed to a scheduling solution yet, so it's possible for me to depend on a scheduling-solution feature to solve the backup problem.
I would suggest using the SQLite Command Line Shell:
sqlite3 /path/to/live.db '.backup /backup/path.db'
This will allow you to backup without causing regular DB operations to fail. From the documentation:
The copy operation may be done incrementally, in which case the source
database does not need to be locked for the duration of the copy, only
for the brief periods of time when it is actually being read from.
Related
We have recently started using Duplicati for backup of some of our data systems. We run an ERP solution that uses Pervasive (v10).
When Duplicati begins its backup process, to the best of my understanding, it's using either the file date, and or the file byte size to determine what to back up.
The problem that I see with that solution is that some of the data is missing from the table. For example, the workorders module we are certain had new rows of data on the server (source machine) that were NOT copied over to the new file.
Last night we backed up our ERP platform then restored to a new location so as to do a compare of what was backed up during the evening against what the source machine had. We noticed that there are rows missing from one table in the restored backup, that are there in the source table.
The backup is being created from the data directory. We are NOT using the integrated backup that came with the ERP suite.
What I personally believe is happening is that the database isn't writing out the data to the table until the last client disconnects from the ERP software. Also, the byte size of the file missing data and the source machine are the same, even though the source file holds more data.
Last week we did the same test that we did last night and I noticed when I closed the ERP suite, the file updates its modified stamp and the new rows are added to the table, but not before the client disconnects.
Can someone shed some light on why this is happening?
Are the data files open according to Pervasive when the backup occurs? If so, you should be using some sort of agent to close the files or put them into Continuous Operation mode or Backup Agent.
From the docs:
Continuous Operations provides the ability to backup data files while
database applications are running and users are connected.
When Continuous Operation mode is started, a delta file (.^^^) is created and the original data file is 'closed' so backup programs can access the file and back it up.
Backup Agent puts a GUI front end to Continuous Operation mode but is only supported with PSQL v11 and newer.
With Duplicati, you can set --disable-filetime-check=true to ignore the timestamps and sizes, and scan each file for changes.
This option is not active by default, because it may take a long time to fully read the file contents. For normal file operations, the OS should set the timestamp, but some applications, like TrueCrypt, will revert the timestamp.
I want to import some database at my neo4j local server. I unpacked data from archive do data/databases change active.db and allow_format_migration had changed to true.
But now when I use bin/neo4j start I had error in log/neo4j. There are many lines but I think problem is with it:
Please see the attached cause exception "The database is not cleanly shutdown. The database needs recovery, in order to recover the database, please run the old version of the database on this store.".
What I did wrong?
I've read ton's of information and found the answer.
Just need to delete transaction files inside your database folder and run neo4j.
I try use neo4j HA cluster (neo4j 2.0.1), but I got error "No such log version: ..." after database copied from master.
I deleted all *.log files, but it was not help for me.
Can you help me with this problem?
TIA.
"Log" in Neo4j refers to the write-ahead-log that the database uses to ensure durability and consistency. It is stored in files named something like nioneo_logical_log.vX, where X is a number. You should never delete these files manually, even if the database is turned off, this may lead to data loss. If you wish to limit the amount of disk space used by neo for logs, use this configuration:
keep_logical_logs=128M
The error you are seeing means that the database copied cannot perform an incremental update to catch up with the master, because the log files have been deleted. This may happen normally, if the copy is very old, or if you have keep_logical_logs set to false.
What you need to do to resolve this is to set keep_logical_logs to some value that makes sense to you, perhaps "keep_logical_logs=2 days" to keep logs up to two days old around. Once you've done that, you need to do a new full store copy on your slave, which can be triggered by shutting the slave off, deleting the database folder (data/graph.db), and starting the slave back up again.
Newer versions of Neo will do this automatically.
We're considering using EF 4.3.1 code-based migrations, but aren't clear about how to integrate Migrations with our present dev/deployment methodology...
The app in-question is a desktop WPF app, with each desktop having its own SQL Server instance (each with 4 separate databases). It is deployed into a "field" environment with zero local IT support. Any database migration must be done using SQL scripts executed by the installer (probably InstallShield). There will not be anyone available who can run a command at a PMC prompt to upgrade the db when it is deployed/upgraded at a field location. Thus the ultimate "output" from EF Migrations must be a set of SQL scripts, which the installer will selectively apply.
Also, we have multiple developers making concurrent database changes.. there is NO DBA. Each developer simply checks-in their code (model) changes to TFS, and the next time they do get-latest, the changes to the model automatically cause a new database to be created on their dev system. So how can we now have each developer perform their own local migrations (rather than deleting/recreating their local databases), and then manage/consolidate/combine those migrations? And what about collisions?
During dev and unit-testing, each developer may delete their (entire) local database multiple times during a single checkout/checkin iteration. This works great with Code First, since the database gets automatically rebuilt when the app is restarted. But this means that the _MigrationHistory table in the database also gets deleted. How do we handle this? Don't we need the migration history of each dev system? If not, then where/how do we detect the aggregate changes which need to be applied to the delivered system?
I can see the value of using Migrations to deal with the mechanics of migrating a database, but what's not clear is how to take advantage of it without introducing a centralized database "change-control" bottleneck into the dev cycle, and thus losing one of the key benefits of Code First.
Any insight/advice would be greatly appreciated!
DadCat
I know this is an old question but I thought I'd post some of my experiences with EF Migrations (v6.1).
Each dev will be fine. Migrations are put into classes with a timestamp in the name, so no collisions will happen. The DB on the dev's machine will be updated after doing a get latest and running the app (or the update-database command).
Deleting the local db and recreating is fine. Just make sure the dev runs update-database before adding additional migrations or things will get out of sync. I'm a bit confused as to why they'd need to delete the local DB, but that's out of scope. You may find that your process needs to change to accommodate EF Migrations.
I can't help you with the installer question, as a similar question brought me here. The update-database command does have a -script option that will generate the proper change script, but I'm unclear how to automate that on a build server.
I need to update my application from a central server.
The application checks always if it is a correct version, against the server installation.
So when it is not, I need it to update itself.
So how can I copy the EXE if it is running? What solution do I have?
I rename the current running exe to MyTempExe.exe, copy the new exe to the correct location (request elevated privileges if necessary) and then run a separate app to restart the main app. On start up I check for MyTempExe.exe delete it if it's there.
The reason I use a separate app for the restart is I don't have a set time frame for the app to close down and need to wait for it to finish whatever it's doing, on shutdown it writes information to disk about its current state that the updated app will use to resume where the old one left off.
I don't know if it's the best solution but it's the one I use.
As you can see by all the answers there is no set way to do this, so I thought I would add the way we have successfully done this.
We never run an application directly from the network.
We run the application from the local machine and have it copy from the network on startup.
We do this using an application launcher. It downloads an XML file that contains CRC and Version Resource Values for the application files. The XML File is created during the deployment process, in a FinalBuilder Script.
The application then compares the XML File to local content, and copies down needed files. Finally we then launch the application in question. This has worked well for deploying an application that serves around 300 local users. Recently we switch from a file copy to an HTTP download as we found problems with remote user disconnecting drives.
We still still build installations (With Innosetup) to get the basic required files deployed.
Package your app with an installer such as Inno. Download and execute the installer. Have the installer search for and kill your app, or instruct the user to close it. The setup will replace your .exe, and if the app can't be killed or the user is non-cooperative, it'll issue a re-start notice.
Download new EXE to TEMP
Create Batch from EXE, content:
taskkill /PID %process id of running EXE%
copy %new EXE% %running EXE%
%EXE%
all values in %...% are placeholders
execute batch from the running EXE
delete batch
I use TMS TWebUpdate myself, for software updates. The advantage is that there a bunch of extra actions you can put into the script, if you need anything other than plain EXE updates.
I have two components at work the application executable itself and a web-service (SOAP) which provides version details and file downloads.
The application calls a method on the SOAP service to ask for the number of files in the project (project is identified by using the application.exename usually).
The soap service gets its info from an INI file, which has entries like:
[ProjectName]
NumberOfFiles=2
File1=myapp.exe;1.0.0.1
File2=mydll.dll;1.0.0.2
You just update this file at the same time as uploading your new files.
The process of updating the application this:
Get number of files available on the web service
For each file, the application asks for the name and version number from the SOAP server.
The application compares this information to its own version info and decides if the file needs updating, building a local list of files that need updating.
For each file that needs updating the application downloads the file to filename.ext.new
Finally, the application renames all filename.ext to filename.ext.old and renames filename.ext.new to filename.ext and then restarts itself. (No real need for an external app to restart your own program).
Note 1, that you may have to ask for elevation to replace files, depending on where you install your files.
Note 2: be kind to your users, think carefully before you force updates on users.
Note 3: You cannot delete a running exe, but you can rename it and then restart the new version.
Edit===
For some reference data files which cannot contain version information resources, you can have entires like File99=MyDataFile;1.1.2011 the 3 elements to the version number indicates to the client that it should check against the file date/stamp.
You could have a separate update executable whose task is to check the server version, download an updated executable if necessary, and then run the local executable.
Or you could have one executable running in two different modes: 1. on startup, check for an update, if there is one, download the executable to a download directory, run it and quit.
2. The new executable would check if it's running from the installation directory, if not, it would copy itself there, overwriting the old version, start the copy from there, and quit.
My way is the other way round: If a new version is online, promt the user to update. If he want's to (or is forced to...) I end the app and start a new exe (updater). this updater loads the update and replaces the old exe (not running). then it starts the new exe. ready. (You can of course replace other files too.) BUT: Using an Installer like InnoSetup gives you more possibilities and doesn't mix up with the regular uninstaller, so it is really better...
You can do this without running another application. Push the updates to the client from the server while running, storing in a temporary directory on the client. When you want to upgrade move all your running files to another temporary directory, move the new files into the original application directory, and just restart the application using the standard executable name on shutdown.
I upgrade client applications running on unattended machines automatically this way.