I have an installation of TFS 2018 which is split over an app and database server. Both servers run Windows Server 2016. Frequently I get an error that the scheduled backup has failed because BackupSets.xml is locked.
Process Explorer shows that it is locked by SYSTEM. There is plenty of disk space on the backup disk (Currently 1.3TB and the full backup is 150GB).
The only way I have found to resolve this is to reboot both the app and database servers. This is not a viable long term solution to this.
Has anybody else encountered anything similar? Can you suggest a solution?
[23.03.2018 19:04:51] [Error]
Exception Message: The process cannot access the file
'\\DatabaseServer\TFSBackups\BackupSets.xml' because it is being used by
another process. (type IOException)
Exception Stack Trace: at System.IO.__Error.WinIOError(Int32 errorCode,
String maybeFullPath)
at System.IO.File.InternalDelete(String path, Boolean checkHost)
at Microsoft.TeamFoundation.Admin.BackupSets.Save(String folder)
at Microsoft.TeamFoundation.Admin.Jobs.DatabaseBackupJob.TakeTransactionalBackups (IVssRequestContext requestContext, BackupSettings settings, ITFLogger logger)
at Microsoft.TeamFoundation.Admin.Jobs.DatabaseBackupJob.RunBackupJob(IVssRequestContext requestContext, BackupJobType requestedJobType, String& resultMessage)
[23.03.2018 19:04:51] [Info] Backup Job Finished. Completed: Transactional. Result = Failed
First, suggest you to check the Event View on both server to see if there are more log info for troubleshooting. Also check the process especially for TFS related if some thing hangs on.
In this case you could try below solution:
Delete the Backupsets.xml file altogether
Reconfigured Backups using the wizard - Now that it didn't find the
xml file at all it created a new one.
Ran a full backup - which should be successfully.
Double check if the scheduled backups will now also work from now on.
Related
We are receiving the error below for both the 'Full' and 'Incremental' Analysis Database Sync jobs in TFS 2015. I've searched for this error online and none of the answers worked in our situation. Does anyone have experience with this along with a possible solution?
[Full Analysis Database Sync]: --->
AnalysisDatabaseProcessingType=Full, needCubeSchemaUpdate=True. --->
Microsoft.TeamFoundation.Warehouse.WarehouseException: TF221122: An
error occurred running job Full Analysis Database Sync for team
project collection or Team Foundation server TEAM FOUNDATION. --->
Microsoft.TeamFoundation.Warehouse.WarehouseException: Failed to
Process Analysis Database 'Tfs_Analysis'. ---> System.SystemException:
The trust relationship between the primary domain and the trusted domain failed. at
System.Security.Principal.NTAccount.TranslateToSids(IdentityReferenceCollection
sourceAccounts, Boolean& someFailed) at
System.Security.Principal.NTAccount.Translate(IdentityReferenceCollection
sourceAccounts, Type targetType, Boolean forceSuccess) at
System.Security.Principal.NTAccount.Translate(Type targetType) at
Microsoft.TeamFoundation.Warehouse.OlapCreator.AddAccountToRole(Role
role, String accountName, Boolean throwOnError) at
Microsoft.TeamFoundation.Warehouse.OlapCreator.AddAccountsToRole(Database
database, String roleName, IEnumerable1 accountNames, Boolean
ignoreAccountResolutionFailures) at
Microsoft.TeamFoundation.Warehouse.OlapCreator.CreateOrUpdateRoles(Server
server, Database database, IEnumerable1 serviceAccountNames, Boolean
ignoreServiceAccountResolutionFailures, IEnumerable1
readerAccountNames, Boolean ignoreReaderAccountResolutionFailures) at
Microsoft.TeamFoundation.Warehouse.OlapCreator.UpdateOlapSchema(Server
server, IEnumerable1 readerAccountNames, WarehouseConfig
warehouseConfig) at
Microsoft.TeamFoundation.Warehouse.TFSOlapProcessComponent.UpdateOlapSchema()
at
Microsoft.TeamFoundation.Warehouse.TFSOlapProcessComponent.ProcessOlap(AnalysisDatabaseProcessingType
processingType, WarehouseChanges warehouseChanges, Boolean
lastProcessingFailed, Boolean cubeSchemaUpdateNeeded) --- End of inner
exception stack trace --- at
Microsoft.TeamFoundation.Warehouse.TFSOlapProcessComponent.ProcessOlap(AnalysisDatabaseProcessingType
processingType, WarehouseChanges warehouseChanges, Boolean
lastProcessingFailed, Boolean cubeSchemaUpdateNeeded) at
Microsoft.TeamFoundation.Warehouse.AnalysisDatabaseSyncJobExtension.RunInternal(IVssRequestContext
requestContext, TeamFoundationJobDefinition jobDefinition, DateTime
queueTime, String& resultMessage) at
Microsoft.TeamFoundation.Warehouse.WarehouseJobExtension.Run(IVssRequestContext
requestContext, TeamFoundationJobDefinition jobDefinition, DateTime
queueTime, String& resultMessage) --- End of inner exception stack
trace ---
This is a very common problem due to the error ID TF221122.
To narrow down the issue, suggest you could look in the OLAP log file, if there is useful information for what had happenеd there exactly. Usually the log is located under the OLAP folder of the MSAS installation (\Program Files\Microsoft SQL Server\MSASX.MSSQLSERVER\OLAP\Log), and it could be called msmdsrv.log .
If there isn’t proper information, also check the Event View on server if there are some more error message.
Back to the error info, The trust relationship between the primary domain and the trust domain failed. It seems there's the trust issue in your two domains, please double check it first.
Besides also try to delete the whole Tfs_Analysis Database/Tfs_Warehouse Database, and run the “Rebuild” operation from the Application Tier following the steps below:
Open Team Foundation Server Administration console.
Go to Reporting and there is “Start Rebuild”.
Note that this action could be used to rebuild your data warehouse and analysis database for the cube. Once you start it, it will take offline, delete, and rebuild both your Warehouse and Analysis Services databases. Based on the amount of data you have n your TFS DB, it could take significant amount of time to complete these jobs. The reports will not show any data until the above databases are not rebuilt successfully.
Take a look at below similar issue:
TFS2010 TF221122: An error occurred running job Full Analysis
Database Sync: the trust relationship between the primary domain and
the trusted domain failed
TF221122: An error occurred running job Full Analysis Database Sync for team project collection or Team Foundation server TEAM FOUNDATION
Our server support group found in the Event Viewer on the TFS server that the password was not valid for the user ID in three of the TFS services. I had changed the password for our TFS service account a while ago, but apparently did not update it in the services.
The "trust relationship" failure was resolved once I corrected the TFS service account password in the services below.
Team Foundation Ssh Service
Visual Studio Team Foundation Background Job Agent
Visual Studio Team Foundation Build Service Host 2015
Sporadically get the error
An error was encountered when processing operation 'Delete Directory'
on
'D:\xxx'. The error code was
0x80070091. The directory is not empty.
I'm using web deploy WMSVC and the user has full access to the root of the site (set up through IIS > Configure Web Deploy Publishing
EDIT
Or less frequently:
C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\Web\Microsoft.Web.Publishing.targets (4377): Web deployment task failed. (Unable to perform the operation ("Delete File") for the specified directory ("D:\xxx\bin\Microsoft.Practices.ServiceLocation.dll"). This can occur if the server administrator has not authorized this operation for the user credentials you are using. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_INSUFFICIENT_ACCESS_TO_SITE_FOLDER.)
I follow the instructions for the user (a domain user) but it makes no difference. The publish settings that get saved have publishMethod="MSDeploy" instead of WMSVC.
Note, it deploys correctly 95% of the time.
I know this sounds stupid, but do you have those folders open (or children thereof) on any computer? Especially if you had a dead remote desktop session or someone accessing it over the network share it will get a denial to be able to delete the directory.
Burned me a number of times, you're issue may be different though. I realize this is a late response, but found it while searching for my own issue and though I cuold help.
Team,
Its frustrating so far today with TFS 2012, I have been queuing a build and the build server is unable to copy/drop to the specified folder. I have installed TFS 2012 on Win Server 2008 R2. Basically the error says that the access to the destination folder is denied and so it is not able to copy. The TFS admin console shows the build service is running as LocalService. And I granted full control to the local Service account to the drop folder. I tried to even fiddle with Share tab in the properties of the folder. There is no network as such any way, only single machine with all the TFS services running on it. I even tried to put the drop folder on C drive itself without any success. Can some one please throw some light? What am I missing? Seems to be some thing trivial.
The following is the exception details.
Exception Message: TF270016: An error occurred publishing log files from 'C:\Builds\1\Tfs_TeamProjectTrail\Tfs_Trial1_CI\Sources\Main\Tfs_Trial1\Tfs_Trial1.log' to '\localhost\d$\VD\Tfs_Trial1_CI\Tfs_Trial1_CI_20130309.21\logs'. Details: Access to the path '\localhost\d$\VD\Tfs_Trial1_CI\Tfs_Trial1_CI_20130309.21\logs' is denied. (type PublishLogFileException)
Exception Stack Trace: at System.Activities.Statements.Throw.Execute(CodeActivityContext context)
at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)
at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)
Inner Exception Details:
Exception Message: Access to the path '\localhost\d$\VD\Tfs_Trial1_CI\Tfs_Trial1_CI_20130309.21\logs' is denied. (type UnauthorizedAccessException)
Exception Stack Trace: at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.Directory.InternalCreateDirectory(String fullPath, String path, Object dirSecurityObj, Boolean checkHost)
at System.IO.Directory.InternalCreateDirectoryHelper(String path, Boolean checkHost)
at Microsoft.TeamFoundation.Common.FileSpec.CopyFile(String oldPath, String newPath, Boolean overwriteExisting)
at Microsoft.TeamFoundation.Build.Workflow.Activities.WindowsDropProvider.CopyDirectory(String sourceDirectory, String targetDirectory)
at Microsoft.TeamFoundation.Build.Workflow.Activities.WindowsDropProvider.CopyDirectory(String sourceDirectory, String targetDirectory, String[] renameIfExists)
at Microsoft.TeamFoundation.Build.Workflow.Activities.PublishLogFile.Execute(CodeActivityContext context)
This is just a permissions issue.
Start by running the build service as a local user / domain user (if applicable) and get that to work.
Create a proper share rather than using the default admin shares (d$)
Make sure that both sharing and security tab on the folder properties allow write permissions to that user.
The error log says "\localhost" the correct way to access this would be "\\localhost" (this could me a stackoverflow editor issue though and not relevant)
I don't currently have time to write a complete answer, but I can expand later.
ProcMon is also a great program to use when troubleshooting this kind of thing.
I think this is a good starting point.
Simply make sure Drop directory and Build service have full privilegies for "Local service" group.
Default account with which tfs build service is run is nt authority\network service. After compiling solution\project, it is this user which tries to copy bits to the build share.
So add permissions to this account on your build share.
[TFS2013] I had this error when I turned off the drop folder option, and removed the path. By specifying a drop folder again (UNC) for the output files in the build configuration solved the problem.
It seems TFS is twitchy if you don't have a drop folder at all, even though it gives you the option to not have one.
I cannot seem to get anything out of SQLite other than "Unable to open the database file" on IIS. I'm convinced SQLite's error messages are as brusque as Oracle's.
Pre-deployment in Visual Studio 2010/IIS Express I can both read and write to the file.
When I tried to read/write it with the same VS2010 project deployed to IIS7.5, all "create", "read" and "write" commands fail.
The same occurs when I deploy the database file through the project and try to read it.
I've given Full control access to App_Data and the database file to the following users: IIS_IUSRS, IUSRS, DefaultAppPool, and Everyone.
I've looked at:
SQLite problem "unable to open the database file" (The problem automatically went away for the user) and a number of other similar questions, most of which were solved by changing permissions, changing to a writable directory (App_Data should be writable, no?) or changing a relative path to an absolute one (which |DataDirectory| should resolve to).
<connectionStrings>
<add name="sqlite" connectionString="Data Source=|DataDirectory|\datatables.sqlite;Version=3;" />
</connectionStrings>
Have I missed anything?
<Error>
<Message>An error has occurred.</Message>
<ExceptionMessage>Unable to open the database file</ExceptionMessage>
<ExceptionType>System.Data.SQLite.SQLiteException</ExceptionType>
<StackTrace>
at System.Data.SQLite.SQLite3.Open(String strFilename, SQLiteConnectionFlags connectionFlags, SQLiteOpenFlagsEnum openFlags, Int32 maxPoolSize, Boolean usePool) at System.Data.SQLite.SQLiteConnection.Open() at AjaxSource.Models.Database.query(String sql, Dictionary`2 parameters) in D:\Tools\Dropbox\Projects\myprojects\AjaxSource\AjaxSource\Models\Database.cs:line 48 at AjaxSource.Models.aaDataModel..ctor() in D:\Tools\Dropbox\Projects\myprojects\AjaxSource\AjaxSource\Models\aaDataModel.cs:line 18 at AjaxSource.Controllers.API.TableDataController.Get() in D:\Tools\Dropbox\Projects\myprojects\AjaxSource\AjaxSource\Controllers\API\TableDataController.cs:line 15 at lambda_method(Closure , Object , Object[] ) at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass13.<GetExecutor>b__c(Object instance, Object[] methodParameters) at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.Execute(Object instance, Object[] arguments) at System.Threading.Tasks.TaskHelpers.RunSynchronously[TResult](Func`1 func, CancellationToken cancellationToken)
</StackTrace>
</Error>
I had a problem opening Sqlite database in IIS
(0x80004005): unable to open database file unable to open database
file
what helped me to solve the problem is changing the "Identity" of the application pool to "LocalSystem"
Application Pools -> DefaultAppPool (or another pool you are working with) -> Advanced Settings -> Identity -> LocalSystem
Hopefully it will save someone time...
We used to encounter this problem on the machines of our customers most often, and I have made a lot of investigation upon it and finally we've solved it.
First of all, you need to make sure that your application has read/write access to the database file as well as the folder containing the database file. In most situations, the check will fix the issue; but we are not among those fortunate ones.
In our case, where the application makes very highly concurrent access to the database, it's most probably related to the journal mode of the database, which usually is DELETE by default. That is to say, the rollback journal will be deleted once the corresponding transaction is committed or rolled back. On **nix systems, it's no error; but on Windows systems, it's another story, as can be seen below. On Windows systems, the problem could take place in the following scenario:
A journal file (say A) is created by a SQLite thread.
Another thread P tries to open A.
SQLite has finished its transaction and so deletes the journal file A.
But now thread P has handle for A, so SQLite enters the “Delete Pending” state.
SQLite starts another transaction and has to recreate the journal file (with the same name).
Windows reports ERROR_DELETE_PENDING error (see here for more information: http://blogs.msdn.com/b/oldnewthing/archive/2007/11/09/6001644.aspx)
SQLite report SQLITE_CANTOPEN error (that is, "Unable to open the database file").
P finally releases the file handle and A is deleted.
One solution is to use PERSIST or TRUNCATE journal mode instead of DELETE. See here for more details: http://www.sqlite.org/pragma.html#pragma_journal_mode
In this way, the rollback journal (.-journal) is not deleted at all, and thus we can get rid of the problem you said. I hope this helps.
Well I don't know if this will answer your question, but in my case I had a dummy error which is that I gave a wrong relatif URL to the database file when I copied my code from a commandline projet to the mvc one. Hope someone will find this useful.
It wasn't a permissions problem. The problem is that I didn't check that |DataDirectory| actually resolved to the AppData directory like it's supposed to.
On the IIS 7.5 server, |DataDirectory| resolves to C:\inetpub\wwwroot\AjaxSource\App_Data, but the actual AppData is directory is C:\inetpub\wwwroot\AjaxSource\bin\App_Data.
The errors I caught using Fiddler never mention any paths, but imply a permissions problem. I had one of my views display the resolved directory with:
<h1>#AppDomain.CurrentDomain.GetData("DataDirectory")</h1>
(Oddly, this isn't documented).
I have a windows service that is failing to start, giving an error "Error 1053: The service did not respond to the start or control request in a timely fashion".
Running the service in my debugger works fine, and if I double click on the the service .exe on the remote machine a console window pops up and continues to run without problem - I can even see log messages showing me that the program is processing everything the way it should be.
The service had been running fine previously, though this is my first time, personally, trying to deploy it with the most recent changes made to the program. I've evaluated those changes and cant figure out how they might cause this problem, particuarly since everything runs fine when not started as a service.
The StartRoutine() method of the service impelmentation is empty, so should be returning in a "timely fashion".
I've checked the event logs on the computer, and it doesn't give any additional information other than it didn't hear back from the service in the 30 second requisite time frame.
Since it works on my machine, and as a double-clicked executable, how would I go about figuring out why it fails as a service?
Oh, and it's .NET 2.0, so it shouldn't be affected by the 1.1 framework bug that exhibited this symptom (http://support.microsoft.com/kb/839174)
The box is a windows server 2003 R2 machine running SP2.
This is a misleading error. It's probably an unhandled exception.
Empty your OnStart() handler then try this in your constructor...
public MainService()
{
InitializeComponent();
try
{
// All your initialization code goes here.
// For instance, my exception was caused by the lack of registry permissions
;
}
catch (Exception ex)
{
EventLog.WriteEntry("Application", ex.ToString(), EventLogEntryType.Error);
}
}
Now check the EventLog on your system for your Application Error.
Could be a number of things and it might help to get a stack trace on the machine exhibiting the problem. There are a number of ways to do this but the point is that you have to see where this is failing in the code.
You can do this with remote debugging, but a simple thing might be to just log to the event logger, or file log if you have that. Literally, putting "WriteLine("At class::function()") throughout portions of the code to see if you've made it there.
This will at least get you looking in the right direction (which ultimately is the code).
Update:
See Microsoft's How to Debug Windows Services article for details in troubleshooting startup problems using WinDbg.
This related question details nice ways to debug services that are written in .NET.
I agree with Scott, the easiest way to find out what's happening is to put some traces in the start-up code (maybe it doesn't even come to your start-up code).
If this doesn't help, you can post your code here so others can take a look.
perhaps lacking some dependence, try this :
- deregister your service
- register again
If fail at register means that lack an module.
If the StartRoutine is empty, you are probably starting it somewhere else.
IIRC you need to fire off a worker thread, and then return from StartRoutine.
One of the problems which may lead to this error is if windows service which needs to be deployed consists of some error i.e it may be simple authorization error or anything as in my case I have referenced some folders and files for logging which were not existing, but when provided the right path of those file and folders it solved my problem.
I ran through every post on this particular subject and none of the responses solved the problem, so I'm adding this response in case this helps someone else. Admittedly this only applies to a new service, not this specific case.
I was writing a File listening service. As a console app, it worked perfectly. When I ran it as a service, I got the same error as above. What I didn't know (and many of the MSDN articles about services conveniently leave out) is that you need to have your class executed from within ServiceBase.Run( YourClassName());. Otherwise, your app executes and immediately terminates and because it terminated, you get the error above even if no error or exception occurred. Here is a link to an article about this. It actually discusses setting up your app for dual use - Console app and service: Create a combo command line / Windows service app
I had that issue and the source of my problem was config file. I edited it in notepad and notepad added one special character which cause service not to run properly because config file was ruined. I saw that special character in notepadd++ and after delete it, service started to run successfully as previous did.
In my case, the correct .NET framework was not installed on the server that I was installing the Windows service on.
One other reason is If you copy the DLL in 'debug' mode to installation folder this issue will come.What you need to do is Run the project in 'Release' mode copy the DLL or directly form Release folder rather than Debug folder,,and copy that DLL in to installation folder,it will work.You can see the reduction in size of DLL ,it will not contain any debug symbols and like that