We use TFS 2017 CI / CD pipelines, and it works great. However, the TFS 2017 databases grow averaging around 1GB per day. One database grows from 10GB to 44GB as of 10/23/2018. The growth slowly becomes unsustainable for us. We already adjusted retention policy to minimum.
Researched and read at least 30 articles. Here are some relevant articles:
TFS tbl_Content started growing very fast after using VNext build
https://mattyrowan.com/2014/04/02/need-help-tfs-tbl_content-table-and-database-growth-out-of-control/
https://developercommunity.visualstudio.com/content/problem/63712/tfs-database-size.html
Here are what I did so far:
Reviewed again and again the retention policies, and reduced to minimum (1 day 1 copy). Adjusted 'Keep Deleted' for 10 days.
Uncheck the 'Retain Build' box in release definition
Run the scripts from three articles mentioned above, and found:
a) FileContainer, has 149176 number of files, 43GB, (34GB compressed)
b) FileContainerOwner: Build, 29GB
So the main cause of the growth is Build (and artifacts).
My question is how to shrink the database size down?
I look at the tabs 'History' and 'Deleted' under build definitions.
Some records in 'History' are locked with 'Retained by release'. I can click on records and delete. But it doesn't do anything. The records are still there.
All records in 'Deleted' are still there.
So back to my question again, how do I delete these records so that the space can be reclaimed?
Thanks.
After reset the RetainedByRelease to false and waited for at least 24 hours, the growth spurt stopped and entries in tbl_content were removed daily.
So in a summary, I also did this:
Reset the RetainedByRelease to false using TFS REST API after nuget Microsoft.VisualStudio.Services.Client
Special thanks to these two threads:
https://social.msdn.microsoft.com/Forums/vstudio/en-US/5f649821-b1bf-4008-bba9-0c960e124abb/tfs-releasemanagement-vnext-quotthis-build-has-been-retained-by-a-releasequot-issue?forum=tfsbuild
Trying to get list of TFS users trough client library
The full source code to help fellow developers:
using Microsoft.TeamFoundation.Build.WebApi;
using Microsoft.TeamFoundation.Core.WebApi;
using Microsoft.VisualStudio.Services.Client;
using Microsoft.VisualStudio.Services.Common;
using Microsoft.VisualStudio.Services.WebApi;
using System;
namespace TfsRestAPIs
{
public class RestAPI
{
public static void UpdateRetainedByRelaseToFalse()
{
Uri tfsURI = new Uri("http://TFS2017:8080/tfs/YourProjectCollection");
VssCredentials creds = new VssClientCredentials();
creds.Storage = new VssClientCredentialStorage();
VssConnection connection = new VssConnection(tfsURI, creds);
var projectClient = connection.GetClient<ProjectHttpClient>();
var projects = projectClient.GetProjects().Result;
var buildClient = connection.GetClient<BuildHttpClient>();
foreach (var project in projects)
{
Log(project.Name);
if (project.Name == "YourProjectName")
{
var builds = buildClient.GetBuildsAsync(project.Id).Result;
foreach (Build build in builds)
{
if (build.BuildNumber.StartsWith("YourSearchCondition"))
try
{
if (build.RetainedByRelease.Value)
{
Log(build.BuildNumber + "'s RetainedByRelease=true");
build.RetainedByRelease = false;
var res = buildClient.UpdateBuildAsync(build, build.Id).Result;
Log(" --> RetainedByRelease is set to " + res.RetainedByRelease.Value);
}
}
catch (Exception e)
{
Log(build.BuildNumber + ":" + e.Message);
}
}
}
}
}
private static void Log(string msg)
{
Console.WriteLine(msg);
}
}
}
Related
Can anyone help me to understand the below query,Actually in our tfs we have huge number of build definition those are not in use so far hence i want to delete old build definition by seraching all the builds those are older more than one year.
how to search build definitions in TFS2013 those are no longer in use??
Please help.
Thanks,
You can query out the definitions those are older more than one year by this query in SQL Server.
SELECT [DefinitionId],[DefinitionVersion],[DefinitionName] FROM [Tfs_CollectionLC].[Build].[tbl_Definition]
WHERE [CreatedOn] < '2016-06-12 00:00:00.000'
But for the definitions are no longer in use, you have to get the last build for each definition and check whether the build FinishTime is before a year ago (Where [FinishTime] < '2016-06-12 00:00:00.000' ).
You can use TFS API/Client API to get the build details which including the build FinishTime, then filter them accordingly.You can reference below code which enumerates each team project and gets the latest build status for each of the definitions:
TfsTeamProjectCollection tfs = new TfsTeamProjectCollection(new Uri("http://tfs:8080"));
var vcs = tfs.GetService<VersionControlServer>();
var teamProjects = vcs.GetAllTeamProjects(true);
IBuildServer buildServer = (IBuildServer)tfs.GetService(typeof(IBuildServer));
foreach (TeamProject proj in teamProjects)
{
var defs = buildServer.QueryBuildDefinitions(proj.Name);
System.Console.WriteLine(string.Format("Team Project: {0}", proj.Name));
foreach(IBuildDefinition def in defs)
{
IBuildDetailSpec spec = buildServer.CreateBuildDetailSpec(proj.Name, def.Name);
spec.MaxBuildsPerDefinition = 1;
spec.QueryOrder = BuildQueryOrder.FinishTimeDescending;
var builds = buildServer.QueryBuilds(spec);
if (builds.Builds.Length > 0)
{
var buildDetail = builds.Builds[0];
System.Console.WriteLine(string.Format(" {0} - {1} - {2}", def.Name, buildDetail.Status.ToString(), buildDetail.FinishTime));
}
}
System.Console.WriteLine();
}
Reference this thread:TFS API - How to query builds independent of which build definition they belong to
Update: Based on #Dhurva's comments below:
We can also use the utility from github TFS Manager that finds all build definition from team project:
https://github.com/jelledruyts/TfsTeamProjectManager
I want to display just the first 100 commits of a repository. I used the linux-repo to test this:
const int maxSize = 100;
Stopwatch sw = new Stopwatch();
Console.WriteLine( "Getting Commits in own Thread" );
sw.Start();
using( Repository repo = new Repository( path_to_linux_repo ) )
{
ICommitLog commits = repo.Commits.QueryBy( new CommitFilter { Since = "HEAD" } );
int index = 0;
foreach( Commit commit in commits )
{
if( index++ > maxSize ) break;
}
}
sw.Stop();
Console.WriteLine( "Took {0}ms for {1} entries", sw.ElapsedMilliseconds, maxSize );
This simple loop takes over 9000ms on my machine. Its FAR faster when using a repo with less commits, but why is it so slow in repos with a lot of commits?
Another question: is it possible to just retrieve a given number of commits e.g. to page through
all commits?
I can reproduce here. It's definitely far too long to take. It looks as though libgit2 is enqueueing the full graph before returning, which would be a bug with the given settings. Would you mind opening an issue?
As for retrieving a number of commits, the iteration is pull-based, so you will only grab as many out of the repository as you ask for. the commit log implements IEnumerable so you can use whatever LINQ methods you like (or do it manually as in this example).
UPDATE:
The bug was quite embarrassing, but there's a PR to fix it in libgit2 which will make its way into libgit2sharp releases in due course. With the fix, this test now takes ~80ms. Thanks for bringing it up.
UPDATE 2:
The fix is now available in the vNext branch of LibGit2Sharp.
I'm working on customizing our build activity. I'd like to have your help for an issue.
Following is our version control hierarchy.
Main
|- Dev
|- QA
we are working on Dev branch and while taking the build we need to merge Dev branch to Main then to QA.
Main is the root branch as you might know.
In our build template, I've added two custom activities to merge one from Dev to Main and another one to merge from Main to QA. Following is the code for the custom activity.
protected override string Execute(CodeActivityContext context)
{
string lstrStatus = string.Empty;
string lstrSourceBranchPath = context.GetValue(this.SourceBranchPath);
string lstrTargetBranchPath = context.GetValue(this.TargetBranchPath);
// Obtain the runtime value of the input arguments
Workspace workspace = context.GetValue(this.Workspace);
GetStatus status = workspace.Merge(lstrSourceBranchPath,
lstrTargetBranchPath,
null,
null,
LockLevel.None,
RecursionType.Full,
MergeOptions.None);
// resolve the conflicts, if any
if (status.NumConflicts > 0)
{
Conflict[] conflicts = workspace.QueryConflicts(new string[]
{ lstrTargetBranchPath }, true);
foreach (Conflict conflict in conflicts)
{
conflict.Resolution = Resolution.AcceptTheirs;
workspace.ResolveConflict(conflict);
}
}
// checkin the changes
PendingChange[] pendingChanges = workspace.GetPendingChanges();
if (pendingChanges != null && pendingChanges.Length > 0)
{
workspace.CheckIn(pendingChanges, "Merged by MERGE BRANCHES activity");
}
return lstrStatus;
}
Problem is, merging happens perfectly in the server. But, it's not getting reflected in the local folder. I tried to add SyncWorkspace activity after each Merge custom activity. Still not working.
My guess was that a SyncWorkspace should be the only thing to do.
You could try doing a RevertWorkspace before that.
EDIT
After you now stated that even this wouldn't work, I would generate a bug against MS at least to get an official answer.
In the meanwhile you can try with the following method, which I absolutely see as an overkill: Once you have checked in, redo all the steps within sequence Initialize Workspace.
If even that doesn't work I'd consider two different builds, one that does your merge & one that does the actual build. You can then organize a scheme where your first build, once it's done, triggers the second one. Here is a good resource for that.
We'd like to generate build notes with the following format:
1) Associated ChangeSets:
2) - ChangeSet 45241, by Joe: "Patching fix for foobar"
3) 'Foo.cs' integrated from dev v. 22 to qa v. 7
4) 'Bar.cs' integrated from dev v. 9 to qa v. 3
So far, we have a custom build step that accomplishes 1) and 2). It looks at the information produced by the 'AssociatedChangesetsAndWorkItems' TFS Build Activity. Here is the code:
protected override bool Execute(CodeActivityContext context)
{
StreamWriter sw = new StreamWriter(Path.Combine(BuildNotesPath.Get(context),"build-notes.txt"));
sw.WriteLine("Associated ChangSets:");
foreach (Changeset changeset in BuildAssociatedChangesets.Get(context))
{
sw.WriteLine(string.Format("ChangeSet {0}, by {1}: {2}", changeset.ChangesetId, changeset.Committer, changeset.Comment));
foreach (Change change in changeset.Changes)
{
foreach (MergeSource source in change.MergeSources)
sw.WriteLine(string.Format("\'t{0}': integrated from dev v. {1} to qa v. {2}", source.ServerItem, source.VersionFrom, source.VersionTo));
}
}
sw.Flush();
sw.Dispose();
return true;
}
The problem we're having is that the 'MergeSources' field is always an empty list. What do we have to do to get that field populated?
Take a look at MergeWorkItemsEventHandler.MergeWorkItems and .GetMergeHistory along with the Extensions.PendingMerges method in the TFS 2010 Merge Work Items Event Handler for an example of getting merge sources.
Jakob Enh deserves the credit for this answer. :)
Is there any way to copy a build definition? I work in a mainline source control methodology which utilizes many different branches that live for very short periods (ie. a few days to a week). I'd really like to copy a build template and just change the solution to build. Is there any way to do this?
You can download the new TFS 2010 power tools. It has the option to clone a build definition.
See http://msmvps.com/blogs/molausson/archive/2010/10/21/clone-a-build-definition.aspx for an example
Note: Be aware that the Clone only works when you did NOT pop out the Build window.
You can write an add-in to do it. Here's the code to copy an existing build definition:
static IBuildDefinition CloneBuildDefinition(IBuildDefinition buildDefinition)
{
var buildDefinitionClone = buildDefinition.BuildServer.CreateBuildDefinition(
buildDefinition.TeamProject);
buildDefinitionClone.BuildController = buildDefinition.BuildController;
buildDefinitionClone.ContinuousIntegrationType = buildDefinition.ContinuousIntegrationType;
buildDefinitionClone.ContinuousIntegrationQuietPeriod = buildDefinition.ContinuousIntegrationQuietPeriod;
buildDefinitionClone.DefaultDropLocation = buildDefinition.DefaultDropLocation;
buildDefinitionClone.Description = buildDefinition.Description;
buildDefinitionClone.Enabled = buildDefinition.Enabled;
buildDefinitionClone.Name = String.Format("Copy of {0}", buildDefinition.Name);
buildDefinitionClone.Process = buildDefinition.Process;
buildDefinitionClone.ProcessParameters = buildDefinition.ProcessParameters;
foreach (var schedule in buildDefinition.Schedules)
{
var newSchedule = buildDefinitionClone.AddSchedule();
newSchedule.DaysToBuild = schedule.DaysToBuild;
newSchedule.StartTime = schedule.StartTime;
newSchedule.TimeZone = schedule.TimeZone;
}
foreach (var mapping in buildDefinition.Workspace.Mappings)
{
buildDefinitionClone.Workspace.AddMapping(
mapping.ServerItem, mapping.LocalItem, mapping.MappingType, mapping.Depth);
}
buildDefinitionClone.RetentionPolicyList.Clear();
foreach (var policy in buildDefinition.RetentionPolicyList)
{
buildDefinitionClone.AddRetentionPolicy(
policy.BuildReason, policy.BuildStatus, policy.NumberToKeep, policy.DeleteOptions);
}
return buildDefinitionClone;
}
The following tool (VS Addin) will satisfy your requirement>
Community TFS Build Manager
http://visualstudiogallery.msdn.microsoft.com/16bafc63-0f20-4cc3-8b67-4e25d150102c
I just had a need to copy build definitions, and found Jim's answer above to be helpful. However, being new to the TFS API, I needed help connecting to the server and getting the existing build definition through code. These two links helped fill in the gaps:
http://msdn.microsoft.com/en-us/library/bb286958.aspx
http://geekswithblogs.net/jakob/archive/2010/04/26/creating-a-build-definition-using-the-tfs-2010-api.aspx
You can right click the build definition and select 'clone build definition' to copy the definition file. You can then edit it from there.
Here is the soltion if you want to move the Build definition from one Team Project to other Team project.
public void MoveBuild(string fromTeamProject, string toTeamProject, string buildName, string newBuildName)
{
var _server = TfsTeamProjectCollectionFactory.GetTeamProjectCollection(new
Uri("http://Mytfs:8080/defaultcollection"));
IBuildServer _buildServer = _server.GetService<IBuildServer>();
var buildDetails = _buildServer.QueryBuildDefinitions(fromTeamProject);
foreach (var fromBuild in buildDetails)
{
if (fromBuild.Name != buildName) continue;
var newBuildDefinition = _buildServer.CreateBuildDefinition(toTeamProject);
newBuildDefinition.Name = !string.IsNullOrEmpty(newBuildName) ? newBuildName : fromBuild.Name;
newBuildDefinition.BuildController = fromBuild.BuildController;
// This finds the template to use
foreach (var mapping in fromBuild.Workspace.Mappings)
{
newBuildDefinition.Workspace.AddMapping(
mapping.ServerItem, mapping.LocalItem, mapping.MappingType, mapping.Depth);
}
newBuildDefinition.DefaultDropLocation = fromBuild.DefaultDropLocation;
newBuildDefinition.Description = fromBuild.Description;
// buildDefinition.Workspace.AddMapping(build.Workspace.);
newBuildDefinition.Process = _buildServer.QueryProcessTemplates(fromBuild)[2];
newBuildDefinition.ProcessParameters = fromBuild.ProcessParameters;
newBuildDefinition.Enabled = false;
newBuildDefinition.Save();
}//end of for each loop
}
From your message it is not clear which template is your build definition using (default, upgrade or lab management). If I understand correctly you would like to easily set up a build definition which builds the same solution but from a different branch.
One thing that you could try instead of copying the definition is to edit it. When the branch dies, rename the build definition (might help with reporting), change the workspace mapping of the build and you should be done.
Thanks, Ladislau