Troubleshooting MSBuild failures - tfs

Question

I am trying to convert my shop from using VSS to TFS.
Our current codebase consists of a large number of Visual Studio 2008 solutions and I've noticed some of them seem to cause MSBuild to fail, but I cannot figure out why. The actual solutions themselves and the projects they hold build correctly, but the overall build still fails.
The ErrorsAndWarnings.txt and Release.txt file show no errors, just warnings (various little things in the code we need to clean up)
The BuildLog.txt file is, of course, enormous and has tons and tons of difficult-to-comprehend text in it. As best I can tell though, when it gets to one of these SLN files that give it problems, MSBuild just flat out crashes and dies.
Task "MSBuild"
Global Properties:
Configuration=Release
Platform=Any CPU
OutDir=C:\TFS\REPOSITORY\Full\Binaries\Release\
PublishDir=C:\TFS\REPOSITORY\Full\Binaries\Release\
SkipInvalidConfigurations=true
RunCodeAnalysis=false
VCBuildOverride=C:\TFS\REPOSITORY\Full\Sources\Code\Solution\Solution.sln.Release.vsprops
VCBuildAdditionalLibPaths=
VCBuildAdditionalOptions=
VCBuildToolPath=
VCBuildUseEnvironment=
TeamBuildConstants=_TEAM_BUILD_
TargetsNotLogged=GetTargetPath;GetNativeManifest;GetCopyToOutputDirectoryItems
C:\Program Files\MSBuild\Microsoft\VisualStudio\TeamBuild\Microsoft.TeamFoundation.Build.targets(1008,5):
error MSB4018: The "MSBuild" task failed unexpectedly.
error MSB4018: System.NullReferenceException: Object reference not set to an instance of an object.
error MSB4018: at Microsoft.Build.BuildEngine.SolutionWrapperProject.AssignDependencyLevel(ProjectInSolution project, SolutionParser solution, Dictionary`2 projectsByDependencyLevel)
error MSB4018: at Microsoft.Build.BuildEngine.SolutionWrapperProject.AssignDependencyLevel(ProjectInSolution project, SolutionParser solution, Dictionary`2 projectsByDependencyLevel)
error MSB4018: at Microsoft.Build.BuildEngine.SolutionWrapperProject.AssignDependencyLevels(SolutionParser solution, Dictionary`2 projectsByDependencyLevel)
error MSB4018: at Microsoft.Build.BuildEngine.SolutionWrapperProject.CreateSolutionProject(SolutionParser solution, Project msbuildProject, BuildEventContext projectBuildEventContext, String wrapperProjectToolsVersion, Engine parentEngine, String solutionProjectCache)
error MSB4018: at Microsoft.Build.BuildEngine.SolutionWrapperProject.Generate(SolutionParser solution, Project msbuildProject, String toolsVersionOverride, BuildEventContext projectBuildEventContext)
error MSB4018: at Microsoft.Build.BuildEngine.Project.Load(String projectFileName, BuildEventContext buildEventContext, ProjectLoadSettings projectLoadSettings)
error MSB4018: at Microsoft.Build.BuildEngine.Engine.GetMatchingProject(Project existingProject, String projectFullPath, BuildPropertyGroup globalPropertiesToUse, String toolsVersion, String[] targetNames, BuildEventContext buildEventContext, Boolean toolsVersionPeekedFromProjectFile)
error MSB4018: at Microsoft.Build.BuildEngine.Engine.BuildProjectFileInternal(BuildRequest buildRequest)
error MSB4018: at Microsoft.Build.BuildEngine.Engine.EngineBuildLoop(BuildRequest terminatingBuildRequest)
error MSB4018: at Microsoft.Build.BuildEngine.TaskExecutionModule.BuildProjectFile(Int32 handleId, String[] projectFileNames, String[] targetNames, IDictionary[] globalPropertiesPerProject, IDictionary[] targetOutputsPerProject, EngineLoggingServices loggingServices, String[] toolsVersions, Boolean useResultsCache, Boolean unloadProjectsOnCompletion, BuildEventContext taskContext)
error MSB4018: at Microsoft.Build.BuildEngine.EngineProxy.BuildProjectFilesInParallel(String[] projectFileNames, String[] targetNames, IDictionary[] globalProperties, IDictionary[] targetOutputsPerProject, String[] toolsVersions, Boolean useResultsCache, Boolean unloadProjectsOnCompletion)
error MSB4018: at Microsoft.Build.Tasks.MSBuild.ExecuteTargets(ITaskItem[] projects, Hashtable propertiesTable, ArrayList targetLists, Boolean stopOnFirstFailure, Boolean rebaseOutputs, IBuildEngine2 buildEngine, TaskLoggingHelper log, ArrayList targetOutputs, Boolean useResultsCache, Boolean unloadProjectsOnCompletion, String toolsVersion)
error MSB4018: at Microsoft.Build.Tasks.MSBuild.BuildProjectsInParallel(Hashtable propertiesTable, ArrayList targetLists, Boolean success, Boolean[] skipProjects)
error MSB4018: at Microsoft.Build.Tasks.MSBuild.Execute()
error MSB4018: at Microsoft.Build.BuildEngine.TaskEngine.ExecuteInstantiatedTask(EngineProxy engineProxy, ItemBucket bucket, TaskExecutionMode howToExecuteTask, ITask task, Boolean& taskResult)
Done building target "CoreCompileSolution" in project "TFSBuild.proj" -- FAILED.
Done Building Project "C:\TFS\REPOSITORY\Full\BuildType\TFSBuild.proj" (CompileSolution target(s)) -- FAILED.
Done executing task "MSBuild" -- FAILED.
Done building target "CoreCompileConfiguration" in project "TFSBuild.proj" -- FAILED.
Done Building Project "C:\TFS\REPOSITORY\Full\BuildType\TFSBuild.proj" (CompileConfiguration target(s)) -- FAILED.
Done executing task "MSBuild" -- FAILED.
Done building target "CoreCompile" in project "TFSBuild.proj" -- FAILED.
Done Building Project "C:\TFS\REPOSITORY\Full\BuildType\TFSBuild.proj" (CoreCompile target(s)) -- FAILED.
Done executing task "MSBuild" -- FAILED.
Done building target "CallCompile" in project "TFSBuild.proj" -- FAILED.
I know pasting incredibly verbose error messages in Stack Overflow is lame but the core thing to take from the above is MSBuild is crashing and I can't figure out why. It gives the usual "Object reference not set..." error, which is pretty much universally useless when it's not your code or program you're troubleshooting.
I see that it's referencing TeamFoundation.Build.targets. I look up that file that file but it doesn't really jump out at me what the problem could be. (the line after the comment is the (1008,5) referenced above)
<!-- Build using MSBuild task -->
<MSBuild BuildInParallel="$(BuildSolutionsInParallel)"
Projects="$(Solution)"
Properties="Configuration=$(Configuration);Platform=$(Platform);$(OutDirOption);$(PublishDirOption);SkipInvalidConfigurations=$(SkipInvalidConfigurations);$(FxCopDirOption);$(ReferencePathOption);$(CodeAnalysisOption);
VCBuildOverride=$(VsPropsFile);VCBuildAdditionalLibPaths=$(VCBuildAdditionalLibPaths);VCBuildAdditionalOptions=$(VCBuildAdditionalOptions);VCBuildToolPath=$(VCBuildToolPath);VCBuildUseEnvironment=$(VCBuildUseEnvironment);
TeamBuildConstants=$(TeamBuildConstants);TargetsNotLogged=$(TargetsNotLogged);$(CustomPropertiesForBuild);$(CustomProperties)"
Targets="$(Targets)"
StopOnFirstFailure="$(StopOnFirstFailure)">
<Output TaskParameter="TargetOutputs" ItemName="CompilationOutputs" />
</MSBuild>
At first I thought this was telling me that one of the items that need to be substituted in the node was missing, but the other solutions which don't have issues have similar/identical setups.
I notice some of the items in the call stack of the MSBuild deal with targets and the issue seems to stem from the TeamFoundation.Build.targets file but I'm stuck as to what to do next to troubleshoot it.
Has anyone run into this before?

Answer 1

I believe the solution file is corrupt, and VS is being more tolerant. (If I remember correctly this crash can be caused by an unmatched curly parenthesis)
If it's feasible, I would recreate the solution file from within VS.
Dan (msbuild team)

Answer 2

When you said:
The actual solutions themselves and the projects they hold build correctly
do you mean that they build correctly within the IDE, or that you can go to those solution directories on the command line and run 'msbuild foo.proj' or 'msbuild bar.sln'?
Anyway, things I would try are first narrow down the offending solution/project, by telling TFS not to build in parallel, turn off multi-CPU support (e.g. '-maxcpucount:1'), do a binary search by trimming away solutions/projects. Once you get to the offending set, build with the '-v:diag' parameter to get even more details about the state of the MSBuild engine.
Alternately, you can write a small C# app that invokes the MSBuild engine directly and debug that application by catching first chance exceptions and having Reflector handy. See
sample code in MSBuild Engine Class.

Answer 3

Hmmm. You get a NullReferenceException, meaning that one of the parameters into that task is a null when it shouldn't be. Dumb question, but have you set up your TFSBuild.proj file with the solutions you want to build? IE. the stuff you put into the
- Itemgroup.
If you have, I would suggest adding message tasks to the Teambuild targets file just above the failing task to write out diagnostic information, like so:
That should spew some diagnostics info into your buildlog just above the log exerpt you pasted on your next build.

Answer 4

You may want to give the MSBuild-Sidekicks a try. They have a nifty little debugger for MSBuild included and offer a 14-day trial version: http://www.attrice.info/msbuild/