If I have a monorepo using bazel, how can I get a list of targets that need to be rebuilt for the current commit, compared to the previous commit?
Ideally, there would be a way to work back from the changed sources file to the immediate and transitive targets that require them.
If I know which targets needed a rebuild, I will know which components I'll need to redeploy.
Unfortunately, there aren't any out-of-the-box solutions to this right now. See this discussion of a similar question:
https://groups.google.com/d/topic/bazel-discuss/I9udqWIcEdI/discussion
Related
I'm struggling to get Azure Devops Server 2019 (on prem) to build a complicated project setup of mine.
I have multiple solutions that build various BizTalk apps. Some of these solutions references some of the projects in other solutions/repos. This works fine in Visual Studio (providing everybody names their repos as the default, which they do).
To automate this build, I've created a multi-stage build pipeline that builds each solution in order.
Whilst solution 1 will build successfully, when it comes to solution 2, the msbuild tasks will not reference the outputs for the projects that have been included as a project reference in the solution.
This seems to be because these referenced projects aren't marked for build - because they wouldn't be able to build themselves without them in turn referencing their other projects in the main solutions etc.
Bundling everything into one big super solution file is not feasible.
I'm wanting msbuild to /reference the projects that have already been built in the previous step, as per being included as project references. But msbuild is a complicated beast and I cannot figure out a way of achieving this.
Is what I'm trying to achieve even possible? Can anybody point me in the right direction?
TFS / msbuild, building project references
This is a known issue about project reference in different Repos, but unfortunately it doesn't have a best answer. Because the best solution is always to have a single repository.
Git thinks of files as the content of the whole repository, not as a collection of files. Therefore this is quite hard to do. As workaround, you can consider to use Git Submodules or Git Subtree:
Check the details info from here:
Git and Visual Studio project references
Besides, the solution we are using now is to put the shared code in the NuGet package. Then, you can use the package from any repository, and you don't have to perform any unstable git settings and keep access control as they should be.
Hope this helps.
It seems that msbuild was changed to only reference project references, and that you can revert that to reference all references by setting the property OnlyReferenceAndBuildProjectsEnabledInSolutionConfiguration to true when running msbuild.
Source: https://stackoverflow.com/a/25144169
Have confirmed this works.
I've finally revisited this and done some more experimentation.
The only way, that I can see, to get TFS/msbuild to /reference other projects outputs is to mark them in the solution build configuration to build.
Now if you do this in Visual Studio and try and build a solution that has projects from another solution then the build will fail if the previous solution hasn't been fully built yet (as the other chained dependencies won't have been built). This makes sense.
But with TFS/msbuild, the build will succeed. From what I can tell there is some magic going on that ensures the dependencies across all solutions are somehow resolved. This might be luck, it might be specific behaviour (it's working for me so far).
The problem of course is that it's incompatible with Visual Studio. So I have an extra "Build" definition setup in each Solution file that has all the projects set to build.
This seems the easiest way to manage lots of interrelated solutions without having one big one.
originally, I use gradle to build my android project, but recently, I migrate it to bazel, and I find that bazel is truly fast than gradle, so I want to know why, but the doc of bazel doesn't give much idea about this, can anyone help me？
Thanks very much!
Full disclosure: I work on Bazel.
That's not an easy question to answer for two reasons. First, performance is highly dependent on the scenario. For example, we'd generally expect a clean build to be slower than a build where only a single file has changed. Second, I don't know how Gradle works internally, and they've done a lot of work recently to improve Gradle performance.
But I can talk about Bazel and what we're doing to make it fast. We've been working on build performance for ~10 years, starting long before we made it public.
The key feature is that we require all dependencies to be declared, and we track them explicitly. If you use a header file in C++, or depend on a Java library, you must declare this dependency in your BUILD file (and we enforce that these are declared by sandboxing individual actions). There are three effects from this:
First, we can heavily parallelize the build, because we know which things depend on which other things.
Second, we can make incremental builds very fast, because we can tell what parts of the build have to be re-done when you change a specific file (BUILD file, header file, source file, ...).
Third, we almost never have to do clean builds. Other build tools often require 'make clean' to get into a predictable state - since Bazel knows all the dependencies, it can get to a predictable state on every single build.
Another effect is that we can cache remotely (i.e., across users), and even execute on another machine, although neither of these are fully supported at the time of this writing.
I have a TFS 2010 with some projects and a common library used in 5 of them. We use VS 2013 and we have Rolling Builds enabled in most if not all build definitions. When the common library is checked in, all of the projects referencing it are recompiled - but the order is poor, the most commonly used project is compiled as the last one. Is there a way to change that so it gets compiled first?
This question hints at a lot of problems and possible solutions. The simplest answer is probably to just add more build servers to run all the builds in parallel.
Otherwise you need to consider turning off rolling builds and writing your own build scheduler. That or other strategies such as building the DLL once and checking it in as source to the other builds or running the build on a branch that is outside of the other 5 builds and merging the source into those builds when they want to pick up the change.
If you are otherwise happy with how things are now and don't want to do a lot of work to solve the problem.... then just scale out your build farm with additional agents.
There's a "Priority in queue" on the build definition, but it sounds like you might want to change your solution's > Project Dependencies' > Build Order?
Using TFS 2010 I need to build a solution that depends on a couple of other solutions held in different team projects. I'm editing my build definition and I create a list of "Projects to Build'. When it comes to build the parent solution I get an error because it's unable to reference assemblies created by the other solutions. So I go back to edit build definition and add /p:ReferencePath="c:\builds\3\referencedproject\binaries\" to the "MSBuild Arguments" Build process parameters.
Problem is, I don't want to hard code the c:\build\3. I guess there may be a $() property I can use in its place - can anyone please advise?
The other problem I have is that the 'Main' project and the two other projects that it references live at the same level in the source code. As far as I can tell, I have to set the source control folder to be the level above this - which happens to be the root. This means that TFS Build does a get of all the projects on the root - which includes dozens of projects that are not required for my build. It's not a critical issue since it makes no attempt to compile these non-related projects but it does increase the time for the build cycle to complete. Is this only way to avoid this to "group" the projects that are required for my build into a different TFS source folder?
For the references issue there are a few options. The most common one taking an explicit dependency on a specific version by checking in the binary to TFS.
For example, if you have Team Project A that has a dependency on Team Project B, I would assume they are setup as separate projects because they evolve differently, probably have different teams working on them, and have different release cycles. The common approach to managing this dependency is to checkin B.dll into Team Project A (usually in a lib folder specifically for this purpose), then use a file reference from within Project A's solution/projects to the dll in the lib folder.
This approach lets the Project A team explicitly choose which version of B.dll they wish to depend on, and make an explicit decision to adopt newer versions of B.dll on their own timetable.
For the other question of how to have a build definition download only select source code paths, you can specify multiple lines in the Workspace mapping screen when setting up a build definition. For example you could have the following:
$\ProjectA -> $(SourceDir)\ProjectA
$\ProjectB -> $(SourceDir)\ProjectB
This would download Project A + B but not C.
I'll start with problem 2 as this should be easiest to solve.
You have 2 options both involve changing the workspace mapping of your Build Definition.
You don't have to map at the folder "above", you can map individual folders so if your source looks like this.
$/TP/SolutionA
$/TP/Folder1/SolutionB
$/TP/Folder1/SolutionC
$/TP/Folder2/SolutionD
and you only want to include SolutionA and SolutionC in your build, you could set the workspace up as follows.
This will get just the code you need and preserve the relative paths between them.
Another option is to use cloaking, you map the "Root" folder and then cloak any folders you want the build to ignore.
Both of these methods will restrict the amount of source being downloaded when the build runs, and also prevent "continuous" builds from starting when checkins occur in the folders that haven't been mapped / Cloaked.
Problem number 1.
As Dylan suggests, probably the best thing to do is to use Binary References between solutions. Especially solutions in seperate team projects.
Check out my answer to this question for a full description.
We have a TFS 2008 project with two branches ("Main" and "NewFeature").
Each is a complete, independent "copy" (variant) of the source code.
By changing the workspace mappings, we can map either variant onto our local PCs and have been working with both branches with no problems.
However, if I set up the mappings to switch our build server on to the NewFeature branch (which should simply swap in the NewFeature source code without changing anything else as far as the build server is concerned) I get errors:
There is no working folder mapping for $/Main/Product.sln
i.e. when it is building from the NewFeature branch, something is still looking in the Main branch, even though there are no references anywhere in the source code to this branch. It appears to be caching some reference to Main?!
I have done a completely clean build (deleted the build folder from the server and run the build with /p:ForceGet=true to make sure the mapping is flushed through to the server, and there are no files on the server that might cache the workspace bindings), but this doesn't help.
Any suggestions?
Verify that:
$(SolutionToBuild) uses a relative path when referencing Product.sln
the relative path between $/NewFeature/.../TFSBuild.proj and $/NewFeature/Product.sln is the same as it is in the Main branch.
/ EDIT /
Note, however, it's not important that $/Main and $/Branches/Feature live at the same level in the tree hierarchy. Nor should the local path on the build server matter.* All that matters is what's underneath each branch. If the contents is internally consistent then all of your existing build scripts should work without modification.
For concrete examples of how I like to tie everything together, see my past answers, e.g.:
Modular TeamBuilds
SDLC Mangement for TFS Build Scripts
Where to put my database project in TFS?
How do you share external dependencies between Visual Studio solutions?
My way is not the only way, but I can attest that it works better than all the other variations I've encountered over the years :)
*Frankly, trying to micromanage Team Build can become a lot more painful than the proposed restructuring to your MSBuild scripts. For reliability you have to place your tfsbuildservice.exe.config customizations under version control somewhere...if you own >1 build server (or might in the future) then you have to consider a change deployment strategy...you end up needing a meta-SCM process to manage your SCM process!
I also had this problem when running a build from a branch in TFS 2010. TFS was reporting that "There is no working folder mapping for $/Main/Product.sln" The solution turned out to be to edit the build definition as follows (I am using the "Default Template" build process template—I have not tried this with a custom template):
Go to the Process section/tab of the build definition.
Expand 1. Required and look for Projects to Build. Make sure this entry is pointing to the solution file inside the branch you are building.
Expand 2. Basic and look for Automated Tests. Point this to the correct test settings file in the branch being built.
OK, the results are in - I've found a workaround.
Due to our legacy build processes (build, copy, obfuscate, build custom installers, copy to drop folder), I can't easily place the branch alongside the main branch. It needs to replace it.
So, if I have Main and NewFeature, I wish to unmap Main and map NewFeature in its place (i.e. use "c:\Main" on the build server, and simply change the source code that appears there)
Solution #1 (the most simple, obvious and logical solution) is to use these mappings:
$/NewFeature -> c:\Main
Expected result: NewFeature code structure simply replaces Main, and the build server doesn't know it's on a different branch.
Actual Result: Failure with a "you haven't mapped $/Main even though you're not using it" error.
Solution #2 is to do this:
$/Main -> c:\IgnoreThisFolder
$/NewFeature -> c:\Main
This works (it suppresses the warning and thus allows the build to proceed with MSBuild unaware that it is building in a branch). However, it's ugly and the build gets all the Main branch source code unnecessarily.
Solution #3 (untested, too expensive to try unless I know it'll work much better than #2) is:
Move all the source code (from $/Main, $/Branches/Feature) to $/Branches/Main and $/Branches/Feature to get a consistent hierarchy depth, and rewrite the MSBuild script to work with these new paths.
Hope that I can then map in only the branch I need and edit TFSBuild.proj to redirect it to build in that branch.
(Edit: Yes, this works well. We have now reorganised our entire code structure so that everything (all branches) is under a common root in a single Team Project, and branching/building is no longer a problem - it's easy to do whatever we need now. The trick is to insert a root folder into the hierarchy so that you can branch at any level you like. I've added a small tweak to the build script so that we can pass the branch to build as a parameter to MSBuild, so it's easy to build any variant now. Any branches we don't want to work on can just be cloaked and the build server remains happy.)
Summary
All these solutions (to use the technical term) suck. You have to remap the workspace (in this case, it's not simple: 9 mapping entries are required so it's an error prone and tedious thing to do), edit the TFSBuild.proj, delete all the source code, and run a build with /p:ForceGet=true to switch the build between branches. So it takes about an hour to switch branches. Unbelievable - it should take a few minutes at most!
I realise our project is far from ideally set up, but I can't believe it should be this difficult to branch in TFS (It was a piece of cake in SourceSafe, Accurev, and Perforce, so why so painful in TFS?).
How does everyone else organise their TFS branches? How do you switch developers between branches? How do you switch server builds between branches? Does it really have to be this painful?
When you Edit the build definition there are two places that need to be changed.
Source Settings - Point to your new project location
Process - (This sometimes takes a while to load so be patient) Under Required, change the "items to build" location to the new solution.
Hope this helps.
New update:
As reported in the other answer, I found a workaround that was ok for a short-lived feature branch, but it really didn't work very well. I've since come back to the problem, and the full solution is ridiculously simple:
In the TFSBuild.proj, the path was based on $(BuildProjectFolderPath). This path resolves to a server-side (source control path) like $/Main - not a local path (D:\ServerBuildFolder\Main).
Unfortunately, for historical reasons our source code is split across several team projects, which means the one "branch" is fragmented into several branched folders in Source Control (i.e. $/Main/Code and $/Libraries/Code. You can't create a branch that contains $/Main and $/Libraries). We thus have to reassemble these disparate fragments from Source Control back into a coherent hierarchy using workspace mappings.
This means that Richard was spot on - the relative path from the TFSBuild.proj file to the .sln file was incorrect, because MSBuild/TFS is assuming that the .sln lies within the same Team Project and source control hierarchy (so was looking for $/Main/Libraries.sln instead of $/Libraries/Libraries.sln).
The solution is simple: I replaced $(BuildProjectFolderPath) with a local path (e.g. D:\ServerBuildFolder\Main) for the files, so that the relative reference was resolved in "local space" (after the mappings had been applied), and MSBuild is now running sweetly.
The moral of the story:
1) NEVER use more than one Team Project if there is any chance that you will ever wish to have any kind of reference between those code-bases. Don't be fooled into thinking that a new Team Project will offer some kind of painless logical distinction between applications/libraries. Extra projects have proven to just be an administration nightmare - loads of extra problems for absolutely zero benefit. (It's all one big shared pile under the bonnet, so all the work items and source control folders are still visible in all the projects (!), but it adds a few brick walls that make inter-project links very problematic)
2) Always create a single root-level folder in your Team Project source control, and then put everything else underneath that folder. e.g. For the project "$/Main", create "$/Main/Root" and then put everything from your source hierarchy inside Root.
By following these rules, you will be able to branch the single 'Root' folder in future, and will then only need a single branch and a single extra workspace mapping. This will help you avoid premature baldness.
(In my defence, I would have done it this way to begin with - I'm working with a legacy setup. In defence of the legacy setup, it sounds good on paper but just isn't a Microsoft-supported approach!)
I got this error and all I can fathom is that the definition became corrupt or something. I just redid the process stuff (re-added the solution I was trying to build) and remapped the workspaces and it started working again. HTH.