I am new in bazel and conan.
I try run conan in bazel use it https://docs.conan.io/en/1.44/integrations/build_system/bazel.html
In WORSPACE file:
load("//third-party/grpc:direct.bzl", "load_conan_dependencies")
In direct.bzl
load("#//conandeps:dependencies.bzl", "load_conan_dependencies")
conanfile.txt
[requires]
grpc/1.45.2
[generators]
cmake
When i try bazel sync
then has message:
ERROR: error loading package '': at /Users/a19583433/_WORK/videomix/third-party/grpc/direct.bzl:5:6: Label '//conandeps:dependencies.bzl' is invalid because 'conandeps' is not a package; perhaps you meant to put the colon here: '//:conandeps/dependencies.bzl'?
What am I doing wrong?
The doc is a little bit misleading - In short, conandeps is not a keyword, and you should replace conandeps to the subfolder that holds your conanfile.txt
e.g.,
you should have your load function like
load("#//subfolder1:dependencies.bzl", "load_conan_dependencies")
when you have the following folder structure
project
│ WORKSPACE
│ somefile.txt
│
└───subfolder1
│ conanfile.txt
│ somefile.txt
It took me quite a bit of time to figure out how to get Bazel and Conan to work together even with the accepted answer here, so I want to provide some additional information that is missing from here and from the documentation that would have helped me.
First off (because it's easy to confuse) what's being discussed here is consuming Conan packages with Bazel, not producing Conan packages.
dependencies.bzl is a generated file. This file gets generated when you run conan install and BazelDeps is listed as a generator in your conanfile (.txt or .py). You must run conan install to trigger BazelDeps to create this file.
After running conan install the dependencies.bzl will be generated in the current working directory along with directories for each of the dependencies listed in your conan file. Each of these will have a generated BUILD file. You should not directly reference these BUILD files. As it is shown in the documentation you simply add load("#//[your-path]:dependencies.bzl", "load_conan_dependencies") to your WORKSPACE and now you can use the libraries from your conan file.
Example of my generated dependencies.bzl:
def load_conan_dependencies():
native.new_local_repository(
name="gtest",
path="/conan-cache/.conan/data/gtest/1.12.1/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9",
build_file="/workspaces/test-project/generated/third-party/gtest/BUILD",
)
Example of my WORKSPACE:
load("//generated/third-party:dependencies.bzl", "load_conan_dependencies")
load_conan_dependencies()
Footnote:
Because we have multiple developers and we want to streamline this process, what we settled on is running conan install as part of a build script that then executes our regular build process. When you first run conan install it will pull the libraries and install them into the conan cache. After that, running the install is effectively a no-op. BazelDeps will overwrite all the generated files with the same contents, but because Bazel uses file content hashes to determine if inputs have changed, this does not result in having to rebuild anything. It just slows down the build slightly by re-running the generator.
Related
I've been trying to get a bazel monorepo with typescript to work. I have a couple of requirements in mind.
I should be able to import local packages using #myworkspace/ instead of ../../../ and so on, without needing Bazel. This is mostly so I get autocomplete while I'm writing.
The #myworkspace/ package should be the same during development and build time but only Bazel-managed dependencies should be resolved on imports when running sandboxed. Just so I know if I've messed up the name of the package in the js_library rule.
There should only be one lock file for the whole project. All dependencies should be located at root/node_modules.
It should be possible to individually deploy node packages i.e. #myworkspace/myCloudFunction.
It should be possible to include local dependencies in packages that will deployed.
I'm new to Bazel and it seems like it requires some mentality changes when coming from the NPM ecosystem. After googling, I've managed to find something that works for points 1 and 2 (But I might be wrong). I've published the playground repo at https://github.com/vitorelourenco/bazelmono-ts (pretty much a copy from https://github.com/lokshunhung/bazel-ts-monorepo with some ideas I took from https://github.com/angular/angular)
My questions about points 3 and 4:
Say I want the lib Lodash available on package #myworkspace/cloudFunction that will be deployed to Google Cloud Functions. If I install Lodash in the #myworkspace/cloudFunction folder, then Lodash will be added to package.json but I'll have a second node_modules folder and a second yarn.lock file, I don't want that. But if I install it in root/, then Lodash will not be added to the dependencies listed on package.json located at #myworkspace/cloudFunction, and when I deploy it, it won't install. Is there a smart way to handle this issue?
Point 5 is very similar. Ideally, the final Bazel output would have the local dependencies bundled in and ready to use but I can't seem to figure out a way to do it yet. I've tried adding a pkg_npm rule to //packages/app in the playgroup repo but couldn't get it to include //packages/common in it.
My repo has some gitignored paths that are sometimes populated during non-Bazel development and contain symlinks to other code paths that can sometimes cause the Bazel loader to too aggressively look for packages, which can cause bazel query to wrongly match packages through symlinks. Workspace root .bazelignore allows me to hide these directories but is there an option to tell Bazel to ignore a directory using a more local config rather than a single global file?
Specifically, I'm using Yarn Workspaces in Node.js. The source tree looks like the following:
BUILD
package.json
yarn.lock
myapp
├─ BUILD
└─ package.json
Running yarn install outside of Bazel creates a directory called node_modules, which includes third-party dependencies as well as a symlink farm to local packages:
BUILD
package.json
yarn.lock
node_modules
├─ myapp -> ../myapp
└─ some_3p_dep
myapp
├─ BUILD
└─ package.json
Expressing this pattern through Bazel is straightforward, and I've written a custom rule that just uses ctx.actions.declare_directory to create node_modules in the bindir (I have to tar/untar the symlink to conceal it from Bazel to prevent it from complaining, but the rule works great).
However, this results in a problem if a user runs the yarn install command locally without Bazel then runs something like bazel query //... | xargs bazel test to run all tests. In general bazel query //... appears to wrongly list not just targets in //myapp but also targets in //node_modules/myapp... after all node_modules/myapp/BUILD technically exists. This is problematic at the very least because it duplicates targets, but also in practice it causes ArtifactPrefixConflictException for any outputs that //node_modules/myapp declares because the paths overlap with the //:node_modules target/output. If I wanted to, I could technically work around the latter by messing with output names, but the duplication would still exist, wasting compute time and leading to different Bazel results depending on if the user is also doing a non-Bazel devloop.
.bazelignore appears to work fine to get Bazel not to traverse node_modules, but it only supports full paths (https://github.com/bazelbuild/bazel/issues/8106) and requires every new node_modules that may come into existence, which is cumbersome because it's a global file. Is there an option that we can declare the ignore more globally or another workaround I'm missing?
I have made a ROS workspace and inside a package.
I did catkin_make and everything is working well.
I would like to give this package (or should I give the entire workspace?) to another person.
I am thinking to give him a zip file of the files and folders (it contains launch files, python scripts, rviz files etc) so I am expecting he will unzip it in his machine
I would like he can run the launch files without problems
What is what he needs to do for this? (of course he will have ROS installed, that is no problem)
I am thinking perhaps he should do source devel/setup.bash but is this enough?
When sharing a workspace with somebody only the source space src has to be shared. It should contain all our packages with their launch files (*.launch), Python (*.py) and C++ nodes (*.cpp, *.hpp), YAML configuration files (*.yaml), RViz configurations (*.rviz), robot descriptions (*.urdf, *.xacro) and describe how each node should be compiled in a CMakeLists.txt. Additionally you are supposed to keep track of all the Debian packages you install inside the package.xml file of each package.
If for some obscure reason there are things that I have to do that can't be accommodated in the standard installation instructions given above, I will actually write a bash script that performs these steps for me and add it either to the package itself or the workspace. This way I can automate also more complex steps such as installing OpenCV or modifying the .bashrc. Here a small example of what such a minimal script (I generally name them install_dependencies.sh) might look like:
#!/bin/bash
# Get current workspace
WS_DIR="$(dirname "$(dirname "$(readlink -fm "$0")")")"
# Check if script is run as root '$ sudo ...'
if ["$EUID" -ne 0]
then
echo "Error: This script has to be run as root '$ sudo ./install_dependencies.sh'
exit 1
fi
echo "Installing dependencies..."
# Modify .bashrc
echo "- Modifying '~/.bashrc'..."
echo "source ${WS_DIR}/devel/setup.bash" >> ~/.bashrc
echo ""
echo "Dependencies installed."
If for some reason even that is not possible I make always sure to document it properly either in a Markdown *.md read-me either in a /doc folder inside your package, in the read-me.md inside the base folder of your repository or inside the root folder of your workspace.
The receiver then only has to
Create a new workspace
Copy or clone the package files to its src folder
Install all the Debian package dependencies listed in the package.xml files with $ rosdep install
(If any: Execute the bash scripts I created by hand $ sudo ./install_dependencies.sh or perform the steps given in the documentation)
Build the workspace with $ catkin_make or $ catkin build from catkin-tools
Source the current environment variables with $ source devel/setup.bash
Make sure that the Python nodes are executable either by $ chmod +x <filename> or right-clicking the corresponding Python nodes (located in src or scripts of your package), selecting Properties/Permissions and enabling Allow executing file as program.
Run the desired Python or C++ nodes ($ rosrun <package_name> <executable_name>) and launch files ($ roslaunch <package_name> <launch_file_name>)
It is up to you to share the code as a compressed file, in form of a Git repository or a more advanced way (see below) but I will introduce some best practices in the following paragraphs that will pay off in the long run with larger packages.
Sharing a package or sharing a workspace?
One can either share a single package or an entire workspace. I personally think that most of the time one should share the entire workspace instead of the package alone even if you only cloned the other packages from a public Github repo. This might save the receiver a lot of headache e.g. when checking out the wrong branch.
Version control with Git
Arguably the best way to arrange your packages is by using Git. I'd actually make a repository for every package you create (if a couple of packages are virtually inseparable you can also bundled them to a single Git repo or better use metapackages). Then create an additional repository for your workspace and include your own packages and packages from other sources as submodules. This allows your code to be modular and re-usable: You can share only a package or the entire workspace!
As a first step I always add a .gitignore file to each package repository which excludes *.pyc files and another one to the workspace repository that ignores the build, devel and install folders.
You can add a particular repository as submodule to your workspace Git repository by opening a console inside the src folder of your workspace repository and typing
$ git submodule add -b <branch_name> <git_url_to_package> <optional_directory_rename>
Note that you can actually track a particular branch of a repository that you include as a submodule. In case you need a submodule at some point follow this guide.
If you share the workspace repository with someone they will have to have access to each individual submodule repository and they will have to not only pull the repository but also update the submodules with
$ git clone --recurse-submodules <git_url_to_workspace_repository>
and potentially update them to the latest commit with
$ git submodule update --remote
After these two steps they should have a full version of the repository with submodules and they should be able to progress with the steps listed in the section above.
1.1 Unit-testing and continuous integration
Before sharing a repository you will have to verify that everything is working correctly. This can take a decent amount of time, in particular if the code base is large and you are modifying it frequently. In the ideal case you would have to install it on a brand new machine or inside a virtual box in order to make sure that the set-up works which would take quite some time. This is where unit testing comes into play: For every class and function you program you will write a test. This way you can simply run these tests and make sure everything is working correctly. Generally these unit tests will be performed automatically and the working branches merged continuously. Generally the test routines are written with the libraries Boost::Test (C++), GoogleTest (generally used in ROS with C++), unittest (for Python) and QtTest (for GUIs). For ROS launch files there is additionally rostest. How this can be done in ROS is described here and here.
ROSjects
If you do not even want the person you are sending the code to to go through the hassle to set it up you might consider sending them a ROSject. A ROSject is an online virtual ROS environment (by the guys behind The Construct, the main source of ROS courses and of ROS tutorials on Youtube) that can be created and shared very easily from your existing Git repository as can be seen here. The simulation runs entirely in the cloud on a virtual machine. This way the potential of failure is very low but it is not a choice if your code is supposed to run on hardware and not only in simulation.
Docker
If your installation procedure is complex you might as well use a container such as a Docker.
More information about using Docker in combination with ROS can be found here. The Docker container might introduce though a bit of overhead and it is probably no choice for code which should have real-time priority in combination with a real-time patched operating system.
Debian or snap package
Another way of sending somebody a ROS package is by packing it into a Debian or snap package. This process takes a while and is in particular favourable if you want to give your code to a large number of users that should use the code out of the box. Instructions on how this can be done for Debian packages can be found here and here, while a guide for snap can be found here.
For some reasons, practical or not, rxjs npm package stores BAZEL.build configuration in the package, so when I'm trying to build my project (which has node_modules folder) bazel tries automatically to build something that it's not supposed to build at all.
My question would be - what is canonical way of ignoring some specific folder while building bazel project recursively?
The only way to achieve what I'm looking for that I know of is to point to it explicitly in the command line
bazel build //... --deleted_packages=node_modules/rxjs/src (see user manual)
But I don't want to type this every time.
Bazel recently added a feature for ignoring folders (similar to gitignore).
Simply add node_modules to the .bazelignore file in the root of your project.
Yes, this is expressible as a bazel target pattern:
bazel build -- //... -//node_modules/rxjs/src/...
Full documentation is available at https://docs.bazel.build/versions/master/user-manual.html#target-patterns
I have rebar project with dependencies, so after clean when I run rebar compile, it downloads dependencies (for git runs git clone, looks like), runs configure for them and then compiles everything. Can I somehow make those dependencies local? I mean to skip downloading them and directly run configure there?
Try to use
rsync option and specify the file path
{rsync, "file:///foo/bar/baz"}
is the shape of it as long as I remember