Is there any project/repo that contains "standard library" functions extracted from the Nixpkgs repo?
Currently reading the Nixpkgs manual, and it seems that the Nixpkgs repo consists of
functions that actually help build (cross-platform) packages (written using the Nix builtin's/primitives)
collection of packages
Tried to find decentralization examples, but there aren't many (or I didn't find them). I like Cachix and the philosophy behind it, but that is for already built packages (unless I misunderstand something, which is more than possible).
Some of the resources found:
Managing private Nix packages outside the Nixpkgs tree [Sander van der Burg]
I create a local repository for NixOS/nixpkgs? [Unix/Linux Stack Exchange]
EDIT: Just found out about Nix User Repository (NUR).
EDIT2: Here's the thread on NixOS discourse.
Nixpkgs is a "megarepo", bundling highly related Nix projects, such as Nixpkgs, its standard library, NixOS and such. I am not aware of any other source or 'mirror' of these projects. If you want this to change, you can help out with the Nix Flakes idea/project/projects.
Today, if you only really need pkgs.lib, you can use lib = import <nixpkgs/lib>.
Related
Very new to Nix and docker here. I am trying to get the License files (SPDX) for some Nix package files. I found a git repository for that and it requires the Nix derivations as input. What I have are default.nix, env.nix and shell.nix files. How do I extract the derivations from these files?
I don't really know how Nix works (although I tried reading a bit) and don't even need it other than this task. Any help would be appreciated. Right now I am trying the following steps.
Take a Nixos docker image and mount the Nix packages onto it.
Somehow extract the derivations from them. (this is where I need your help)
Copy the binary (of the Git link I mentioned above) from a rust debian image into this nix image. And run it on the Nix derivations.
I have made a ROS workspace and inside a package.
I did catkin_make and everything is working well.
I would like to give this package (or should I give the entire workspace?) to another person.
I am thinking to give him a zip file of the files and folders (it contains launch files, python scripts, rviz files etc) so I am expecting he will unzip it in his machine
I would like he can run the launch files without problems
What is what he needs to do for this? (of course he will have ROS installed, that is no problem)
I am thinking perhaps he should do source devel/setup.bash but is this enough?
When sharing a workspace with somebody only the source space src has to be shared. It should contain all our packages with their launch files (*.launch), Python (*.py) and C++ nodes (*.cpp, *.hpp), YAML configuration files (*.yaml), RViz configurations (*.rviz), robot descriptions (*.urdf, *.xacro) and describe how each node should be compiled in a CMakeLists.txt. Additionally you are supposed to keep track of all the Debian packages you install inside the package.xml file of each package.
If for some obscure reason there are things that I have to do that can't be accommodated in the standard installation instructions given above, I will actually write a bash script that performs these steps for me and add it either to the package itself or the workspace. This way I can automate also more complex steps such as installing OpenCV or modifying the .bashrc. Here a small example of what such a minimal script (I generally name them install_dependencies.sh) might look like:
#!/bin/bash
# Get current workspace
WS_DIR="$(dirname "$(dirname "$(readlink -fm "$0")")")"
# Check if script is run as root '$ sudo ...'
if ["$EUID" -ne 0]
then
echo "Error: This script has to be run as root '$ sudo ./install_dependencies.sh'
exit 1
fi
echo "Installing dependencies..."
# Modify .bashrc
echo "- Modifying '~/.bashrc'..."
echo "source ${WS_DIR}/devel/setup.bash" >> ~/.bashrc
echo ""
echo "Dependencies installed."
If for some reason even that is not possible I make always sure to document it properly either in a Markdown *.md read-me either in a /doc folder inside your package, in the read-me.md inside the base folder of your repository or inside the root folder of your workspace.
The receiver then only has to
Create a new workspace
Copy or clone the package files to its src folder
Install all the Debian package dependencies listed in the package.xml files with $ rosdep install
(If any: Execute the bash scripts I created by hand $ sudo ./install_dependencies.sh or perform the steps given in the documentation)
Build the workspace with $ catkin_make or $ catkin build from catkin-tools
Source the current environment variables with $ source devel/setup.bash
Make sure that the Python nodes are executable either by $ chmod +x <filename> or right-clicking the corresponding Python nodes (located in src or scripts of your package), selecting Properties/Permissions and enabling Allow executing file as program.
Run the desired Python or C++ nodes ($ rosrun <package_name> <executable_name>) and launch files ($ roslaunch <package_name> <launch_file_name>)
It is up to you to share the code as a compressed file, in form of a Git repository or a more advanced way (see below) but I will introduce some best practices in the following paragraphs that will pay off in the long run with larger packages.
Sharing a package or sharing a workspace?
One can either share a single package or an entire workspace. I personally think that most of the time one should share the entire workspace instead of the package alone even if you only cloned the other packages from a public Github repo. This might save the receiver a lot of headache e.g. when checking out the wrong branch.
Version control with Git
Arguably the best way to arrange your packages is by using Git. I'd actually make a repository for every package you create (if a couple of packages are virtually inseparable you can also bundled them to a single Git repo or better use metapackages). Then create an additional repository for your workspace and include your own packages and packages from other sources as submodules. This allows your code to be modular and re-usable: You can share only a package or the entire workspace!
As a first step I always add a .gitignore file to each package repository which excludes *.pyc files and another one to the workspace repository that ignores the build, devel and install folders.
You can add a particular repository as submodule to your workspace Git repository by opening a console inside the src folder of your workspace repository and typing
$ git submodule add -b <branch_name> <git_url_to_package> <optional_directory_rename>
Note that you can actually track a particular branch of a repository that you include as a submodule. In case you need a submodule at some point follow this guide.
If you share the workspace repository with someone they will have to have access to each individual submodule repository and they will have to not only pull the repository but also update the submodules with
$ git clone --recurse-submodules <git_url_to_workspace_repository>
and potentially update them to the latest commit with
$ git submodule update --remote
After these two steps they should have a full version of the repository with submodules and they should be able to progress with the steps listed in the section above.
1.1 Unit-testing and continuous integration
Before sharing a repository you will have to verify that everything is working correctly. This can take a decent amount of time, in particular if the code base is large and you are modifying it frequently. In the ideal case you would have to install it on a brand new machine or inside a virtual box in order to make sure that the set-up works which would take quite some time. This is where unit testing comes into play: For every class and function you program you will write a test. This way you can simply run these tests and make sure everything is working correctly. Generally these unit tests will be performed automatically and the working branches merged continuously. Generally the test routines are written with the libraries Boost::Test (C++), GoogleTest (generally used in ROS with C++), unittest (for Python) and QtTest (for GUIs). For ROS launch files there is additionally rostest. How this can be done in ROS is described here and here.
ROSjects
If you do not even want the person you are sending the code to to go through the hassle to set it up you might consider sending them a ROSject. A ROSject is an online virtual ROS environment (by the guys behind The Construct, the main source of ROS courses and of ROS tutorials on Youtube) that can be created and shared very easily from your existing Git repository as can be seen here. The simulation runs entirely in the cloud on a virtual machine. This way the potential of failure is very low but it is not a choice if your code is supposed to run on hardware and not only in simulation.
Docker
If your installation procedure is complex you might as well use a container such as a Docker.
More information about using Docker in combination with ROS can be found here. The Docker container might introduce though a bit of overhead and it is probably no choice for code which should have real-time priority in combination with a real-time patched operating system.
Debian or snap package
Another way of sending somebody a ROS package is by packing it into a Debian or snap package. This process takes a while and is in particular favourable if you want to give your code to a large number of users that should use the code out of the box. Instructions on how this can be done for Debian packages can be found here and here, while a guide for snap can be found here.
I implemented a ros package that depends on some other packages.
These packages depend on even more packages and so on...
How can I find out which packages are actually used when building and running nodes in my package?
(Except for looking at ALL the package.xml files manually, because there are multiple cases in which some packages are listed there but already deprecated and not actually used anymore)
So I'm looking for something like a tool/command/script that can list all actual package dependencies.
I think you can do this natively with rospack. To see everything a package depends on (including dependencies of dependencies) without duplicates, just do
rospack depends my_package
You can get it formatted with indents to see all dependency chains of each package (will include duplicates across chains if more than one package shares the same dependency):
rospack depends-indent my_package
And if you only wanted to know the immediate dependencies of your package, you can do:
rospack depends1 my_package
I'm not sure that addresses the problem you identify that it shouldn't identify deprecated dependencies, but if a package is still specifying a dependency explicitly in a package.xml, how is the system to know that isn't really a dependency? It'd be better to get those package.xml files up to date.
I want to create a ROS publisher node outside a catkin workspace.
Can it be created?
Of course you can. Treat ROS like any other cpp library or python package.
In python you have to keep PYTHONPATH environment variable pointing to ros packages in /opt/ros/kinetic/lib/python2.7/dist-packages.
In cpp you have to tell compiler where to look for includes (/opt/ros/kinetic/include), libraries (/opt/ros/kinetic/lib) and which library to import. For the simplest application -lroscpp -lrostime -lrosconsole should be sufficient. Ex:
g++ node.cpp -o node -I/opt/ros/kinetic/include -L/opt/ros/kinetic/lib -lroscpp -lrostime -lrosconsole
Remember that you still need ros environment variables like ROS_MASTER_URI.
However, I don't know if there is an easy way to generate custom ros messages without using catkin_make and cmake files.
Yes, in Python, you can write ROS nodes outside your catkin workspace.
Launch your roscore and then run the Python script in a new terminal normally like python filename and it runs as it would if you had placed in inside your catkin workspace and built and sourced it.
I have successfully created subscriber and publisher nodes and run them on an actual TurtleBot2 without the nodes being inside the catkin workspace.
I don't think the method I have described works for C++. It only works for Python. In C++ you will have to link the libraries while compiling. So, check that. For example, we do g++ filename.cpp -lm, where -lm links the math library to be used in the filename.cpp, so you might need to check how to do that. On the other hand, it is easier to just add the file in your catkin workspace or just shift to Python.
You have to include ros/ros.h and std_msgs/message_name.h where message_name is replaced by whatever message you're using in code. You might find these files somewhere or you can get the source code of the files online by searching.
I am just starting to get to grips with Nix, so apologies if I missed the answer to my question in the docs.
I want to use Nix to setup a secure production machine with the minimal set of libraries and executables. I don't want any compilers or other build tools present because these can be security risks.
When I install some packages, it seems that they depend on only the minimum set of runtime dependencies. For example if I install apache-tomcat-8.0.23 then I get a Java runtime (JRE) and the pre-built JAR files comprising Tomcat.
On the other hand, some packages seem to include a full build toolchain as dependencies. Taking another Java-based example, when I install spark-1.4.0 Nix pulls down the Java development kit (JDK) which includes a compiler, and it also pulls the Maven build tool etc.
So, my questions are as follows:
Do Nix packages make any distinction between build and runtime dependencies?
Why do some packages appear to depend on build tools whereas others only need runtime? Is this all down to how the package author wrapped up the application?
If a package contains build dependencies that I don't want, is there anything that I, as the operator, can do about it except design my own alternative packaging for the same application?
Many thanks.
The runtime dependencies are a subset of the build-time dependencies that Nix determines automatically by scanning the generated output for the hash part of each build-time dependencies' store path. For example, if you build a package using the compiler /nix/store/abcdef...-foo-1.20, then Nix will scan all files in the generated output for the hash bit abcdef.... If that hash is found, then the output is assumed to reference the compiler in some way, so it's kepts as a runtime dependency. If that hash does not occur, however, then the generated output has no reference to the compiler and therefore cannot access it at runtime, so foo-1.20 is treated as a build-time-only dependency.
Some packages record large parts of their build environment for informational/debugging purposes. Perl, for example, stores every little detail about the tools used to compile it, so all those store paths end up being treated as runtime dependencies despite the fact that Perl doesn't actually need them at runtime, but Nix can't know: it just knows that the Perl store path references those tools. Now, Nixpkgs maintainers usually make an effort to clean that up, i.e. by pruning the logfile that contains all those store paths from the installation, etc., but for sure there are plenty of packages in the database still that haven't been optimized to that end yet.
Let's assume that you'd like to compile a version of openssh that does not depend on PAM. Then you can remove the build input from the expression by means of an override, i.e. you replace the pam argument that's normally passed to the openssh build function with null. To do that, store the following file in ~/.nixpkgs/config.nix
{
packageOverrides = super: let self = super.pkgs; in {
openssh-without-pam = super.openssh.override {
pam = null;
};
};
}
and now install that package by running:
$ nix-env -f "<nixpkgs>" -iA openssh-without-pam