How to build Nix packages from source? - nix

I think NixOS is great, but can't figure out how to build a package from source. Understanding the Nix expression language is not the problem, but to know what to put in a default.nix in order to build a package.
Take for example the Nix expression for gedit:
{ stdenv, intltool, fetchurl, enchant, isocodes
, pkgconfig, gtk3, glib
, bash, wrapGAppsHook, itstool, libsoup, libxml2
, gnome3, librsvg, gdk_pixbuf, file, gspell }:
stdenv.mkDerivation rec {
inherit (import ./src.nix fetchurl) name src;
propagatedUserEnvPkgs = [ gnome3.gnome_themes_standard ];
nativeBuildInputs = [ pkgconfig wrapGAppsHook ];
buildInputs = [ gtk3 glib intltool itstool enchant isocodes
gdk_pixbuf gnome3.defaultIconTheme librsvg libsoup
gnome3.libpeas gnome3.gtksourceview libxml2
gnome3.gsettings_desktop_schemas gnome3.dconf file gspell ];
enableParallelBuilding = true;
preFixup = ''
gappsWrapperArgs+=(--prefix LD_LIBRARY_PATH : "${stdenv.lib.makeLibraryPath [ gnome3.libpeas gnome3.gtksourceview ]}")
'';
meta = with stdenv.lib; {
homepage = https://wiki.gnome.org/Apps/Gedit;
description = "Official text editor of the GNOME desktop environment";
maintainers = gnome3.maintainers;
license = licenses.gpl2;
platforms = platforms.linux;
};
}
How can someone arrive to this solution? I must be missing something.

It is no surprise that such a package definition is a bit overwhelming to anyone. When you understand what the effect of various attributes, your example package becomes quite straight-forward.
In my experience, you arrive at such a package definition as follows. I'll assume you're not packaging something from a language-specific repository for now. Something you might build with make, for example.
Start with mkDerivation
Watch the build fail
Fix an error, by adding dependencies, extra commands, configure flags, patches, etc.
Repeat
So there you have it, it's trial and error. Experience with the required build tools helps a lot.
Some hints:
Look at similar packages, how they are defined
Read the Nixpkgs documentation
Scan through the support code for language ecosystem packages
https://nixos.wiki/wiki/Generic_Algorithm_on_Doing_Packaging

Related

How to get the library paths of a meson dependency?

Use case: I have a dependency that falls back to a subproject:
./
./subprojects/
./subprojects/mylib.wrap
src/meson.build contains:
mylib_dep = dependency('mylib') # Searches for mylib with pkg-config then fall backs to mylib.wrap.
myexec_exe = executable ('myexec', 'myexec.c', dependencies : mylib_dep)
Dependency mylib_dep provides libraries, which, if not installed on the system, make the main executable of my project unusable:
$ meson build && cd build && meson compile src/my_exec
...snip'd...
$ src/my_exec
src/my_exec: error while loading shared libraries: libmylib.so: cannot open shared object file: No such file or directory
My testing script build/tests/mytests.sh is configure_filed from tests/mytests.sh.in to indicate the location of myexec, and I'd like to pass to it the library paths, so that it can adjust LD_LIBRARY_PATH and run the executable. For instance, in tests/meson.build:
conf_data = configuration_data ()
conf_data.set_quoted ('MYEXEC_PATH', myexec_exe.full_path ())
conf_data.set_quoted ('MYLIB_PATH', mylib_dep.??????)
mytest_exe = configure_file (input : 'mytests.sh.in', output : 'mytests.sh', configuration : conf_data)
and in tests/mytests.sh.in:
MYEXEC_PATH=#MYEXEC_PATH#
MYLIB_PATH=#MYLIB_PATH#
export LD_LIBRARY_PATH=$(dirname "$MYLIB_PATH"):$LD_LIBRARY_PATH
$MYEXEC_PATH
Question: What should go at the ?????? above? In other words, given a dependency object, how can I extract the libraries within it, and get their full paths?
Usually in meson you wouldn't configure_file this, you'd pass the library/executable(s) to the script as arguments in the test command:
test(
'mytest',
find_program('mytest.sh')
args : [executable_target, library_target, ...],
)
It can be frustrating trying to get this sort of info out of Meson. Fortunately, if Meson used CMake to find the dependency, you may be able to get the library path from the underlying CMake variables, which are available in the Meson dependency object. Eg, something along the following lines worked for me:
mylib_dep = dependency('mylib')
if mylib_dep.found()
mylib_path = mylib_dep.get_variable(default_value : '', cmake : 'PACKAGE_LIBRARIES')
message('Library path is:', mylib_path)
endif

Flatpak Meson Not Finding Vala Libraries From Gnome Builder

From Linux, I'm using Meson (0.44.0) within Gnome Builder (3.26.4) for a console program that will use Gee and GXml. My intent is to write this in Genie.
When I use Meson within Gnome Builder it fails but the same succeeds when invoked from the command line using valac (0.38.8) as follows:
valac --pkg=gtk+-3.0 --pkg=gee-0.8 --pkg=gxml-0.16 main.gs
There is no error from the above. I've tried setting up meson.build with gee and gxml as dependency and alternatively as vala_args. Same error.
Checking pkg-config, I get the following:
$ pkg-config --libs gxml-0.16
-L/usr/local/lib64 -lgxml-0.16 -lgio-2.0 -lxml2 -lgee-0.8 -lgobject-2.0 -lglib-2.0
$ pkg-config --libs gee-0.8
-lgee-0.8 -lgobject-2.0 -lglib-2.0
$ pkg-config --libs gee-1.0
-lgee -lgobject-2.0 -lglib-2.0
Perhaps I'm doing something wrong. Here is the local meson.build file followed by the top level meson.build and the error:
example_sources = [
'main.gs'
]
example_deps = [
dependency('gio-2.0', version: '>= 2.50'),
dependency('gtk+-3.0', version: '>= 3.22'),
dependency('glib-2.0', version: '>= 2.50')
]
gnome = import('gnome')
example_sources += gnome.compile_resources(
'example-resources',
'example.gresource.xml',
c_name: 'example'
)
executable(
'example',
example_sources,
vala_args: '--target-glib=2.50 --pkg=gee-0.8 --pkg=gxml-0.16',
dependencies: example_deps,
install: true
)
with top-level meson.build:
project(
'example',
['c', 'vala'],
version: '0.1.0',
meson_version: '>= 0.40.0',
)
subdir('src')
And the error is:
uses Gee
error: The namespace name 'Gee' could not be found
I'm invoking the build from within Gnome-Builder. Can someone help me understand what is happening? I've tried to find why valac succeeds and meson fails in the documentation but cannot find a solution.
Gee and GXml should be dependencies, just like GIO, GLib and GTK+. So you should try:
example_deps = [
dependency('gio-2.0', version: '>= 2.50'),
dependency('gtk+-3.0', version: '>= 3.22'),
dependency('glib-2.0', version: '>= 2.50'),
dependency('gobject-2.0'),
dependency('gee-0.8'),
dependency('gxml-0.16'),
]
Usually you won't need to go beyond that. This makes the --pkg options in the vala_flags unnecessary. Meson does that for you. The way Meson works is it uses valac to produce C code then in a separate stage uses a C compiler to produce the binary. By using --pkg you are only telling valac which VAPI file to use, but not notifying the C compiler which pkg-config package to use for the C library.
Also notice I've added gobject-2.0 as a dependency. If I remember correctly GNOME Builder misses that and it does affect the build.
The error message, error: The namespace name 'Gee' could not be found, is troubling. This is an error from the Vala compiler and I would have thought that the compiler would be able to find the VAPI file using the vala_args method you've tried. Maybe you have Gee built from source and not installed system wide?
Meson does allow another VAPI search directory to be added:
add_project_arguments(['--vapidir',
join_paths(meson.current_source_dir(), 'vapi')
],
language: 'vala'
)
There are more details on the Vala page of the Meson Build documentation.
Genie support was added to Meson with version 0.42. So meson_version: should be >= 0.42.0.
If there are still problems then here is an MCVE using Genie, Gee and Meson. This should be compiled from the command line. Save the following Genie program as genie-gee.gs:
[indent=2]
uses Gee
init
var my_list = new ArrayList of string()
my_list.add( "one" )
my_list.add( "two" )
for item in my_list
print( item )
Then save the following Meson file as meson.build:
project('minimal-genie-gee-example',
'vala', 'c'
)
genie_gee_deps = [
dependency('glib-2.0'),
dependency('gobject-2.0'),
dependency('gee-0.8'),
]
executable('genie-gee',
'genie-gee.gs',
dependencies: genie_gee_deps
)
From the command line use Meson to set up the build directory:
meson setup builddir
This should show the dependencies have been found, for example:
Native dependency gee-0.8 found: YES 0.18.0
Then use Ninja build to build the project:
ninja -C builddir
For anyone using Fedora ninja is ninja-build.
Any problems with Meson setting up the build directory are logged to builddir/meson-logs/meson-log.txt.
If this works, but it fails in GNOME Builder, then my only other thought is that GNOME Builder has been installed using Flatpak. The sandboxed environment of Flatpak may be affecting the access to dependencies.
Update: Following the discussion in the comments it appears the runtime used by GNOME Builder was the problem. Builder has a great feature of being able to select the Flatpak runtime used to build your software. If you are following the 'traditional' way of developing by installing libraries and header files on your workstation then make sure Host Operating System is selected instead of a Flatpak runtime. It would appear the GNOME Flatpak runtime does not include libgee.
Update2: When writing a Flatpak builder manifest and a dependency is not in the Flatpak runtime/SDK then add the dependency as another module in the Flatpak builder manifest. This allows GNOME Builder to use Flatpak to build the software with the Flatpak runtime. An example manifest is given in AsymLabs answer.
Well after some exploration and AlThomas' advice above, here is what I discovered. OpenSUSE Tumbleweed provides four (or more) ways to install Gnome-Builder. These are:
1) Via Gnome Software Center. This installs org.gnome.Builder/stable in a sand boxed environment using Flatpak.
2) Via Flathub.org using Flatpak from the command line. This installs org.gnome.Builder/master (nightly) in a sand-boxed environment.
3) Via the package manager zypper and the command line. This installs a stable Gnome-Builder and related libraries system-wide.
4) Via Yast2. This provides the same as Zypper.
All three installations (same version 3.26.4 - different branches/tags - stable, master, nightly - two sand-boxed and one system wide) can be installed side by side and used as needed. During initial setup and testing, all variants yielded the same outcome - when using Gee and GXml only a Default build would work (the Flatpak Manifest would not build) but this has been resolved (it now appears that this is purely a Flatpak issue was a conflict between Flatpak and Fuse).
The Default build enables the Host runtime system. To set the Default build environment, upon opening a project within Gnome-Builder, choose Build Preferences from the upper left popover menu and select Default.
The drawback to a Default configuration is that it is not possible to Export Bundle, but local builds can utilize system-wide features.
So what is a Flatpak Manifest and why is it so important? It is the top level JSON file that contains project information. The Flatpak Manifest, in this case org.gnome.Example.json, pulls together all the features of the project so that it may be packaged for distribution. This includes the runtime, sdk, system connectivity to X11, IPC, Wayland, DBus, etc, the build system (Meson by default), cleanup directives, configuration and build options, submodule details (dependencies) and many other features. One Flatpak package can be installed in just about any Linux distribution, whether Debian, Ubuntu, Red Hat, OpenSuse or their derivatives, for example, and is sand-boxed for security and portability purposes. It will be, in future, fully cross-platform.
For instruction and testing, there are Flatpak Manifest examples to illustrate how they work. There are ways to alter the sand-box permissions using build finish directives. Flatpak documentation is excellent.
Within Gnome Builder when you first create a project, choose Vala + Gnome Application and a valid Flatpak Manifest will be installed. By default this is intended for a GUI rather than command line application; nonetheless it generates a default Flatpak Manifest that can be used as a template (Gnome Builder will allow multiple manifests - just select the build required). The following is the resulting improved Flatpak Manifest that will build submodules for both Gee and GXml (this has been tested within Gnome Builder and works):
{
"app-id": "org.gnome.Example",
"runtime": "org.gnome.Platform",
"runtime-version": "master",
"sdk": "org.gnome.Sdk",
"command": "example",
"finish-args": [
"--share=network",
"--share=ipc",
"--socket=x11",
"--socket=wayland",
"--filesystem=xdg-run/dconf",
"--filesystem=~/.config/dconf:ro",
"--talk-name=ca.desrt.dconf",
"--env=DCONF_USER_CONFIG_DIR=.config/dconf"
],
"build-options": {
"cflags": "-O2 -g",
"cxxflags": "-O2 -g",
"env": {
"V": "1"
}
},
"cleanup": [
"/bin",
"/include",
"/lib",
"/lib/pkgconfig",
"/share",
"/share/vala",
"*.la",
"*.a"
],
"modules": [
{
"name": "libgee",
"buildsystem": "meson",
"config-opts": [
"--libdir=lib"
],
"builddir": true,
"sources": [
{
"type": "git",
"tag": "meson",
"url": "https://github.com/GNOME/libgee.git"
}
]
},
{
"name": "libgxml",
"buildsystem": "meson",
"config-opts": [
"--libdir=lib"
],
"builddir": true,
"sources": [
{
"type": "git",
"branch": "master",
"url": "https://gitlab.gnome.org/GNOME/gxml.git"
}
]
},
{
"name": "example",
"buildsystem": "meson",
"config-opts": [
"--libdir=lib"
],
"builddir": true,
"sources": [
{
"type": "git",
"url": "file:///home/<user>/Projects/example"
}
]
}
]
}
Hat's off to the folks who are developing this package. Combining Flatpak, Meson, Gtk3/4/5/.., Vala, Genie (and soon the Vulkan 3D graphics engine) and beautifully minimalistic UI guidlines/standards in one lightweight development platform is something magical, akin to a modern day alchemy.
As an aside, I tried using Gtk3 with a number of languages, including C/C++, D, Haskell and Python but none of these alternatives could produce stand-alone binaries that were as compact, efficient and fun to write as Vala and Genie. These are greatly underrated languages.
Concluding, anyone who needs a good starting point when trying to understand these technologies and how Gnome-Builder is bringing them together can read AlThomas' post above and this one, along with the comments. It may save a lot of time.

qcircuit.sty not found when visualizing a QISKit quantum scheme

I followed this manual in order to visualize my QISKit v0.4.8 quantum circuit by means of the latex_drawer() function (in newer versions of QISKit the function is circuit_drawer()). However, I got the following error when running the code:
! LaTeX Error: File `qcircuit.sty' not found.
Is qcircuit.sty a part of QISKit? What is the best way to fix the issue?
By the way, this is the function that I use to produce the image of a circuit:
def circuit_image(circuit, basis="u1,u2,u3,cx"):
filename = 'circuit'
tmpdir = 'tmp'
if not os.path.exists(tmpdir):
os.makedirs(tmpdir)
filename_tex = filename + ".tex"
filename_pdf = filename + ".pdf"
latex_drawer(circuit, os.path.join(tmpdir, filename_tex), basis=basis)
os.system("pdflatex -output-directory {} {}".format(tmpdir, filename_tex))
images = pdf2image.convert_from_path(os.path.join(tmpdir, filename_pdf))
shutil.rmtree(tmpdir)
return images[0]
qcircuit is a third-party LaTeX package that is not part of QISKit. If your LaTeX distribution does not come with this package, you can install it by yourself.
For example, for Ubuntu 14.04.5 LTS do the following:
Download the zip-file
Unpack it to /usr/share/texlive/texmf-dist/tex/latex
Run sudo texhash
For more information on installing LaTeX packages, see the following pages:
Best way to install packages for TeXLive in Ubuntu?
How to install LaTeX packages for Tex Live?

Bazel cc_library dependency on other cc_library when each compile with a different crosstool

I have a code generator tool that generates C/C++ code. This code generator tool is compiled with crosstool1. The generated C/C++ code needs to be compiled with crosstool2.
So the actions are:
Using Crosstool1 compile 'code_generator'.
Execute 'code_generator' and generate 'generated_code.cpp'
Using Crosstool2 compile 'generated_code.cpp'
Is it possible to make a cc_library() determine the crosstool to use? I saw that Skylark rules now allow a 'toolchains' parameter which I'm not sure how this is used, also I do not want to do the heavy lifting of C/C++ compiling bare bone with Skylark.
Is there an example of using a proper Host Crosstool and Target Crosstool except for the Tenserflow example? I get a headache each time I read it :D
Assume //crosstool1:toolchain is a label for cc_toolchain_suite rule describing first crosstool, //crosstool2:toolchain is a label for cc_toolchain_suite for second crosstool, and the build file for the project is:
cc_binary(
name = "generator",
srcs = [ "main.cc" ],
)
genrule(
name = "generate",
outs = ["generated.cc"],
cmd = "$(location :generator) > $#",
tools = [":generator"],
)
cc_binary(
name = "generated",
srcs = [ "generated.cc" ],
)
Then running:
bazel build --host_crosstool_top=//crosstool1:toolchain --crosstool_top=//crosstool2:toolchain :generated
will do exactly what you describe, it will use crosstool1 to build :generator, and crosstool2 to build generated. Genrules use host configuration by default, so all should just work.

How to integrate LuaJIT with LuaRocks on Windows?

I downloaded the source of LuaJIT and compiled it with msvc120.dll (VS 2013 x64). When I run it from the command line I have no problems executing some basic lua. Now the LuaJIT installation guide mentions moving luajit.exe and lua51.dll into their own folder. From there it says to create a lua folder and under that a jit folder with the contents of src/jit moved underneath the newly created jit folder.
From my understanding my folder should look like and contain:
luajit.exe
lua51.dll
/lua
/jit
bc.lua
[rest of jit files]
vmdef.lua
Is this correct or am I missing files?
Now after I built my luajit I tried to wire it up into my luarocks to act as my interpreter using
install.bat /LUA C:\LuaJIT\2.0.3\[folder with above content]
However this cannot find the header files. I then copied over what are the header files into the folder above and that wires it up, but I can never actually get anything to compile when pointed over to LuaJIT. Edit: The error I get is the following,
C:\LuaJIT\2.0.3\bin\lua51.dll : fatal error LNK1107: invalid or corrupt file: cannot read at 0x2D0
Error: Failed installing dependency: https://rocks.moonscript.org/luafilesystem-1.6.2-2.src.rock - Build error: Failed compiling module lfs.dll
Is the correct way to handle this to simply point to my lua binaries and from there leverage LuaJIT to run my files or am I doing something wrong with wiring up LuaJIT and luarocks? The former seems to work for the most part, since I only ran into one library compilation issue, lua-cjson.
I've run on exactly the same problem, but they've found a solution right here:
https://github.com/keplerproject/luafilesystem/issues/22
I knew that for "linking DLLs statically" there is a so-called "export" .lib file, which is passed to the linker (and not the DLL itself).
So, for example, when compiling, LuaRocks was doing this:
cl /nologo /MD /O2 -c -Fosrc/mime.obj -ID:/LuaJIT-2.0.4/include/ src/mime.c -DLUA_COMPAT_APIINTCASTS -DLUASOCKET_DEBUG -DNDEBUG -DLUASOCKET_API=__declspec(dllexport) -DMIME_API=__declspec(dllexport) mime.c
link -dll -def:core.def -out:mime/core.dll D:/LuaJIT-2.0.4/bin/lua51.dll src/mime.obj
My LuaJIT was compiled from source in D:\LuaJIT-2.0.4\src, but I made two folders myself: D:\LuaJIT-2.0.4\include with all *.h files copied from src and D:\LuaJIT-2.0.4\bin with luajit.exe, lua51.dll, and then later lua51.exp and lua51.lib. Still same error, but this was the right track.
Fix
Now, check where your LuaRocks configs are:
luarocks.bat help
Scroll down to a section like:
CONFIGURATION
Lua version: 5.1
Configuration files:
System: D:/luarocks/config-5.1.lua (ok)
User : (... snip ...)
Edit the System configuration file, specifically see the part:
variables = {
MSVCRT = 'VCRUNTIME140',
LUALIB = 'lua51.dll'
}
Here! LUALIB should be the .lib file. If your export lib is alongside the DLL, you just need to change to:
variables = {
MSVCRT = 'VCRUNTIME140',
LUALIB = 'lua51.lib' -- here!
}
Verification
And now:
luarocks.bat install luasocket
(...)
link -dll -def:core.def -out:socket/core.dll D:/LuaJIT-2.0.4/bin/lua51.lib src/luasocket.obj (...)
(...)
luasocket 3.0rc1-2 is now built and installed in D:\luarocks\systree (license: MIT)
Note the first argument passed to the linker.

Resources