Is it possible to keep my Nix packages in sync across machines not running NixOS? - nix

I know with NixOS, you can simply copy over the configuration.nix file to sync your OS state including installed packages between machines.
Is it possible then, to do the same using Nix the package manager on a non-NixOS OS to sync only the installed packages?

Please note, that at least since 30.03.2017 (corresponding to 17.03 Nix/NixOS channel/release), as far as I understand the official, modern, supported and suggested solution is to use the so called overlays.
See the chapter titled "Overlays" in the nixpkgs manual for a nice guide on how to use the new approach.
As a short summary: you can put any number of files with .nix extension in $HOME/.config/nixpkgs/overlays/ directory. They will be processed in alphabetical order, and each one can modify the set of available Nix packages. Each of the files must be written as in the following pattern:
self: super:
{
boost = super.boost.override {
python = self.python3;
};
rr = super.callPackage ./pkgs/rr {
stdenv = self.stdenv_32bit;
};
}
The super set corresponds to the "old" set of packages (before the overlay was applied). If you want to refer to the old version of a package (as in boost above), or callPackage, you should reference it via super.
The self set corresponds to the eventual, "future" set of packages, representing the final result after all overlays are applied. (Note: don't be scared when sometimes using them might get rejected by Nix, as it would result in infinite recursion. Probably you should rather just use super in those cases instead.)
Note: with the above changes, the solution I mention below in the original answer seems "deprecated" now — I believe it should still work as of April 2017, but I have no idea for how long. It appears marked as "obsolete" in the nixpkgs repository.
Old answer, before 17.03:
Assuming you want to synchronize apps per-user (as non-NixOS Nix keeps apps visible on per-user basis, not system-wide, as far as I know), it is possible to do it declaratively. It's just not well advertised in the manual — though it seems quite popular among long-time Nixers!
You must create a text file at: $HOME/.nixpkgs/config.nix — e.g.:
$ mkdir -p ~/.nixpkgs
$ $EDITOR ~/.nixpkgs/config.nix
then enter the following contents:
{
packageOverrides = defaultPkgs: with defaultPkgs; {
home = with pkgs; buildEnv {
name = "home";
paths = [
nethack mc pstree #...your favourite pkgs here...
];
};
};
}
Then you should be able to install all listed packages with:
$ nix-env -i home
or:
$ nix-env -iA nixos.home # *much* faster than above
In paths you can put stuff in a similar way like in /etc/nixos/configuration.nix on NixOS. Also, home is actually a "fake package" here. You can add more custom package definitions beside it, and then include them your "paths".
(Side note: I'm hoping to write a blog post with what I learned on how exactly this works, and also showing how to extend it with more customizations. I'll try to remember to link it here if I succeed.)

Related

How to issue Message Before Build--or seq problems

I'm trying to add helpful messages for arbitrary builds. If the build fails the user can, for example, install the package with different arguments.
My interface idea is to provide a function, build-with-message, that would be called with something like this:
build-with-message
''Building ${pkg.name}. Alternative invocations are: ..''
pkg
My implementation is based on builtins.seq
build-with-message = msg : pkg :
seq
(self.runCommand "issue-message" {} ''mkdir $out; echo ${msg}'')
pkg;
When I build a package with build-with-message I never see the message. My hunch is that seq evaluates the runCommand far enough to see that a set is returned and moves on to building the package. I tried with deepSeq as well, but a deepSeq build fails on runCommand. I also tried calling out some attributes from the runCommand, e.g.
(self.runCommand "issue-message" {} ''mkdir $out; echo ${msg}'').drvPath
(self.runCommand "issue-message" {} ''mkdir $out; echo ${msg}'').out
My thought being that calling for one of these would prompt the rest of the build. Perhaps I'm not calling the right attribute, but in any case the ones I've tried don't work.
So:
Is there a way to force the runCommand to build in the above scenario?
Is there already some builtin that just lets me issue messages on top of arbitrary builds?
Here's me answering my own question again, consider this a warning.
Solution:
I've in-lined some numbered comments to help with the explanation.
build-with-message = msg : pkg :
let runMsg /*1*/ = self.runCommand "issue-message"
{ version = toString currentTime; /*2*/ } ''
cat <<EOF
${msg}
EOF
echo 0 > $out /*3*/
'';
in seq (import runMsg /*4*/) pkg; /*5*/
Explanation:
runMsg is the derivation that issues the message.
Adding a version based on the current time ensures that the build of runMsg will not be in /nix/store. Otherwise, each unique message will only be issued for the first build.
After the message is printed, a 0 is saved to file as the output of the derivation.
The import loads runMsg--a derivation, and therefore serialized as the path $out. Import expects a nix expression, which in this case is just the number 0 (a valid nix expression).
Now, since the runMsg output will not be available until after it has been built, the seq command will build it (issuing the message) and then build pkg.
Discussion:
I take note of Robert Hensing's comment to my question--this may not be something Nix was not intended for. I'm not arguing against that. Moving on.
Notice that issuing a message like so will add a file to your nix store for every message issued. I don't know if the message build will be garbage collected while pkg is still installed, so there's the possibility of polluting the nix store if such a pattern is overused.
I also think it's really interesting that the result of the runMsg build was to install a nix expression. I suppose this opens the door to doing useful things.

Using TensorFlow Audio Recognition Model on iOS

I'm trying to use the TensorFlow audio recognition model (my_frozen_graph.pb, generated here: https://www.tensorflow.org/tutorials/audio_recognition) on iOS.
But the iOS code NSString* network_path = FilePathForResourceName(#"my_frozen_graph", #"pb"); in the TensorFlow Mobile's tf_simple_example project outputs this error message: Could not create TensorFlow Graph: Not found: Op type not registered 'DecodeWav'.
Anyone knows how I can fix this? Thanks!
I believe you are using the pre-build Tensorflow from Cocapods? It probably does not have that op type, so you should build it yourself from latest source.
From documentation:
While Cocapods is the quickest and easiest way of getting started, you
sometimes need more flexibility to determine which parts of TensorFlow
your app should be shipped with. For such cases, you can build the iOS
libraries from the sources. This guide contains detailed instructions
on how to do that.
This might also be helpful: [iOS] Add optional Selective Registration of Ops #14421
Optimization
The build_all_ios.sh script can take optional
command-line arguments to selectively register only for the operators
used in your graph.
tensorflow/contrib/makefile/build_all_ios.sh -a arm64 -g $HOME/graphs/inception/tensorflow_inception_graph.pb
Please note this
is an aggresive optimization of the operators and the resulting
library may not work with other graphs but will reduce the size of the
final library.
After the build is done you can check /tensorflow/tensorflow/core/framework/ops_to_register.h for operations that were registered. (autogenerated during build with -g flag)
Some progress: having realized the unregistered DecodeWav error is similar to the old familiar DecodeJpeg issue (#2883), I ran strip_unused on the pb as follows:
bazel-bin/tensorflow/python/tools/strip_unused \
--input_graph=/tf_files/speech_commands_graph.pb \
--output_graph=/tf_files/stripped_speech_commands_graph.pb \
--input_node_names=wav_data,decoded_sample_data \
--output_node_names=labels_softmax \
--input_binary=true
It does get rid of the DecodeWav op in the resulting graph. But running the new stripped graph on iOS now gives me an Op type not registered 'AudioSpectrogram' error.
Also there's no object file audio*.o generated after build_all_ios.sh is done, although AudioSpectrogramOp is specified in tensorflow/core/framework/ops_to_register.h:
Jeffs-MacBook-Pro:tensorflow-1.4.0 zero2one$ find . -name decode*.o
./tensorflow/contrib/makefile/gen/obj/ios_ARM64/tensorflow/core/kernels/decode_bmp_op.o
./tensorflow/contrib/makefile/gen/obj/ios_ARM64/tensorflow/core/kernels/decode_wav_op.o
./tensorflow/contrib/makefile/gen/obj/ios_ARMV7/tensorflow/core/kernels/decode_bmp_op.o
./tensorflow/contrib/makefile/gen/obj/ios_ARMV7/tensorflow/core/kernels/decode_wav_op.o
./tensorflow/contrib/makefile/gen/obj/ios_ARMV7S/tensorflow/core/kernels/decode_bmp_op.o
./tensorflow/contrib/makefile/gen/obj/ios_ARMV7S/tensorflow/core/kernels/decode_wav_op.o
./tensorflow/contrib/makefile/gen/obj/ios_I386/tensorflow/core/kernels/decode_bmp_op.o
./tensorflow/contrib/makefile/gen/obj/ios_I386/tensorflow/core/kernels/decode_wav_op.o
./tensorflow/contrib/makefile/gen/obj/ios_X86_64/tensorflow/core/kernels/decode_bmp_op.o
./tensorflow/contrib/makefile/gen/obj/ios_X86_64/tensorflow/core/kernels/decode_wav_op.o
Jeffs-MacBook-Pro:tensorflow-1.4.0 zero2one$ find . -name audio*_op.o
Jeffs-MacBook-Pro:tensorflow-1.4.0 zero2one$
Just verified that Pete's fix (https://github.com/tensorflow/tensorflow/issues/15921) is good:
add this line tensorflow/core/ops/audio_ops.cc to the file tensorflow/contrib/makefile/tf_op_files.txt and run tensorflow/contrib/makefile/build_all_ios.sh again (compile_ios_tensorflow.sh "-O3" itself used to work for me after adding a line to the tf_op_files.txt, but not anymore with TF 1.4).
Also, use the original model file, don't use the stripped version. Some note was added in the link above.

Make shell aliases declaratively depend on packages

In NixOS, definitions of shell aliases can be defined inside the configurations.nix file like so:
environment.shellAliases = {
"my_some_cmd" = "some_cmd -flag 123";
}
This gets assigned even when the referred command (here: some_cmd) is not available in the system. Say, this command is included in a package. So it would be desirable to declare that the alias should only be assigned if the package is installed.
How could that be done? Would I have to just work with an wrapping if-statement or are there other ways to archive this?
If the if statement is the way to go, how could that be implemented?
You can bypass the need to install the package by using the full path to the command, example:
environment.shellAliases = {
"colored-tree" = "${pkgs.tree}/bin/tree -C";
};

result from /proc/self/exe is unfriendly in a clearcase view

If I execute a binary in a clearcase view, and look at /proc/self/exe for that on Linux, I see something like the following:
$ cd /proc/19220
$ ls -l exe
lrwxrwxrwx 1 peeterj pdxdb2 0 2012-11-30 13:04 exe -> /home/peeterj/views/peeterj_clang-7.vws/.s/00024/8000028250b8f1d1llvm-config
The clang llvm-config program, not unreasonably, uses this output to try to figure out the absolute fully qualified path that it is located in (I assume in case argv[0] isn't fully qualified).
Is there a way to find the location within the view that this corresponds to. For example, in this case, the llvm-config exe is actually in:
/vbs/bldsupp/linuxamd64/clang/debug/bin
(I'm wondering if it's feasible to modify clang's GetExecutablePath() function to figure this out.)
No trivial solution here (for an old version of ClearCase though):
The technote "PK27447: WITHIN A CLEARCASE DYNAMIC VIEW, THE READLINK() CALL ON LINUX RETURNS THE WRONG PATH FOR THE EXECUTABLE'S /PROC/SELF/EXE VALUE" suggests:
Local fix
Use getcwd(), get_current_dir_name(), getwd() in applications slated for a VOB/View context
Create an interposer library to intercept the readlink() call, and modify to use any of the above calls to return the proper data
The cause:
/proc/self/exe returns the improper path while getcwd succeeds.
Unfortunately, for /proc/self/exe to return the proper value [from within a VOB/View context] would require a change within the Linux kernel to allow MVFS to "override" the present setting.
IBM LTC has been working on having the Linux community adopt this change so that we can then incorporate the new features within MVFS.
Related: Bug Sun 6189256.

Error "Invalid Locales set !!" when trying to install sqldemo on Informix

I am extremely new to Informix and am having some trouble trying to get sqldemo installed.
Set up so far:
openSuse 12.1 (32 bit)
Informix Growth Edition 11.70 UC6
Informix SQL Developer 7.50 UC6
Informix RDS 7.50 UC6
Informix ID 7.50 UC6
After struggling a few days and a lot of reading of http://publib.boulder.ibm.com/infocenter/idshelp/v117/index.jsp, I managed to get Informix installed and On-line.
I also opted to install the demo database instance that comes with the installation.
I now and attempting to get started with Informix 4GL by Example.
I am trying to get the sqldemo database up. I don't know if it will replace the previous instance installed with Informix, but that is a different problem.
Right now as per the document, running the following should set up the DB:
sqldemo stores2t -log
I however get an error: "Invalid Locales set !!".
I have tried looking up this error and also in the documentation.
I have tried setting the CLIENT_LOCALE and DB_LOCALE in my .profile file.
For example:
export CLIENT_LOCALE=en_US.CP1252 and
export DB_LOCALE=en_US.819
This has not helped.
A push in the right direction, or perhaps some other documentation I could read that would explain things better would really be appreciated.
If any other information is required from me, please do not hesitate to ask.
Update 1
Thanks so much for the response.
A couple of things firstly that I have tried since your post.
Changed the the CLIENT_LOCALE and DB_LOCALE as you specified - Same error - So i removed it as you said it should not be set.
Fixed a problem in my PATH and made sure it has /usr/informix/bin - Same Error
INFORMIXDIR is /usr/informix
INFORMIXSERVER is ol_informix1170 (This is from the database that was installed with the informix install, don't know if this must be changed? and if so to what?)
Ran the script you mentioned, result :
INFORMIXDIR=/usr/informix
INFORMIXSERVER=ol_informix1170
INFORMIXSQLHOSTS=/usr/informix/etc/sqlhosts
LANG=en_US.UTF-8
ONCONFIG=onconfig
I noticed I had set the language to UK, which made the Locales en_gb instead if en_us, so tried changing that in my .profile, which did not help, so also tried changing the language to US and the locales to en_us, but this made no difference.
As for what you said about the sqldemo script and the already installed db, It is fine if that db is removed as this is just a test VB box for me to learn on.
Could the $INFORMIXSERVER set as ol_informix1170 be the problem?
Thank you once again for the help.
Neill
Update 2
Thanks again for the response.
A few things to note.
The dbenv results I posted is all that shows which i assume/presume (uh-oh) means that the other environment variables are not set. Which of the environment variables you posted are absolutely necessary for it to work?
As above, Where would I find the terminfo file, or does this need to be created?
As above, the SQLEXEC variable... where would I find sqlrm? I can somewhat remember from the documents I have read I think it should be $INFORMIXDIR/lib? but I only have an esql directory. Is this correct.
Barring that something in the first 3 above is not causing more problems, when trying your suggestion of DEMOPATH=en_us/0333 sqldemo stores2t -log I receive the following error:
Sorry, cannot read the mkstores3 program required to build the demonstration database. Check the /etc subdirectory of INFORMIXDIR (/usr/informix).
Checking /usr/informix/etc shows indeed that there is no mkstores3 file.
Attempting your further note of isqldemo, I get the following error:
/usr/informix/bin/isqldemo: line 58: /usr/informix/demo/sql/en_us/e01c/isqldemo: No such file or directory.
I guess this makes perfect sense as there is no e01c directory, just the 0333 directory.
Right now anything you can tell me would indeed be a consolation because my newb-ness to generally Linux and definately Informix is a big factor. Interesting that this bug has been around for so long. I guess way more experienced folk than I figured out how solve it on their own, or just never bothered with the sqldemo.
I guess that will teach me to read this:
INFORMIX-4GL by Example
Version 4.1
July 1991
Going to check now if any updated text exists, but would still appreciated more help in solving this problem. Do you think reverting to a previous snapshot before Informix was installed and not opting for the ol_informix1170 database to be included could be a possible solution? I wouldn't really see that it would be, but what do I know.
Many many thanks for your continued time and effort.
Regards,
Neill
Update 3
So I see indeed the document I was reading is ancient. I have found an updated one (2002) which uses a different script (dbaccessdemo7).
I tried running that, have run into an error, but tomorrow is another day.
For now I am going to mark this as solved because of the bug detected and resolved. I am not going to put more time and effort into sqldemo.
Thank you so much, and if I struggle with dbaccessdemo 7, I will post a new question.
Regards,
Neill
The sqldemo script won't create a new server; it may clobber your existing database (a single server may house multiple databases; indeed, there are 4 sys* databases created when a server is initialized) but it won't harm your server otherwise.
Probable cause of the error
The normal problem with invalid locales is that you've not set $INFORMIXDIR. You need $INFORMIXDIR set unless /usr/informix is (a symlink to) the correct location. You also need $INFORMIXSERVER set, and you usually need $INFORMIXDIR/bin on $PATH. Strictly, $INFORMIXSERVER is the only mandatory variable; in practice, you worry about the other two too.
The $INFORMIXDIR setting is used to locate the locale information (which is found in $INFORMIXDIR/gls) and the message files (which are found in $INFORMIXDIR/msg).
Note that CP1252 is a Windows code page. Normally on Unix, you'd either not set CLIENT_LOCALE or DB_LOCALE, or you could set them to:
export CLIENT_LOCALE=en_us.8859-1
export DB_LOCALE=en_us.8859-1
or you can choose another more appropriate (to you) locale. The 8859-15 locale includes the Euro symbol, for example, or the utf-8 locale dictates UTF-8 in the database. But, for initial debugging, stick with the 8859-1 locale, aka 819 or 0333 (all based on the IBM CCSID). If it doesn't work with 8859-1, then we have one set of problems; if it works with 8859-1 but not some other codeset or locale, then we have a different set of problems.
Follow-up info if the solution above fails
If that isn't the trouble, then I'll ask for some more details — notably, your Informix environment as reported by the dbenv script below:
: "#(#)$Id: dbenv.sh,v 2.11 2007/09/02 00:18:58 jleffler Exp $"
#
# Printout INFORMIX database environment
informix1="DB[^=]|DELIMIDENT=|SQL|ONCONFIG|TBCONFIG|INFOR"
informix2="ARC_|CLIENT_LOCALE=|GL_|GLS8BITSYS|CC8BITLEVEL|ESQL|FET_BUF_SIZE="
informix3="INF_ROLE_SEP=|NODEFDAC=|ONCONFIG|OPTCOMPIND|PDQ|PSORT"
informix4="PLCONFIG|SERVER_LOCALE|FGL|C4GL|NE_"
informix5="TCL_LIBRARY|TK_LIBRARY"
informix="$informix1|$informix2|$informix3|$informix4|$informix5"
system="COLLCHAR=|LANG=|LC_|LD_LIBRARY_PATH(_64)?=|PATH=|SHLIB_PATH="
jlss="IXD(32|64)?="
env |
egrep "^($informix|$system|$jlss)" |
sort
It's an old script; that's why the shebang is missing.
Second set of diagnosis
I was hoping for the complete output of the dbenv script; it is surprising how often something shows up. However, given what you've said, it is likely to be OK.
The INFORMIXSERVER setting sounds fine.
I'm struck by the LANG=en_US.UTF-8 setting; Informix does pay attention to $LANG and the $LC_* variables (that's why dbenv prints those out). That may be a factor in the problem. However, I would have expected CLIENT_LOCALE and SERVER_LOCALE to deal with that if it was the problem. Also, on my Mac, I have LANG=en_US.UTF-8 and yet I can connect to (8859-1) databases OK.
This is beginning to look like an install problem...or sqldemo problem...
I transitioned from a Mac to a RHEL 5 (archaic) x86/64 machine, and tried running sqldemo over there:
$ dbenv
DBDATE=Y4MD-
DBEDIT=vim
INFORMIXDIR=/work4/informix/tools-7.50.FC4
INFORMIXSERVER=toru_31
INFORMIXSQLHOSTS=/work4/informix/ids-11.70.FC4/etc/sqlhosts
INFORMIXTERM=terminfo
IXD64=/work4/informix/ids-11.70.FC4
IXD=/work4/informix/tools-7.50.FC4
IXH=/work4/informix/ids-11.70.FC4/etc/sqlhosts
IXO=/work4/informix/ids-11.70.FC4/etc/onconfig.toru_31
IXS=toru_31
LANG=en_US.UTF-8
LD_LIBRARY_PATH=/lib64:/usr/lib64:/work4/informix/tools-7.50.FC4/lib:/work4/informix/tools-7.50.FC4/lib/tools:/work4/informix/tools-7.50.FC4/lib/esql:/work4/informix/ids-11.70.FC4/lib:/work4/informix/ids-11.70.FC4/lib/esql:/work4/informix/ids-11.70.FC4/lib/cli
ONCONFIG=onconfig.toru_31
PATH=/work4/informix/tools-7.50.FC4/bin:.:/work4/jleffler/bin:/u/jleffler/bin:/work4/informix/ids-11.70.FC4/bin:/u/jleffler/linux/x86_64/bin:/work4/informix/11.70.FC1:/usr/atria/bin:/work4/jleffler/perl/v5.12.1/bin:/usr/bin:/bin:/usr/X11R6/bin:/atria_release/cm_dist/vobs/imitools/bin:/opt/rational/clearcase/bin:/opt/rational/clearquest/bin
SQLCMDLOG=/work4/jleffler/.sqlcmdlog
SQLEXEC=sqlrm
TERMINFO=/work4/jleffler/terminfo
TERM=xterm-color
$ sqldemo st2 -log
Invalid Locales set !!
$
Oh yeah? No; my locales are fine, thank you!
Well, so be it...I can reproduce your problem! That's step 1. Step 2 is to look at the expletive deleted script.
PRODUCT=sql
DEMOFILE=sqldemo
DEFLANG=en_US.8859-1
INFORMIXDIR=${INFORMIXDIR:=/usr/informix}
INFENV=$INFORMIXDIR/bin/infenv
CONVLOC=$INFORMIXDIR/bin/convloc
if [ $# -gt 0 -a "X$1" = "X-e" ] ; then
LOCALE=$DEFLANG # -e means en_US.8859-1 required
shift
else
LOCALE=`$INFENV DBLANG` # get DBLANG value
if [ "x${LOCALE}" = "x" ]; then
LOCALE=`$INFENV CLIENT_LOCALE` # try CLIENT_LOCALE instead
if [ "x${LOCALE}" = "x" ]; then
LOCALE=`$INFENV DB_LOCALE` # finally default to DB_LOCALE
fi
fi
fi
if [ "x${LOCALE}" = "x" ]; then
LOCALE=$DEFLANG # finally default to DB_LOCALE
fi
export LOCALE
if [ "x${DEMOPATH}" = "x" ]; then
echo "Invalid Locales set !!"
else
exec $INFORMIXDIR/demo/$PRODUCT/$DEMOPATH/$DEMOFILE $*
fi
exit $?
Note that test for ${DEMOPATH}; note that DEMOPATH is not set in the script. So, we've got to get it set. What to? Well, ls $INFORMIXDIR/demo/sql shows that there are various locale-specific sub-directories (en_us,
ja_jp,
ko_kr,
th_th,
zh_cn,
zh_tw) and under the en_us directory there's 0333 (only).
Please run:
DEMOPATH=en_us/0333 sqldemo stores2t -log
This more or less worked for me — I believe it would work for you. I have a slightly unusual setup in that I have just I4GL (p-code and c-code) and ISQL in the $INFORMIXDIR; the server is run out of a different directory. This means I don't have server utility programs like dbload (specifically) in $INFORMIXDIR/bin. When the sqldemo script tried to load the data with dbload, therefore, it failed for me. It would work for you because you have all the Informix software in a single directory. To add insult to injury, it runs the dbload program by explicit path, so I can't futz my PATH to make it available.
This should get you going. I have a bug to report...it is CQ idsdb00244894.
I'm sorry that you ran into so much trouble. You shouldn't have done so.

Resources