I exhausted my virtual memory when trying to install SyntaxNet from this Dockerfile using the Docker Toolbox. I received this message when compiling the Dockerfile:
ERROR: /root/.cache/bazel/_bazel_root/5b21cea144c0077ae150bf0330ff61a0/external/org_tensorflow/tensorflow/core/kernels/BUILD:1921:1: C++ compilation of rule '#org_tensorflow//tensorflow/core/kernels:svd_op' failed: gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wl,-z,-relro,-z,now -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-canonical-system-headers ... (remaining 115 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1. virtual memory exhausted: Cannot allocate memory ____Building complete. ____Elapsed time: 8548.364s, Critical Path: 8051.91s
I have a feeling this could be resolved by changing Bazel's default jobs limit with (for example) --jobs=1, however I'm not sure where I would put that in the Dockerfile.
There are two possibilities: You could either modify the Dockerfile so that it creates a ~/.bazelrc that contains the following text:
build --jobs=1
Note that this works, even though the Dockerfile runs bazel test (as opposed to bazel build), because build flags in the .bazelrc also apply to Bazel's test command.
The other possibility would be to modify the RUN command in the Dockerfile to include the --jobs=1 parameter, e.g. RUN [...] && bazel test --jobs=1 --genrule_strategy=standalone [...].
Bazel should then spawn not more than a single child process during the build. You can verify this by running "ps axuf" on your host and looking at the process tree of your container. If you modified the RUN cmd, you should also see the --jobs=1 parameter on Bazel's command-line.
Related
I am working on a C++ project with drake, using bazel as the build system. Previously, I use the drake source code as the external, following the drake_bazel_external example. Everything works fine.
Since I want to use the SNOPT solver in drake, I want to change to use the pre-compiled drake. I follow the drake_bazel_installed example. However, I got the following errors.
Compiling kuka/diffIK_controller.cc failed: (Exit 1): gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 27 arguments skipped)
Use --sandbox_debug to see verbose messages from the sandbox
In file included from bazel-out/k8-opt/bin/external/drake/_virtual_includes/.drake_headers/drake/common/default_scalars.h:3,
from bazel-out/k8-opt/bin/external/drake/_virtual_includes/.drake_headers/drake/systems/framework/leaf_system.h:14,
from ./kuka/diffIK_controller.h:3,
from kuka/diffIK_controller.cc:3:
bazel-out/k8-opt/bin/external/drake/_virtual_includes/.drake_headers/drake/common/autodiff.h:12:10: fatal error: Eigen/Core: No such file or directory
12 | #include <Eigen/Core>
| ^~~~~~~~~~~~
compilation terminated.
I also find that the apps in the drake_bazel_external cannot be compiled successfully by drake_bazel_installed setting. The error message is
ERROR: error loading package 'app': Label '#drake//tools/skylark:py.bzl' is invalid because 'tools/skylark' is not a package; perhaps you meant to put the colon here: '#drake//:tools/skylark/py.bzl'?
Update
The bug can be produced by both the http_archive fetched drake and the apt installed drake (the latest stable drake I think, since I just installed it yesterday). I have isolated the relevant code to reproduce the bug in a github repo. Currently, I can get the original apps in drake_bazel_installed to work.
Update
By adding
# solve the eigen not found bug
build --cxxopt=-I/usr/include/eigen3
to the .bazelrc file, I can solve the above problem. However, when I try to build a program that uses iiwa_status_receiver.h, I encounter a new problem.
ERROR: /home/chenwang/repro_drake_bazel_external/drake_bazel_installed/apps/BUILD.bazel:102:10: Compiling apps/connection_test.cc failed: (Exit 1): gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 32 arguments skipped)
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
In file included from apps/connection_test.cc:10:
bazel-out/k8-opt/bin/external/drake/_virtual_includes/.drake_headers/drake/manipulation/kuka_iiwa/iiwa_status_receiver.h:6:10: fatal error: drake/lcmt_iiwa_status.hpp: No such file or directory
6 | #include "drake/lcmt_iiwa_status.hpp"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
INFO: Elapsed time: 2.967s, Critical Path: 0.24s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully
This problem is also a missing header file problem. I have update the github repo to reproduce this problem.
This is a bug in Drake (filed as https://github.com/RobotLocomotion/drake/issues/17965 now).
To work around it, pass --cxxopt=-I/usr/include/eigen3 on all of your bazel commands, e.g., by adding this line to your projects' .bazelrc file:
build --cxxopt=-I/usr/include/eigen3
Edit: The nightly builds of Apt packages as of 20220923 should have this fixed as well.
I'm running a snakemake pipeline that for a specific rule loads a container:
rule counts:
params:
transcriptome=os.environ["INDEX"],
outdir= (os.environ["OUTDIR"] + "/counts/"),
indir= (os.environ["INDIR"] + "{sample}"),
name = lambda wildcards: SAMPLES[wildcards.sample]
output:
(os.environ["OUTDIR"] + "counts/" + "{sample}" + "/outs/web_summary.html")
container:
"docker://marcusczi/cellranger_clean"
shell:
"""
cellranger count --id={wildcards.sample} --transcriptome={params.transcriptome} --fastqs={params.indir} --sample={params.name}
mkdir -p {params.outdir}
mv ./{wildcards.sample}/ {params.outdir}
"""
Dry run looks fine, the rule itself I'm sure it works (tried it without the container). However, when I run it with docker I get this error:
Activating singularity image /some/path/.snakemake/singularity/c288fbc3fef5771f055a688c6678c24d.simg
/bin/sh: syntax error: unterminated quoted string
[ 1.228141] reboot: Power down
And then it waits for the missing files, and fails.
I think the answer to this situation might be related to this previous question, but I have tried everything i can think of in terms of escaping characters (except for the wildcards and variables within curly brackets because I'm guessing it should be fine, and if not why am i even using snakemake :-( ). The paths for the directories I'm using are valid and exist, the name and wildcard "sample" are in the shape "sample_123", nothing fancy.
It's also worth saying that there are no single or double quotes in any of these variables.
Thank you!!
Software and OS:
I am in macos catalina 10.15.5, running snakemake 5.20.1, and I have been using the beta version of singularity for macos (3.3.0-rc.1.658.g7427b73f1.dirty).
Running singularity outside Snakemake:
I tried running the singularity outside snakemake, the software that I'm trying to run starts, but then complains that there is no disk left on space (which is not true). I'm running the singularity as sudo singularity run -B "$(pwd):$(pwd)" docker://marcusczi/cellranger_clean
I think this latest error might be either 1) I'm not running singularity as I should..? Or 2) A false statement of what is happening since cellranger (the software I'm trying to run) often has misleading error messages.
Minimal reproducible example:
If you install snakemake, you should be able to reproduce my error when running snakemake -j1 --use-singularity in the same directory of the Snakefile.
Snakefile:
rule all:
input:
"output.txt"
rule counts:
output:
"output.txt"
container:
"docker://marcusczi/cellranger_clean"
shell:
"""
cellranger count --help
echo "hurray!" > {output}
"""
I am trying to recognise image which has been trained using Tensorflow. I followed this steps
and finally i got success to train my own dataset and its give a good prediction result. All of the code is in python. Now i am trying to use this trained model in my iOS project . I am following this tutorial to use my trained model in my iOS project. But when i followed those steps i got an error in my mac terminal window like -
"ERROR: /tensorflow/tensorflow/core/kernels/BUILD:2235:1: C++ compilation of rule '//tensorflow/core/kernels:self_adjoint_eig_v2_op' failed: gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG ... (remaining 124 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 4. gcc: internal compiler error: Killed (program cc1plus)".
What will be the solution for this issue or is there any way to convert this tensor-flow model to iOS supported Core ML Model ? Here i'm sharing a screen shot of that error. Please help me out. Thanks.
Most likely this is due to insufficient memory inside the docker container for bazel to run with default parameters. Try rerunning (after bazel clean) the build with these extra flags to the build command: --local_resources 2048,2.0,1.0 -j 1. You can also try giving a little more resources: --local_resources 4096,2.0,1.0 -j 1
Here are some links for this issue: https://github.com/tensorflow/tensorflow/issues/1530
Tensorflow Serving Compile Error Using Docker on OSX
I am trying to run coverage on my project, after updating to Ubuntu 16.04.
I get
Deleted 665 files
Writing data to coverage.info.cleaned
lcov: ERROR: cannot write to coverage.info.cleaned!
CMakeFiles/coverage.dir/build.make:57: recipe for target 'CMakeFiles/coverage' failed
make[3]: *** [CMakeFiles/coverage] Error 13
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/coverage.dir/all' failed
make[2]: *** [CMakeFiles/coverage.dir/all] Error 2
CMakeFiles/Makefile2:74: recipe for target 'CMakeFiles/coverage.dir/rule' failed
make[1]: *** [CMakeFiles/coverage.dir/rule] Error 2
Makefile:129: recipe for target 'coverage' failed
make: *** [coverage] Error 2
enter code here
Before the update I had no problem running the coverage
Does it help if you use absolute paths instead of relative paths when passing files to lcov?
I ran into a similiar problem where lcov also failed to write the file.
Not sure if it is a bug in lcov, but the problem was that it got confused with relative paths:
lcov -a test_fast_cxxtest_gcov__base.info -a test_fast_cxxtest_gcov__test.info \
-o test_fast_cxxtest_gcov__total.info
Combining tracefiles.
Reading tracefile test_fast_cxxtest_gcov__base.info
Reading tracefile test_fast_cxxtest_gcov__test.info
lcov: WARNING: function data mismatch at /home/phil/ghost/constants.h:1862
Writing data to test_fast_cxxtest_gcov__total.info
lcov: ERROR: cannot write to test_fast_cxxtest_gcov__total.info!
Running it with strace revealed that it executes chdir("/") on several locations, which changes the working directory to /. That explains why it cannot write the file.
One workaround is to use absolute paths. For instance, if you are using GNU make, you can use the abspath command:
lcov -a $(abspath test_fast_cxxtest_gcov__base.info) \
-a $(abspath test_fast_cxxtest_gcov__test.info) \
-o $(abspath test_fast_cxxtest_gcov__total.info)
After that change, it was finally able to write the file.
(Other options like trying to set the directories using the --base-directory or --directory option did not have an effect, as far as I saw.
The version of lcov that I tested with is 1.12.)
The problem is not limited to Ubuntu, as I ran into it on Arch Linux. It could be a regression introduced in 1.12, however, so I reported it (see issue #77630).
Update: Lcov is not part of GCC, so my original bug report was closed, but I got an answer from the Lcov mailing list. The problem is already fixed in commit 632c25. Users of Arch Linux based distros can try the latest snapshot with aur/lcov-git.
(I have the answer to this already; I'm going to answer my own question so that I can share what I've learned and save someone else this trouble in the future)
When I attempt to build Erlang on Solaris 10 Sparcv9, the build fails partway through:
cd lib && \
ERL_TOP=/var/tmp/pkgbuild-0/erlang/sparcv9/erlang-otp-73b4221 PATH=/var/tmp/pkgbuild-0/erlang/sparcv9/erlang-otp-73b4221/bootstrap/bin:${PATH} \
make opt SECONDARY_BOOTSTRAP=true
make[1]: Entering directory `/var/tmp/pkgbuild-0/erlang/sparcv9/erlang-otp-73b4221/lib'
make[2]: Entering directory `/var/tmp/pkgbuild-0/erlang/sparcv9/erlang-otp-73b4221/lib/hipe'
=== Entering application hipe
make[3]: Entering directory `/var/tmp/pkgbuild-0/erlang/sparcv9/erlang-otp-73b4221/lib/hipe/misc'
erlc -W +debug_info +warn_exported_vars +warn_missing_spec +warn_untyped_record -o../ebin hipe_consttab.erl
make[3]: *** [../ebin/hipe_consttab.beam] Bus Error (core dumped)
make[3]: Leaving directory `/var/tmp/pkgbuild-0/erlang/sparcv9/erlang-otp-73b4221/lib/hipe/misc'
Why is this and what can I do to complete my Erlang build?
The reason that the build fails is due to a broken build environment.
In this specific case the Sun GCC build is being used. This particular version of GCC was compiled to use a mixture of the the GNU assembler and the Sun linker.
The Sparc platform is highly sensitive to alignment of code and it will fault (for example, with a bus error) if unaligned code is executed.
The GNU assembler used by the stock GCC build on Sparc Solaris 10 doesn't work so hard to automatically align functions generated by the compiler, leading to unaligned code.
The solution is to build your own GCC and make sure that you use the system assembler and linker; you can achieve this by using the following options to GCC's configure script:
--with-as=/usr/ccs/bin/as \
--without-gnu-as \
--without-gnu-ld \
--with-ld=/usr/ccs/bin/ld \
The resultant GCC build will generated properly aligned code and allow you to build Erlang successfully.