I'm baffled. I have a VM running Ubuntu 14.04. I've followed procedures here: http://clang.llvm.org/docs/LibASTMatchersTutorial.html and am at the step to run ninja. This builds llvm and clang. Now, my VM is no slouch, I gave it 6GB of RAM and 4 CPUs and a 20GB swap file. The biggest problem comes at link time - it seems to start a large number of ld processes, each using at least 3-4GB or virtual memory, and at some point a lot of CPU each. But the the swap file grew to over 12GB and the processes are all IO bound, but I don't know if they are doing something useful, or thrashing. All I know is the disk is getting hammered and the jobs run forever. I've actually just dropped the CPU count to the VM to 1, to see if it might be more efficient with less parallelism, as I surmised the issue may be thrashing.
I suppose my disk could be slow... Any ideas? Should I be using make instead of ninja? My expertise is not Linux (although I'm getting there :-) ) So I'm following the tutorial but perhaps it is not recommended the "best" way to build the clang / llvm programs.
I have been there, It's happening with the latest svn release (but not if you get clang 3.8 or older releases). What is happening is that since during development a lot of debug information is also being generated for each compilation unit the file sizes are becoming big.
The solution is to turn off all the debug info that's been attached by default. You probably are not going to debug clang, so won't need it. SO instead of just doing this
cmake -G Ninja ../llvm -DLLVM_BUILD_TESTS=ON
What you should do is
cmake -G Ninja ../llvm -DLLVM_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Release
All the other steps remain the same. Now I have not tested this with ninja, but have verified it with make on ubuntu (this tutorial, I modified the same thing in step 7). This should owkr as weel.
Related
Some days ago I upgraded my codebase from jest 26 to 27.
Running tests in my local environment worked like a charm but, when I tried to run them on my CI machine, the tests "never" stop.
Actually, in CI, the process exited correctly when trying to run tests sequentially but not in parallel, eg with --runInBand --detectOpenHandles --forceExit.
I tried to build and run the same docker locally... and it worked.
Changing randomly some docker's CPU / RAM configs, I achieved the same result: the process hangs.
Tests hanging:
Running top in docker:
As you can see, it's not a problem of CPU / RAM considering what top says.
Do you have any hints?
Do you need more information?
After some days debugging, I figured out that the problem was the amount of RAM and some memory leaks in the tests.
I advise you to use the node --expose-gc ./node_modules/.bin/jest --logHeapUsage command to dive into the problems you can have. https://jestjs.io/docs/cli#--logheapusage
This post helped me out as well: https://chanind.github.io/javascript/2019/10/12/jest-tests-memory-leak.html
TL;DR:
I fixed memory leaks by downgrading node from 16.13.2 to 16.10.0.
Details:
Based on Mattia Larentis answer:
I had the same problem, code and tests were written for client app, used packages:
node: 16.13.2
jest: 27.4.7
ts-jest: 27.1.3
typescript: 4.5.5
After debugging and analysing jest tests, I detected memory leaks too, using:
node --expose-gc ./node_modules/.bin/jest --logHeapUsage
Then I analysed node heap snapshots (for more info see Your Jest Tests are Leaking Memory).
I found a lot of strings, containing transpiled TS code.
I went to ts-jest & jest repos to look for issues about memory leaks.
I found these:
ts-jest: Module caching memory leak
jest: [Bug]: Memory consumption issues on Node JS 16.11.0+
node: vm Script memory leak in Node.js 14 / 16
Then, I downgraded node version from 16.13.2 to 16.10.0, checked memory leaks using
node --expose-gc ./node_modules/.bin/jest --logHeapUsage
And found that problem was gone.
I have reviewed blogpost from 2008 to date. I have Inherited a ruby on rails project for which I need to increase the test code.
I work on a laptop asus computer with an 8gen cpu i7U with 16gb ram and a 512gb ssd.
Initially I was running ubuntu 19.10, when I started the project and with about 1200 tests. it takes more than 1hr to run. Whereas on a 2015 macbook pro with 8gb of ram and an hdd, it takes only 2-3 min.
The log / test.log does not report errors, the tests do not hang, but waiting too long is not efficient, especially when i'll be increasing the number of tests.
So I Uninstall ubuntu, wipe off the ssd, install solus, arch and ubuntu, with the same setup for all through asdf as version manager and in no distro the time is less than 1hr.
Does anyone know why this happens in linux? The mac setup is also through asdf and it is fast enough.
Without knowing the specifics of the codebase or the tests, this question is equivalent to "how long is a piece of string."
There are many differences between linux and macOS. Cryptographic libraries may have different defaults. Memory limits for threads will be different. Memory limits for processors may be different.
Unless you can isolate specific tests which are wildly different and extrapolate from there, it's almost certainly going to come down to OS-level differences.
I'm building an elixir/phoenix application using a docker container.
This has been working for some time now, but recently it stopped working, with the error always being associated with a lack of memory.
For instance, the most frequent point of failure is during the mix compile task of Elixir (the most time heavy task in the Dockerfile), which crashes with the error:
eheap_alloc: Cannot allocate 147852528 bytes of memory (of type "old_heap").
Crash dump is being written to: erl_crash.dump...done
Sometimes it might be able to get through that step, but will again fail at a later step, like brunch build which compiles the frontend code. Sometimes it just fails at some other step with no specific error message, just saying:
Killed
While this is happening, I can easily check htop and see that I'm using 3 or 4GB of RAM, out of 16GB total, so there's no lack of physical RAM at all.
After some digging, I found that sudo sysctl vm.overcommit_memory=1 could help, but no luck there either.
The exact same build runs fine on my other computer, which runs Arch Linux, while this one runs Ubuntu 16.04
I am looking for a way to set up or modify an existing Docker image for installing tensorflow that will install it such that the SSE4, AVX, AVX2, and FMA instructions can be utilized for CPU speed up. So far I have found how to install from source using bazel How to Compile Tensorflow... and CPU instructions not compiled.... Neither of these explain how to do this within Docker. So I think what I am looking for is what you need to add to an existing docker image that installs without these options so that you can get a compile version of tensorflow with the CPU options enabled. The existing docker images do not do this because they want the image to run on as many machines as possible. I am using Ubuntu 14.04 on linux PC. I am new to docker but have installed tensorflow and have it working without getting the CPU warnings I get when I use the docker images. I may not need this for speed, but I have seen posts that claim the speed up can be significant. I searched for existing docker images that do this and could not find anything. I need this to work with gpu so needs to be compatible with nvidia-docker.
I just found this docker support for bazel and it might provide an answer, however I do not understand it well enough to know for sure. I believe this is saying that you can not build tensorflow with bazel inside a Dockerfile. You have to build a Dockerfile using bazel. Is my understanding correct and is this the only way to get a docker image with tensorflow compiled from source? If so, I could still use help in how to do it and still get the other dependencies that I would get if using an existing docker image for tensorflow.
Dockerfiles that build with CPU support can be found here.
Hope that helps! Spent many a late night here on Stack Overflow and Github Issues and stuff. Now it's my turn to give back! :)
The GPU stuff in particular is really hairy - especially when enabling the XLA/JIT/AOT stuff as well as the Graph Transform Tools.
Lots of hacks embedded in my Dockerfiles. Feel free to review and ask me questions!
The contributing guidelines mention building TensorFlow from source with Docker to run the unit tests:
Refer to the
CPU-only developer Dockerfile and
GPU developer Dockerfile
for the required packages. Alternatively, use the said
Docker images, e.g.,
tensorflow/tensorflow:nightly-devel and tensorflow/tensorflow:nightly-devel-gpu
for development to avoid installing the packages directly on your system.
I'm just starting to learn Grails and Groovy. Did a fresh install of Grails 1.3.8 today and it crawls - I must have something in my environment causing a problem. I ran create-app (following a tutorial), then run-app. Both took quite a while but hey, first time thru. Ran run-app again, took almost 5 minutes, apparently int eh compiling phase. Windows 7 machine with 4 GB memory, Intel i5 CPU. Don't know where to start looking, but know this isn't normal.
When I've run into this normally I'm out of heap or permGen space. Fire up visualvm and check both of those memory limits. It'll slowdown considerably before crashing if your memory limits are too small
I wonder how it happens for you. I have 2gb ram and I run Ubuntu on pc and it's so fast. First check your memory limits.
And for your question, you can find a good disscussion over here.