I can't find any information about how Bazel tracks file. The documentation doesn't mention if they use something like facebook's watchman.
It obviously takes some kind of hash and compares, but how exactly does it do it? Because it knows if things hasn't changed immediately and it wouldn't be able to read all those files in such a short time.
Also if you are watching many files it would take up a lot of space with a mono repo like Google? I know that is one of the problems scaling git because "git status" will become to slow unless some intelligent caching is used.
Bazel uses OS filesystem monitoring APIs like inotify on Linux and FSEvents on Mac OS
Check out these classes:
https://github.com/bazelbuild/bazel/blob/c5d0b208f39353ae3696016c2df807e2b50848f4/src/main/java/com/google/devtools/build/lib/skyframe/DiffAwareness.java
https://github.com/bazelbuild/bazel/blob/1d2932ae332ca0c517570f559c6dc0bac430b263/src/main/java/com/google/devtools/build/lib/skyframe/LocalDiffAwareness.java
https://github.com/bazelbuild/bazel/blob/c5d0b208f39353ae3696016c2df807e2b50848f4/src/main/java/com/google/devtools/build/lib/skyframe/MacOSXFsEventsDiffAwareness.java
Related
I got some strange glibc-related linker errors for builds with distributed build cache configured on build nodes running different Linux distributions.
Now I somehow suspect build artifacts from those machines with different glibc versions getting mixed up, but I don't know how to investigate this.
How do I find out what Bazel takes into account when building the hash for a certain build artifact?
I know I can explicitly set environment variables which then will affect the hash. But how can I be sure a given compiler, a certain version of glibc, etc. will lead to different hashes for built artifacts?
And how do I check/compare what's been taken into account?
This is a complex topic and a multi-facet question. I am going to answer in the following order:
How do I check/compare what's been taken into account?
How to investigate against which glibc a build linked?
How can I be sure a given compiler, a certain version of glibc, etc. will lead to different hashes for built artifacts?
How do I check/compare what's been taken into account?
To answer this, you should look into the the execution look, specifically you can read up on https://bazel.build/remote/cache-remote#compare-logs. The *.json execution log should contain everything you need to know (granted, it might be a bit verbose) and is a little easier to process with shell-magic/your editor.
How to investigate against which glibc a build linked?
From the execution log, you can get all the required hashes to retrieve cached artifacts/binaries from your remote cache. Given these files, you should be able to use standard tools to get to the glibc version (ldd -r -v binary | grep GLIBC).
How can I be sure a given compiler, a certain version of glibc, etc. will lead to different hashes for built artifacts?
This depends on the way you have setup for compilation toolchain. The best case would be a fully hermetic compilation toolchain, where all necessary files are declared using attributes like https://bazel.build/reference/be/c-cpp#cc_toolchain.compiler_files.
But this would also mean to lock-down the compiler sysroot. This should include all libraries you are linking against if you want full hermeticity. If you want to use some system libraries, you need to tell bazel where to find them and to factor in their hash: https://stackoverflow.com/a/43419786/20546409 or https://www.stevenengelhardt.com/2021/09/22/practical-bazel-depending-on-a-system-provided-c-cpp-library/
If you use the auto-detected compiler toolchain, some tricks are used to lock-down the sysroot paths, but expect some non-hermiticity. https://github.com/limdor/bazel-examples/tree/master/linux_toolchain is a nice write-up how to move from the auto-detected toolchain to something more hermetic.
The hack
Of course, you can hack around this. Note, this is inherently a bad idea:
create a script that inspects the system, determines everything important like the glibc version, maybe the linux distribution (flavor)
creates a string describing this variation and hash-summing it
use that as the instance key/name for your remote cache
The list of current NaCl ports is here: https://code.google.com/p/naclports/wiki/PortList
I'm curious if there is (or will be) a repository for PNaCl executables (.pexe files) since they only need a .nmf manifest wrapper to run?
Please list any PNaCl resources here, I'm looking for ImageMagick specifically.
I know I could probably build the .pexe myself but I don't have the time to learn Native Client.
The short answer is: this isn't available yet, but we are working on it.
The long answer:
There isn't currently a binary repository for naclports. We are planning to add that soon (likely this quarter).
I took a quick look at the ImageMagick port, and it looks like it currently only generates libraries, not an application binary (i.e. nexe/pexe). My guess is that it wouldn't be too hard to add this, however.
Even with a pexe, you'll need to have a way to launch the application, give it commandline options, and a set of files to process. I've discussed a way to do this with my team, but we haven't started work on it yet (though again, we'll likely work on this in Q1 2014).
So, for an embedded project with external NOR flash and existing commercial RTOS.
Can I use the coffee filesystem in a form of a library, i.e. to compile it in? From what I've seen it seems possible, only the dependencies go quite deep in to Contiki. I'll surely dig in, but may be someone have done something of the kind already?
This does look very doable.
From a quick grep in core/cfs/ it looks like there are no Contiki-specific abstractions used in the implementation of CFS. And another very quick search on GitHub shows that swift-nav/piksi_firmware contains and uses CFS without Contiki.
I've done some search out there but couldn't find too much really helpful info on it, but could someone try to explain the basic of Java memory maps? Like where/how to use it, it's purpose, and maybe some syntax examples (inputs/outputs types)? I'm taking a Java test soon and this could be one of the topics, but through all of my tutorials Jmap has not come up. Thanks in advance
Edit: I'm referring to the tool: jmap
I would read the man page you have referenced.
jmap prints shared object memory maps or heap memory details of a given process or core file or a remote debug server.
NOTE: This utility is unsupported and may or may not be available in future versions of the JDK. In Windows Systems where dbgeng.dll is not present, 'Debugging Tools For Windows' needs to be installed to have these tools working. Also, PATH environment variable should contain the location of jvm.dll used by the target process or the location from which the Crash Dump file was produced.
http://docs.oracle.com/javase/7/docs/technotes/tools/share/jmap.html
Its not a tool to be played with lightly. You need a good profiler which can read it output as jhat is only useful for trivial programs. (YourKit works just fine for 1+ GB heaps)
The team I work for manages a large collection of technical documentation which is written in LaTeX.
Currently all the documentation we have is manually built by the editors and then checked into a version control system. Sometimes people forget to compile their documents so we have a situation where the PDF and .tex files are often out of step. Unfortunately when this happens our users find themselves reading old versions of our document.
I've managed to hack a simple script to build PDFs using Make - it's rather clumsy.
I was wondering if there was a better way to do it? Most people in our department use Eclipse + Pydev for a Python project which means we are all very familiar with this IDE. I know that Ant plays nicely with Eclipse, so might we be able to use this tool for our doc building?
So what's the best way of doing this? I hope I will not have to learn everything there is to know about a new build-system in order to automate the building of some quite simple docs.
There is an external Ant task for LaTeX PDF generation, though the site is in German.
To use it, download the jar to a location on your machine, then define a taskdef as follows:
<taskdef name="latex" classname="de.dokutransdata.antlatex.LaTeX"
classpath="/path/to/ant/lib/ant_latex.jar"/>
Then to use it, define a target like this:
<target name="doLaTeX">
<latex
latexfile="${ltx2.file}"
verbose="on"
clean="on"
pdftex="off"
workingDir="${basedir}"
/>
</target>
Where ltx2.file is the file to process.
This is a link to the howto page listing the parameters. If you need any more options, my German is just about passable enough to explain, maybe.
There is also a maven plugin for LaTeX, but I can't find any documentation.
Haven't tried it, but I remember seeing a blog post about it.
If you know python, this blog post might be interesting
EDIT: Also, I would assume that you're using some kind of version control system, and I can't say for sure, but I use git to manage all my latex docs, and it might be possible to use some kind of post-commit hook to execute a script to rebuild the document. This would depend on how your repository is structured... just thinking out loud, so to speak.
I went into great detail on a large number of build systems for latex in this question, but its slightly different in your case. I think you want rubber or latexmk. The latex-makefile seems a good idea, but only supports building via postscript, which might not be your build process.
In general, its a good idea to keep generated files outside of version control for just this reason. A good exception is when specialist build tools are not widely available, and your situation sounds similar. You might do better with a commit-hook to build automatically upon commit.
I guess I should also point out that committing something without first building it and checking it is a deadly sin, so a better solution might be to stamp that out.
Maven is a better alternative as build system compared to Ant. So I would recommend a maven-plugin to generate PDF from LaTeX sources. Have a look at mathan-latex-maven-plugin