How do I specify another directory where cargo keeps the unpacked sources, e.g. on Windows this is %userprofile%/.cargo/registry/src?
I working that is going to be a collection of crates like:
myproject-tree/
foo
+Cargo.toml
bar
+Cargo.toml
So I would like the dependencies to be loaded in unpacked somewhere near the myproject-tree directory, e.g.
myproject-tree/
cargo-cache/
I think that would make it easier to inspect the code of dependencies (e.g. in the code editors with file tree UI). I can see that there is CARGO_HOME that seems like it has something like this, but it does not look convenient, e.g. each time I am going to work with myproject-tree I have to set it.
Looks like this is possible. I think it called "vendoring" and repositories like Fuchsia and Mozilla has source copy of third-party crates in their repositories. This is what I have found:
Cargo supports "Source replacement" thing configured in .cargo/config, which can be in a project relative dir:
With this configuration Cargo attempts to look up all crates in the
directory "vendor" rather than querying the online registry at
crates.io. Using source replacement Cargo can express:
Vendoring - custom sources can be defined which represent crates on
the local filesystem. These sources are subsets of the source that
they're replacing and can be checked into packages if necessary.
Mirroring - sources can be replaced with an equivalent version which
acts as a cache for crates.io itself.
There is a tool https://github.com/alexcrichton/cargo-vendor that can used to populate the directory with source codes of the dependencies:
Simply run cargo vendor inside of any Cargo project:
Related
Let's say, I have a completely new VPS server which I've just rolled out, which I haven't installed anything on yet.
And I've compiled and build a production release of Phoenix application on my local machine which is identical to a VPS server Linux distributive- and version-wise.
In the directory _build/prod/rel/my_app123 there have been generated 4 subdirectories:
bin
erts-12.3
lib
releases
Will copying the content of rel/my_app123/, that is, these 4 subdirectories, over to a VPS will be absolutely enough in order to run an application?
Or will I have install something extra as well? Elixir and Erlang?
How about production dependencies from mix.exs? Or are these have been included and compiled into into a release?
P.S. Assume that my web application has no "js", "css" and the like files, and doesn't use a database.
When you run mix release, it bundles all of your Elixir/Erlang dependencies for the MIX_ENV in question into the release directory, the erlang BEAM runtime/VM that you were using in your build, and any files that you specify in your mix project in mix.exs.
Because the BEAM runtime and code that bootstraps loading your code are included in the release, you won't need to install Elixir or Erlang on the target machine.
Things that are not included include:
any non-Elixir dependencies. For example, if you rely on openssl, you'll need to make sure you have a binary-compatible version of that installed on the machine you plan to run on (typically, the equivalent major verson release).
Portable bytecode. BEAM isn't like the Java VM. The compiled BEAM code needs to run on a substantially similar architecture. Build on an Arm64 machine for deployment on an Arm64 virtual machine, or x86 for Intel-compatible hardware, for instance. And it's probably best to use the same major OS distribution. There may be cases where "Any Linux * Same CPU architecture" is fine, but for example, building on a Windows or MacOS install of Elixir/OTP and deploying on Linux is a non-starter; you'd need to use a sufficiently similar OS.
As an example, one of my projects has its releases built on Alpine using Docker, so we only really have to worry about CPU compatibility. In our case we do need to make sure some external non-Elixir dependencies our app binds to are included on the docker image.
RUN apk add --no-cache libstdc++ openssl ncurses-libs wkhtmltopdf xvfb \
fontconfig \
freetype \
ttf-dejavu
(ignore the fact that wkhtmltopdf is kind of deprecated, we're working on it. But for now it's a non-elixir dependency we rely on).
If you're building for a, say, an EC2 instance and not using Docker, you'd just need to make sure your release is built on a similar OS to what you're using for production, and make sure the production AMI (image) has those non-Elixir dependencies on it, or will at the time of deployment, perhaps using apt or another package manager. For a VPS, the solution for non-elixir dependencies will depend on whether they have the option for customizing the base machine image (maybe with Packer or Ansible)
Since you may seem to have been a bit confused about it in the comments, yes, MIX_ENV=prod mix release will build all of your production Elixir/Erlang dependencies and include them in the /_build/prod folder.
I include the whole ./prod folder in our release, but it looks like protocol consolidation binaries and the lib folder .Beam files are all in the rel folder so that's a bit unnecessary.
If you do a default build, the target will be inside your _build directory, with sub-directories for the config environment and your application, e.g. _build/dev/rel/your_app/. That directory should contain everything you need to run your app -- the prompt after running mix release provides some clues for this when it says something like:
Release created at _build/dev/rel/your_app!
I find it more useful, however, to zip up the app into a single portable file (and yes, I agree that the details about how to do this are not necessarily the first things you see when reading about Elixir releases). The trick is to customize your mix.exs by fleshing out the releases option -- this is usually done via a dedicated private function but the organization of how you supply the options is up to you.
What I find is often useful is the generation of a single zipped .tar.gz file. This can be accomplished by specifying the include_executables_for option along with steps. It looks something like this:
# mix.exs
defmodule YourApp.MixProject do
use Mix.Project
def project do
[
# ...
releases: releases()
# ...
]
end
defp releases do
[
my_app: [
include_executables_for: [:unix],
steps: [:assemble, :tar]
]
]
end
When you configure your application this way, running mix release will generate a nice portable file containing your app with everything it needs. Unzipping this file is education for understanding everything your app needs. By default this file will be created at a location like _build/dev/yourapp-1.0.0.tar.gz. You can configure the build path by specifying a path for your app. See Mix.Release for more options.
I have a base image (mybase:1.0.0) which builds and installs a few CMake projects successfully.
I want to make a more specialised image (myapp:1.0.0) which is composed from mybase:1.0.0 and adds my application, which depends on the libraries and binaries installed in mybase:1.0.0.
Each CMake build creates install_manifest.txt which has a path to where each binary or library was installed>
Example manifest:
/usr/local/bin/myapp
/usr/local/lib/mylib.so
In my specialised image I would like to copy these paths to my new container, which should be based not on mybase:1.0.0 but another specialised (and compatible) image (in my case nvidia/cuda:10.2-base-ubuntu18.04).
To do so I would have my dockerfile something like:
FROM nvidia/cuda:10.2-base-ubuntu18.04
COPY --from mybase:1.0.0 /usr/local/bin/myapp /usr/local/bin/myapp
COPY --from mybase:1.0.0 /usr/local/lib/mylib.so /usr/local/lib/mylib.so
My problem is that there are many (~200) files which need to be installed, and including them all in the dockerfile would be very ugly, and when the libraries update and the base image is built again, the dockerfile would need to be updated.
One solution would be to copy all the files from the build directory directly to /usr/local/bin and /usr/local/lib. But in order to reduce file size significantly, the build directories are removed (aside from the manifests) after the base image has installed them (which is a requirement).
A workaround to this would be to have a second 'pre-base' image which deletes the installed images after build (the build libraries have interdependencies which require them to be installed).
Is there a way to do a COPY --from where the contents of a manifest.txt specify the SRC paths?
Or any practices which do similar that I can use?
I asked for a feature request to do so, and I received a response on how to do it using the docker experimental API.
See: https://github.com/docker/for-linux/issues/1060#issuecomment-655566235
I have cloned and built the waf script using:
./waf-light configure
Then to build my project (provided by Gomspace) I need to add waf and the eclipse.py to my path. So far I haven't found better than this setenv script:
WAFROOT=~/git/waf/
export PYTHONPATH=$WAFROOT/waflib/extras/:$PYTHONPATH
export PATH=~/git/waf/:$PATH
Called with:
source setenv
This is somehow a pretty ugly solution. Is there a more elegant way to install waf?
You don't install waf. The command you found correctly builds waf: /waf-light configure build Then for each project you create, you put the built waf script into that projects root directory. I can't find a reference, but this is the way in which waf:s primary author Thomas Nagy wants the tool to be used. Projects that repackage waf to make the tool installable aren't "officially sanctioned."
There are advantages and disadvantages with non-installation:
Disadvantages:
You have to add the semi-binary 100kb large waf file to your repository.
Because the file contains binary code, people can have legal objections to distributing it.
Advantages:
It doesn't matter if new versions of waf break the old API.
Users don't need to install waf before compiling the project -- having Python on the system is enough.
Fedora (at least Fedora 22) has a yum package for waf, so you could see that it's possible to do a system install of waf, albeit with a hack.
After you run something like python3 ./waf-light configure build, you'll get a file called waf that's actually a Python script with some binary data at the end. If you put it into /usr/bin and run it as non-root, you'll get an error because it fails to create a directory in /usr/bin. If you run it as root, you'll get the new directory and /usr/bin/waf runs normally.
Here's the trick that I learned from examining the find_lib() function in the waf Python script.
Copy the waf to /usr/bin/waf
As root, run /usr/bin/waf. Notice that it creates a directory. You'll see something like /usr/bin/.waf-2.0.19-b2f63c807a4215294bf6005410c74c18
mv that directory to /usr/lib, dropping the . in the directory name, e.g. mv /usr/bin/.waf-2.0.19-b2f63c807a4215294bf6005410c74c18 /usr/lib/waf-2.0.19-b2f63c807a4215294bf6005410c74c18
If you want to use waf with Python3, repeat Steps 2-3 running the Python script /usr/bin/waf under Python3. Under Python3, the directory names will start with .waf3-/waf3- instead instead of .waf-/waf-.
(Optional) Remove the binary data at the end of /usr/bin/waf.
Now, non-root should be able to just use /usr/bin/waf.
That said, here's something to consider, like what another answer said: I believe waf's author intended waf to be embedded in projects so that each project can use its own version of waf without fear that a project will fail to build when there are newer versions of waf. Thus, the one-global-version use case seems to be not officially supported.
The app I work on needs to use the wpftoolkit.extended.dll (i.e. no source, no msi/installer, we've only got the dll). So far we've placed the dll in a c:\libs folder on both the dev's laptop and the teambuild server and it built ok on both; now for deploying we want to add it to an installer (.vdproj) and we think we'll need it in tfs's repository somewhere. However, when tested the app now only builds on the dev's laptop and not on the teambuild server (looks like a relative path thing).
So... rather than fixing the actual problem, I'm wondering what's the best/cleanest/commonlyAccepted way to do this? where should I keep the dll in the repository and where should I place the dll on the host we're deploying to?
You should use folder structure on the source control like the following
/Main Contains the .sln file
/Source
/MyApp1 Contains MyApp1.sln file
/Source Contain folder for all
/ClassLibrary1 Contains ClassLibrary1.csproj
/MyApp1Web Contains Default.aspx
/Build Contains build output (binaries)
/Docs Contains product docs etc
/Tests
**/3rdpartyDlls** Contains all vesions of third-party dlls
For more information about the source control folders and best practices, it's recommended to read the book patterns & practices Team Development with TFS Guide (Final Release)
I have an erlang application I have been writing which uses the erldis library for communicating with redis.
Being a bit of a newbie with actually deploying erlang applications to production, I wanted to know if there was anyway to 'bundle' these external libraries with the application rather than installing into my system wide /usr/lib/erlang/lib/ folder.
Currently my directory structure looks like...
\
--\conf
--\ebin
--\src
I have a basic Makefile that I stole from a friend's project, but I am unsure how to write them properly.
I suspect this answer could involve telling me how to write my Makefile properly rather than just which directory to plonk some external library code into.
You should really try to avoid project nesting whenever possible. It can lead to all sorts of problems because of how module/application version is structured within Erlang.
In my development environment, I do a few things to simplify dependancies and multiple developed projects. Specifically, I keep most of my projects sourced in a dev directory and create symlinks into an elibs dir that is set in the ERL_LIBS environmental variables.
~/dev/ngerakines-etap
~/dev/jacobvorreuter-log_roller
~/dev/elib/etap -> ~/dev/ngerakines-etap
~/dev/elib/log_roller -> ~/dev/jacobvorreuter-log_roller
For projects that are deployed, I've either had package-rpm or package-apt make targets that create individual packages per project. Applications get boot scripts and init.d scripts for easy start/stop controls but libraries and dependancy projects just get listed as package dependencies.
I use mochiweb-inspired style. To see example of this get your copy of mochiweb:
svn checkout http://mochiweb.googlecode.com/svn/trunk/ mochiweb
and use
path/to/mochiweb/scripts/new_mochiweb.erl new_project_name
to create sample project of the structure (feel free to delete everything inside src afterwards and use it for your project).
It looks like this:
/
/ebin/
/deps/
/src/
/include/
/support/
/support/include.mk
Makefile
start.sh
ebin contains *.beam files
src contains ***.erl files and local *.hrl files
include contains global *.hrl files
deps contains symlinks to root directories of dependencies
Makefile and include.mk takes care of including appropriate paths when project is built.
start.sh takes care of including appropriate paths when project is run.
So using symlinks in deps directory you are able to fine tune the versions of libraries you use for every project. It is advised to use relative paths, so afterwards it is enough to rsync this structure to the production server and run it.
On more global scale I use the following structure:
~/code/erlang/libs/*/
~/code/category/project/*/
~/code/category/project/*/deps/*/
Where every symlink in deps points to the library in ~/code/erlang/libs/ or to another project in the same category.
The simplest way to do this would be to just create a folder named erldir and put the beams you need into it and then in your start script just use the -pa flag to the erlang runtime to point out where it should fetch the beams.
The correct way (at least if you buy into the OTP distribution model) would be to create a release using reltool (http://www.erlang.org/doc/man/reltool.html) or systools (http://www.erlang.org/doc/man/systools.html) which includes both your application and erldis.
Add the external libraries that you need, anywhere you want them, and add them to your ERL_LIBS environment variable. Separate the paths with colon in unix or semicolon in dos.
Erlang will add the "ebin"-named subdirs to its code loading path.
Have your *.app file point out the other applications it depends on.
This is a good halfway-there approach for setting up larger applications.
Another way is put your lib path in ~/.erlang.
code:add_pathz("/Users/brucexin/sources/mochiweb/ebin").
code:add_pathz("/Users/brucexin/sources/webnesia/ebin").
code:add_pathz("./ebin").
code:add_pathz("/Users/brucexin/sources/erlang-history/ebin/2.15.2").