I'm trying to compile PyPy for use on an OpenWrt configuration, but I am having a really hard time doing it.
My main problems are:
Each time I change the Makefile I am forced to start the translating process of PyPy again. Is there a way to avoid this?
Would copying just the compiled pypy-c and lib*.so binaries do it, or would I have to copy everything from the compiled files of PyPy?
Here is the directory structure after running make on the files.
How do I specify the version of GCC to use? I've tried to do this without success.
How would I get the interpreter to run after installing it on OpenWrt?
One can also compile PyPy using PyPy itself, which gives a .tar file with a different structure (no Makefile, pypy executable etc). Can I use that instead of building it from the source?
Here is my Makefile.
include $(TOPDIR)/rules.mk
PKG_NAME:=PyPy
PKG_VERSION:=5.8.0
PKG_RELEASE:=1
PKG_MD5SUM:=504c2d522595baf8775ae1045a217a2b120732537861d31b889d47c340b58bd5
PKG_SOURCE_NAME:=pypy2
PKG_SOURCE_URL:=https://bitbucket.org/pypy/pypy/downloads/
PKG_SOURCE:=$(PKG_SOURCE_NAME)-v$(PKG_VERSION)-src.tar.bz2
PKG_BUILD_DEPENDS:=python
PKG_CAT:=bzcat
PKG_BUILD_DIR:=$(BUILD_DIR)/$(BUILD_VARIANT)$(PKG_SOURCE_NAME)-v$(PKG_VERSION)-src
include $(INCLUDE_DIR)/package.mk
$(call include_mk, python-package.mk)
define Package/PyPy
SECTION:=utils
CATEGORY:=Utilities
TITLE:=PyPy
URL:=https://pypy.org/index.html
DEPENDS:=+libffi +libexpat +libunwind
endef
define Package/PyPy/description
PyPy is an alternate implementation of the Python programming language written in Python.
This distribution is for Linux architecture, using Python 2.
endef
define Package/PyPy/install
$(INSTALL_DIR) $(1)/usr/bin
$(INSTALL_BIN) $(PKG_BUILD_DIR)/pypy-c $(1)/usr/bin
$(CP) (PKG_BUILD_DIR)/*.so $(1)/usr/lib
$(INSTALL_BIN) $(PKG_BUILD_DIR)/pypy/bin/* $(1)/usr/bin
endef
define Build/Compile
$(MAKE) -C $(PKG_BUILD_DIR)
endef
$(eval $(call BuildPackage,PyPy))
I would need to see the entire process: the part of the Makefile above, the other pieces such as rules.mk, package.mk, command line arguments and environmental variables in order to understand what is not working
The Makefile in question is not supported, the developers do not use it, and as you discovered it does not work well. As described in the build page, building PyPy from source is a four part process, and the Makefile mashes three of those stages together so any changes currently require restarting from 0. Note that the underlying build process uses PYPY_USESSION_DIR, PYPY_USESSION_BASENAME, and a numbering system to ensure that each time the first stage (translation from RPython to C) puts the results in a new, clean directory. For this reason if you restart the process you will lose your previous work.
Python is both a binary interpreter and a stdlib of "battery included" modules. If you wish to use the binary interpreter, you need to install the binary and the stdlib support files together since they work as a unit. That is why we specify a fourth step in the build page, packaging. Please read that section carefully for methods of using the interpreter+stdlib
PyPy does not really support cross compiling, one method that used to work is described in the documentation of RPython. PyPy does support native gcc and/or clang. What compiler are you trying to use on what hardware?
Run the interpreter as you would any python interpreter; specify the path to the executable file, which (as stated above) needs to know how to find the support files such as the stdlib and the site-packages directory of third-party modules installed specifically for PyPy.
Using PyPy to build PyPy should be no different than using CPython to build PyPy, except it will be much faster. The differences you see must be an artifact of how you are building.
I would recommend you NOT use the unsupported Makefile until you understand the build, compile, package, install process, and then once you have a working installation help the PyPy project improve the process until it can be automated into a Makefile
If you are running OpenWRT on supported hardware and operating system, you would be much happier using a binary distribution and not compiling from source. In this case your Makefile should download a binary distribution and simply install it.
If you are using a different CPU, chances are PyPy will not work out-of-the-box on your hardware, you will have to run tests to make sure the JIT will actually make things faster.
Related
Let's say, I have a completely new VPS server which I've just rolled out, which I haven't installed anything on yet.
And I've compiled and build a production release of Phoenix application on my local machine which is identical to a VPS server Linux distributive- and version-wise.
In the directory _build/prod/rel/my_app123 there have been generated 4 subdirectories:
bin
erts-12.3
lib
releases
Will copying the content of rel/my_app123/, that is, these 4 subdirectories, over to a VPS will be absolutely enough in order to run an application?
Or will I have install something extra as well? Elixir and Erlang?
How about production dependencies from mix.exs? Or are these have been included and compiled into into a release?
P.S. Assume that my web application has no "js", "css" and the like files, and doesn't use a database.
When you run mix release, it bundles all of your Elixir/Erlang dependencies for the MIX_ENV in question into the release directory, the erlang BEAM runtime/VM that you were using in your build, and any files that you specify in your mix project in mix.exs.
Because the BEAM runtime and code that bootstraps loading your code are included in the release, you won't need to install Elixir or Erlang on the target machine.
Things that are not included include:
any non-Elixir dependencies. For example, if you rely on openssl, you'll need to make sure you have a binary-compatible version of that installed on the machine you plan to run on (typically, the equivalent major verson release).
Portable bytecode. BEAM isn't like the Java VM. The compiled BEAM code needs to run on a substantially similar architecture. Build on an Arm64 machine for deployment on an Arm64 virtual machine, or x86 for Intel-compatible hardware, for instance. And it's probably best to use the same major OS distribution. There may be cases where "Any Linux * Same CPU architecture" is fine, but for example, building on a Windows or MacOS install of Elixir/OTP and deploying on Linux is a non-starter; you'd need to use a sufficiently similar OS.
As an example, one of my projects has its releases built on Alpine using Docker, so we only really have to worry about CPU compatibility. In our case we do need to make sure some external non-Elixir dependencies our app binds to are included on the docker image.
RUN apk add --no-cache libstdc++ openssl ncurses-libs wkhtmltopdf xvfb \
fontconfig \
freetype \
ttf-dejavu
(ignore the fact that wkhtmltopdf is kind of deprecated, we're working on it. But for now it's a non-elixir dependency we rely on).
If you're building for a, say, an EC2 instance and not using Docker, you'd just need to make sure your release is built on a similar OS to what you're using for production, and make sure the production AMI (image) has those non-Elixir dependencies on it, or will at the time of deployment, perhaps using apt or another package manager. For a VPS, the solution for non-elixir dependencies will depend on whether they have the option for customizing the base machine image (maybe with Packer or Ansible)
Since you may seem to have been a bit confused about it in the comments, yes, MIX_ENV=prod mix release will build all of your production Elixir/Erlang dependencies and include them in the /_build/prod folder.
I include the whole ./prod folder in our release, but it looks like protocol consolidation binaries and the lib folder .Beam files are all in the rel folder so that's a bit unnecessary.
If you do a default build, the target will be inside your _build directory, with sub-directories for the config environment and your application, e.g. _build/dev/rel/your_app/. That directory should contain everything you need to run your app -- the prompt after running mix release provides some clues for this when it says something like:
Release created at _build/dev/rel/your_app!
I find it more useful, however, to zip up the app into a single portable file (and yes, I agree that the details about how to do this are not necessarily the first things you see when reading about Elixir releases). The trick is to customize your mix.exs by fleshing out the releases option -- this is usually done via a dedicated private function but the organization of how you supply the options is up to you.
What I find is often useful is the generation of a single zipped .tar.gz file. This can be accomplished by specifying the include_executables_for option along with steps. It looks something like this:
# mix.exs
defmodule YourApp.MixProject do
use Mix.Project
def project do
[
# ...
releases: releases()
# ...
]
end
defp releases do
[
my_app: [
include_executables_for: [:unix],
steps: [:assemble, :tar]
]
]
end
When you configure your application this way, running mix release will generate a nice portable file containing your app with everything it needs. Unzipping this file is education for understanding everything your app needs. By default this file will be created at a location like _build/dev/yourapp-1.0.0.tar.gz. You can configure the build path by specifying a path for your app. See Mix.Release for more options.
Introduction:
I have to create a pip wheel of Tensorflow 2.2.0 with cuda libraries dynamically linked(specifically cudart.so). To accomplish this i am currently using the tensorflow-dev docker image.
I am able to build the tf wheel file, an able to install and use it while inside the build container.
Issue:
The issue is that importing the generated wheel file in a CentOS server, i get the following error:
ImportError: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /home1/private/mavridis/Vineyard/tensorflowshared/test/lib64/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)
Having looked around, the issue is caused by the build container using a newer libc:
ldd --version
ldd (Ubuntu GLIBC 2.27-3ubuntu1) 2.27
Compared to CentOS older version:
ldd --version
ldd (GNU libc) 2.17
Expected behavior:
Having already tried the 'vanilla' tenorflow 2.2.0 version with no issues, installed using pip:
pip install tensorflow==2.2.0
I expected my own build to also work.
So i assume there is some configuration option or docker configuration to allow me to use the docker built wheel file, in a CentOS setup, just like the pip installed version. As this wheel file is intended to be deployed to setups beyond my control, solutions involving alternate OSes and/or libc replacement are not applicable.
Build configuration:
During build i use the following configuration/ command line:
export TF_NEED_CUDA=1
export TF_USE_XLA=0
export TF_SET_ANDROID_WORKSPACE=0
export TF_NEED_OPENCL_SYCL=0
export TF_NEED_ROCM=0
bazel build --config=opt --config=cuda --output_filter=DONT_MATCH_ANYTHING --linkopt=-L/usr/local/cuda/lib64 --linkopt=-lcudart --linkopt=-static-libstdc++ //tensorflow/tools/pip_package:build_pip_package
Regarding options used:
--output_filter=DONT_MATCH_ANYTHING : Silence warnings
--linkopt=-L/usr/local/cuda/lib64 --linkopt=-lcudart : Dynamic linking of cudart.so
--linkopt=-static-libstdc++ : Static link libstc++ as libstc++ also caused the libc error, this however is not possible for libm
I expected my own build to also work.
That expectation is (obviously) incorrect. The symbols your program or library requires from GLIBC depend on exactly which functions you call.
Consider the following program:
int main() { exit(0); }
When compiled/linked on a GLIBC-2.30 system, this program only depends on GLIBC_2.2.5 (because it doesn't call any newer symbols).
Now change the program slightly:
int main() { gettid(); exit(0); }
Compile/link it again, and all of a sudden this program now requires GLIBC_2.30 (because that's where gettid() was added to GLIBC), and will not work on any system which has older GLIBC.
So i assume there is some configuration option or docker configuration
Sure: your Docker image must have GLIBC that is not newer than what your target system have, i.e. GLIBC-2.17. Your current image contains GLIBC-2.27 (or newer).
You need a different Docker image, and you'll likely have to build it yourself, since GLIBC-2.17 is over 7 years old, and predates TensorFlow by many years.
Update:
What i don't understand is how come the pip tensorflow package (which i assumed was build with the docker image i am using) works with CentOS?
It works by accident, just like my first program would work on CentOS, but the second one wouldn't.
In short i wanted to generate a pip package that would work on 'any' linux/libc version
That is an impossible goal: Linux predates GLIBC, and it is impossible to build a single package that will work on a Linux distribution which didn't include GLIBC and on a distribution that did.
You have to draw a line somewhere. The developers of tensorflow-dev docker image drew a line at GLIBC-2.27. Packages built on this image should work on any system with 2.27 or later, and might (but are not at all guaranteed to) work on older systems.
just like the pip installed version.
You claim that the pip installed version has no "only GLIBC-xx or later" requirement, but that is not true. I am 99.9% sure that it requires at least GLIBC-2.14.
To find which GLIBC versions that package requires, run this command:
readelf -WV _pywrap_tensorflow_internal.so | grep GLIBC_
I assumed, the pip installed version was built using the publicly available tensorflow-devel docker image.
That is quite likely. And like I said, it happens to work on CentOS, but minute changes may make it not work anymore.
Update 2:
So running the readelf command as you suggested, does show the most recent required versions to be: - pip version: GLIBC_2.12 - mine : GLIBC_2.27 So from what i understand the pip version uses an older version even from CentOS, which explains why it works.
It doesn't "use" older version, it uses whatever version is available.
It requires a minimum version 2.12, while your build requires a minimum version 2.27.
How do they achieve this? Do they use a different image that has an older libc? If so, where can i get it? Or do they use the public image, but build with some bazel flag, that 'limits' symbols to the ones contained up to libc 2.12?
You are still not getting it.
The version that your program requires depends on exactly which functions you call. In my example program, if I only call exit, my program requires vesion 2.2.5, but if I also call gettid, then my program requires version 2.30. Note: these two programs are built on the same system with the same flags.
So no: they (most likely) didn't use a different Docker image, and didn't use "magic" bazel flags. They just happened to not call any functions which require GLIBC version > 2.12, and you did.
P.S. You can find which symbol(s) are causing "bad" dependency in your build like so:
readelf -Ws _pywrap_tensorflow_internal.so | egrep 'GLIBC_2.2[0-9]'
readelf -Ws _pywrap_tensorflow_internal.so | egrep 'GLIBC_2.1[89]'
This would produce output similar to (using my second program):
readelf -Ws a.out | egrep 'GLIBC_2.[23][0-9]'
2: 0000000000000000 0 FUNC GLOBAL DEFAULT UND gettid#GLIBC_2.30 (2)
48: 0000000000000000 0 FUNC GLOBAL DEFAULT UND gettid##GLIBC_2.30
The output above shows that the only symbol my binary requires from GLIBC 2.20 or above is gettid.
To make a counter point to what Employed Russian wrote:
The version that your program requires depends on exactly which functions you call. In my example program, if I only call exit, my program requires vesion 2.2.5, but if I also call gettid, then my program requires version 2.30. Note: these two programs are built on the same system with the same flags.
I don't think that's quite accurate. My understanding, which is corroborated by https://github.com/wheybags/glibc_version_header, is that things work like so (quoting that project, emphasis mine):
Glibc uses something called symbol versioning. This means that when you use e.g., malloc in your program, the symbol the linker will actually link against is malloc#GLIBC_YOUR_INSTALLED_VERSION (actually, it will link to malloc from the most recent version of glibc that changed the implementaton of malloc, but you get the idea).
So my guess (I have not checked) would be that the Tensorflow releases are built against an older glibc (perhaps by way of being built on an older release of their target Linux distro).
I am making a script that requires lmapm, but I'm not quite sure how to install it. I have 4 files,
lmapm.c
test.lua
README
Makefile
And I'm not sure how to use them in my lua environment. Lua 5.1 is installed on my desktop in a folder called "5.1", and it was installed with LuaRocks (If that matters) I know lua libraries are used with require, but this is a c file, not a lua file.
How can I install/use lmapm in my lua programs?
Upon reading the readme it tells me to run make, but makefile is just a "file" on my computer, there is nothing I can run it with.
README: Short description of what you got and how to install & use the module at the end.
test.lua: Lua script to test the module/sample of usage.
lmapm.c: C source code and the module in raw/still unusable form. Needs to get compiled and linked to a dynamic library of target platform.
Makefile: Automatic build instructions to compile&link lmapm.c to what you finally use in Lua.
Makefile serves as a macro which makes building easier with minimal input by users. To run this file, you need program make (comes with GNU toolchain; on Unix install package build-essential, on Windows MSYS). Before you have to fix the path to your Lua and MAPM installation (as mentioned in the official build instructions). Furthermore you need the C compiler and linker (which you already installed on Unix together with make and have to install on Windows by f.e. MinGW).
The result is a dynamic library/Lua C module which you can require simply by its filename. To put it in the scope of Lua, move it in the application or (better) in the Lua modules directory.
Is it possible to use Erlang with ZMQ on Windows? I have tried to use erlzmq2, but rebar fails to compile it with cryptic linker errors. Of course i can invest some time and investigate makefiles, but maybe other way exists?
Update
Whose who are interested in compilation errors can download latest erlang for windows and try to build erlzmq2 (Visual Studio 2012 compiler, msys sh and make). Error looks like:
cl : Command line error D8021 : invalid numeric argument '/Wl,-DLL,-IMPLIB:.libs
\zmq.dll.lib'
Makefile:541: recipe for target 'libzmq.la' failed
make[3]: *** [libzmq.la] Error 2
Please note that other erlang libs are compiling with this setup without any problems.
Your problem lies in compiling ZeroMQ for Windows. You haven't actually gotten to any Erlang yet. Here are some of the clues that tell you this:
Makefile:541: recipe for target 'libzmq.la' failed
This line says there's a problem on line 541 of the Makefile. But in erlzmq2, you can see that neither the main Makefile nor the c_src Makefile (which is what would build libzmq.la) has anything close to that many lines.
make3: * [libzmq.la] Error 2
The [3] means that you're 3 invocations deep into Make. Specifically, you started at the top-level Makefile, which called Rebar, which ran make -C c_src, which downloads ZeroMQ version 3.2.2 and tries to do a ./configure && make
To fix this Unix-style, go into the deps directory of erlzmq2 and figure out how to correctly compile ZeroMQ. Hopefully, you will just need to pass some arguments to configure. Then you can edit c_src/Makefile and set ZMQ_FLAGS to whatever you had to do for configure, clean, and make.
To fix it more Windows-style, follow the Windows build instructions for ZeroMQ. Put the compiled libzmq under deps and just edit the c_src Makefile to a no-op.
Finally, if you don't actually need to run this code on Windows, but are just using Windows as your development environment, I think you'll have the easiest time by running the build inside a Linux VM (not a hard thing at all with tools like Vagrant). Sorry, but Unix is the real system for this stuff; Windows support is an afterthought.
I wanted to integrate Elixir into our project, and the good old codes don't use rebar, so I think writing the rules for building .ex files into Emakefile may be a good idea, yet the man page here didn't mention anything relevant.
Edit:
Our team works mainly in Windows environment, but the deployment will be done on Linux servers, so I need a cross-platform solution. Since Erlang itself is cross-platform, I wanted to do it with erl -make command.
Of course I can write a Makefile, but then I'll need a build.bat or something alike to build the code in our developing environments, since we don't have make command on our dev' machines.
Anyone have a better idea?
Update:
In case anyone wants to know, I'm doing it this way:
Copy the lib/elixir directory in the Elixir source tree to our source dir, say some_project/src/tools/elixir.
Add some_project/src/tools/elixir/src/elixir_transform.erl and some_project/src/tools/elixir/src/* to the Emakefile, in that order. Set the output dir to some_project/ebin (All the other .beam files are located there).
Copy src/elixir.app.src in the Elixir source tree to some_project/ebin/elixir.app, and edit it to fix the version code.
Build the Erlang code by running erl -pa ebin -make, in some_project dir.
Build the Elixir compiler by running erl -pa ebin -s elixir_compiler core -s erlang halt
Now we have a working Elixir environment in our code, and I use the following escript to build our custom .ex files:
%%! -pa ./ebin
main(_) ->
ExList = [
<<"source_1.ex">>,
<<"source_2.ex">>,
<<"source_3.ex">>],
application:start(elixir),
gen_server:call(elixir_code_server, {compiler_options, [{docs, true}, {debug_info, true}]}),
[elixir_compiler:file_to_path(F, <<"./ebin">>) || F <- ExList],
erlang:halt(0).
If you want to explicitly compile Elixir, I would go with the Makefile approach since it will always be supported by Elixir. However, I would recommend the precompiled binaries or even assume Elixir is installed in each developer machine. You can even add a task to your Emakefile to guarantee everyone is using the proper Elixir version.
Finally, about compiling your own Elixir code, I would recommend simply using mix. mix is a binary that ships with Elixir and you can simply do ./src/tools/elixir/bin/mix compile from your Emakefile.
In case using mix is not possible, you should use the parallel compiler, since it will compile files using all cores available in your machine and it will automatically detect and solve dependency in between files. Here is an example of calling the parallel compiler from erlang:
https://github.com/basho/rebar/pull/347/files#L1R62
The API is very simple. It expects a list of file names to compile as binary and the directory to output files to as another binary.