Apache-beam has been frustrating to manage with the correct google-cloud libraries for me to use with Dataflow.
I discovered for what I'm doing I need apache-beam=2.3.0 rather than 2.4.0 (2.4.0 gives a pickling error that I cannot resolve refer to Dataflow Error: 'Clients have non-trivial state that is local and unpickleable')
I need DataflowRunner to use apache-beam=2.3.0 as well so following this persons instructions Custom Apache Beam Python version in Dataflow I just need the actual tar.gz file I thought I had installed it via pip with a pip install apache-beam=2.3.0 so if I look in my system I can't find any tar.gz. When I go to the apache website to download the source code the link is broken.
Where can I find a tar.gz for apache-beam-2.3.0?
The latest and all history releases of apache-beam packages can be found on github - github.com/apache/beam/releases.
Related
I'm trying to deploy an asp.net core 3.1 API on cloud foundry. I don't have admin rights, i just have developer rights. Is there a way to specify the URL of these libraries (libc6-dev, libgdiplus and libx11-dev) (maybe git or some official repository) so that i can execute the manifest.yml file during deployment and install these dependencies? Also to mention, i cannot turn on support for docker file on cloud foundry, as i get a message (insufficient rights)
I would suggest you give the apt-buildpack a try. You can give it additional Ubuntu package names, and it will install those for you.
You do that through an apt.yml file. Check out this post for instructions.
It's important to understand that the apt-buildpack will install these packages into a non-standard location. Since it also runs as a non-root user, it cannot install them into standard locations.
To work around this limitation, it sets variables like $PATH and $LD_LIBRARY_PATH to point to the locations where it has installed items. Most build tools will pick up these env variables and be able to locate what you install.
It's not perfect though, and some tools require additional env variables to be set. If you still get errors when building, look at your build tools and check if there are ways you can point to where apt-buildpack is installing stuff. The path it writes to can vary based on your buildpack order, but if you print out $PATH you can see the location. It's often /home/vcap/deps/0/... but the index can change based on your buildpack order.
I am trying to install google-cloud-bigquery==1.5.0 PyPI packages on GCP composer for a new environment recently has been created. I get this error:
Successfully installed google-cloud-bigquery-1.5.0 google-cloud-core-0.28.1 pypd-1.1.0 strict-rfc3339-0.7
+ [[ -z fail ]]
+ python3 -m pipdeptree --warn fail
Warning!!! Possibly conflicting dependencies found:
* google-cloud-translate==2.0.1
- google-cloud-core [required: >=1.1.0,<2.0dev, installed: 0.28.1]
* google-cloud-storage==1.29.0
I tried another version (2.2.0) and it had conflicts with some other google pre-installed packages.
The new environment image version is composer-1.12.2-airflow-1.10.6.
There is another environment created few months ago and all pypi packages are installed successfully and airflow dags are running smoothly, the image version for it is composer-1.10.0-airflow-1.10.6 .
Question 1: I think the current issue is linked to the image version and probably I have to recreate the new environment with older image version, am I correct?
Question 2: To create new environment I have only three options for image version which are composer-1.12.2-airflow-1.10.6, composer-1.12.2-airflow-1.10.9, composer-1.12.2-airflow-1.10.10. How I can create environment with image version composer-1.10.0 ? We have several other projects and environment with the same location/zone but with composer-1.10.0
Please, have a look to the official documentation for the Apache Airflow and Python versions that Cloud Composer supports. You can refer to the section for the Python packages which comes with composer-1.12.2-airflow-1.10.6. A specific Cloud Composer Airflow version already comes with a set of included packages. When you upgrade/downgrade a specific installed PyPi package or try to use any other packages, you can run into conflicts.
As for now, there is no way to check conflicts ahead of time within Cloud Composer environment. I would like to suggest you to adjust your packages to be able to use in the one of the Composer environments: composer-1.12.4-airflow-1.10.10,
composer-1.12.4-airflow-1.10.9 or composer-1.12.4-airflow-1.10.6.
I am trying to run some old matlab code with octave. Unfortunately this code contains a geotiffread function and I think I should change this function with rasterread (package mapping).
However, when I try to install the mapping package I get this warning:
octave:7> pkg install mapping-1.4.0.tar.gz
configure: WARNING: GDAL library not found. Reading of raster files will be disabled.
For information about changes from previous versions of the mapping package, run 'news mapping'.
I tried to run octave (5.2.0 version) within:
a Debian Buster distribution (snap and flatpak package)
a docker container (MacOS 10.15 host, installed from the mtmiller/octave image).
online with the octave-online service, running this code:
pkg load mapping;
[bands, info] = rasterread ('mexutm250.tiff');
With this output:
octave:3> source("my_script.m")
error: gdalread: reading of raster file with GDAL was disabled during installation
error: called from
rasterread at line 56 column 26
my_script at line 2 column 15
No attempt was successful.
EDIT 2: I know that my octave installations are without GDAL support. I would like to use octave with full mapping package, and GDAL support, without recompile it. There is a way to do it (e.g. update a library path within the docker installation to add the libgdal library)?
If there is no way to add GDAL support without recompile octave, there is a guide to do it with minimal effort?
EDIT 3: I already installed the gdal dependencies:
$ sudo aptitude search gdal |grep ^i
[sudo] password for virtuser:
i gdal-bin - Geospatial Data Abstraction Library - programmi di utilitÃ
i A gdal-data - libreria Geospatial Data Abstraction Library - file di dati
i libgdal-dev - libreria Geospatial Data Abstraction Library - file di sviluppo
i libgdal20 - libreria Geospatial Data Abstraction Library
Thank you.
I got octave with GDAL integration when I installed the octave package from the debian repository. I needed octave 5.2, so I switched to Ubuntu 20.04.
as suggested in one of the comments, checking
>> news mapping
(also at https://octave.sourceforge.io/mapping/NEWS.html)
looking at mapping 1.2.1 where rasterread was introduced, it states:
** New features
Reading GIS raster data: A first go is provided using
functions rasterread.m and rasterinfo.m. Both invoke binary
function gdalread() of which an initial version was provided
by Shashank Khare. rasterread.m and rasterinfo.m can read
and return info on any raster data type that the underlying
GDAL library can read. As such, separate functions for e.g.,
GeoTIFF and ArcGrid etc. are not required.
To make use of these functions the GDAL library must be
present on your system => GDAL is a suggested dependency.
You should be able to install the GDAL library in Debian using your preferred installation method.
Unsure whether or not you'll need to uninstall/reinstall the mapping package afterward, but if an unload/reload doesn't get rid of the message, try that and see if mapping is able to see the library.
I have a Rails project that is going to be using OpenCV, and it depends on a certain version of it (2.4.6.1).
I'm looking for deployment advice. The Ubuntu opencv package is an earlier version and therefore not suitable.
I can see a number of possibilities, but I'm trying to think of what will work best.
Just write it up in a README and expect people to follow it: download this, apt-get that, etc...
Add opencv, tagged at the version we need, as a git subtree, and include a Rake task to build it.
Write a script to download and compile the needed code.
Something else ?
None of them seem all that great, to tell the truth.
Can your application be made to work with OpenCV 2.4.2? That is available in Ubuntu 13.04, and you could request it be backported to 12.04. If not, you could update the source package to 2.4.6.1 (which would require learning about debian packaging but might not be too difficult since you would be modifying an existing package instead of starting from scratch), upload it to a PPA, and instruct your users on Ubuntu to install OpenCV from there. You could also package your rails application and put it in the PPA, which would make overall installation even easier.
Do you use CEAN, copy the source and compile them, copy the BEAM files, or something else. I need to distribute some Erlang code and I'm not sure which to choose.
I've been working on EPM, an Erlang package manager. It pulls from GitHub. It's non-invasive and doesn't require installing anything on your system other than downloading an escript. It works as follows:
jvorreuter$ ./epm install ibrowse mochiweb
epm v0.1.0, 2010
===============================
Install the following packages?
===============================
+ epm-mochiweb-master
+ cmullaparthi-ibrowse-master
([y]/n) y
+ downloading http://github.com/epm/mochiweb/tarball/master
+ running mochiweb build command
+ running mochiweb install command
+ downloading http://github.com/cmullaparthi/ibrowse/tarball/master
+ running ibrowse build command
+ running ibrowse install command
You can read more about it at http://www.jkvor.com/erlang-package-manager
The linked blog post is blank as of August 2013. The GitHub repository is at https://github.com/JacobVorreuter/epm
I clone the git repository, build it, and add the directory to my ERL_LIBS path. I hack the source for my private customizations. For those commits that are sensible, I publish.
I am a much happier developer after finding git and being able to manage my own changes and still be able to get upstream changes that I can rebase my stuff on.
I realize that this looks raw for end users, but I am my own end user.
If I would ship something to other end users I would look into using .ez zip archive files that the erlang code loader can use. See section "Loading of Code From Archive Files" on that page. Then provide a script that invokes erl with the correct arguments.
If the repository isn't available as a git, I git-svn clone it. If I can't do that, I tend to stay away from it.
I use faxien (a package manager for Erlang releases and applications) from the Erlware project: http://www.erlware.org. It and sinan are essential tools for Erlang development :).
I use Agner on Mac, Linux, and Unix; and CEAN on Windows. CEAN's the only Erlang package manager with Windows support.
I package them on a Debian repository on Launchpad.