How to set up panther crawler with docker - docker

Dockerfile:
# https://github.com/symfony/panther#docker-integration
FROM php:latest
RUN apt-get update && apt-get install -y libzip-dev zlib1g-dev chromium && docker-php-ext-install zip
ENV PANTHER_NO_SANDBOX 1
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
docker-compose.yml:
version: "3"
services:
crawler:
build: .
working_dir: /usr/src
volumes:
- .:/usr/src
command: /bin/sh -c "/usr/local/bin/composer install && php index.php"
composer.json:
{
"require": {
"symfony/panther": "^0.6.0"
}
}
index.php:
<?php
// https://github.com/symfony/panther#basic-usage
require __DIR__.'/vendor/autoload.php'; // Composer's autoloader
$client = \Symfony\Component\Panther\Client::createChromeClient();
$client->request('GET', 'https://api-platform.com'); // Yes, this website is 100% written in JavaScript
$client->clickLink('Support');
// Wait for an element to be rendered
$crawler = $client->waitFor('.support');
echo $crawler->filter('.support')->text();
$client->takeScreenshot('screen.png'); // Yeah, screenshot!
All files are in the same location. I run docker-compose build && docker-compose up and I'm getting following error:
crawler_1 | Fatal error: Uncaught RuntimeException: Could not start chrome (or it crashed) after 30 seconds. in /usr/src/vendor/symfony/panther/src/ProcessManager/WebServerReadinessProbeTrait.php:51
This is similar to https://github.com/symfony/panther/issues/200, however in my case I'm not using panther for tests, but only to scrape, and I really don't know how to fix this. I think my problem might be related to invalid docker / docker-compose files.

I had the same error. My solution was install unzip as it says in the readme:
"Warning: On *nix systems, the unzip command must be installed or you
will encounter an error similar to RuntimeException: sh: 1: exec:
/app/vendor/symfony/panther/src/ProcessManager/../../chromedriver-bin/chromedriver_linux64:
Permission denied (or chromedriver_linux64: not found). The underlying
reason is that PHP's ZipArchive doesn't preserve UNIX executable
permissions."
and finally, reinstall the panther library.

Related

How to create CakePHP project (previous version) using Composer (using an old version of PHP, in 2023)

TL;DR Technology moving on and what was deprecated is obsolete now. But fear not, Docker here to solve the problem. See the answer below to see how I create a CakePHP 3.6 project on macOS Ventura.
Here, in 2023, I have my Mac updated to Ventura (13.0.1) with brew installed (Homebrew 3.6.18) and I have this requirement to create a CakePHP project, specifically version 3.6 (I am very well aware of CakePHP 3.x End of Support - but life happens). Now the scene was set, let's go and rock-and-roll!
Heading to installation documentation yields these;
Install PHP - per documentation PHP 7.4 is supported, yay (sort of)
Install Composer - there is at least one another alternative, the Oven way, but I haven't tried it
Create a CakePHP Project
OK, let me install PHP. Oh, I already PHP installed via brew, PHP 8.2.1. Good then let me check if I already have Composer installed as well, composer --version; yes "Composer version 2.4.4 2022-10-27 14:39:29". So the last thing I need to do is to Create a CakePHP Project. Let me run composer create-project --prefer-dist cakephp/app:"^3.6" my_app_name;
Creating a "cakephp/app:^3.6" project at "./my_app_name"
Info from https://repo.packagist.org: #StandWithUkraine
Installing cakephp/app (3.10.1)
- Installing cakephp/app (3.10.1): Extracting archive
Created project in /Users/johndoe/Code/my_app_name
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.
Problem 1
- cakephp/cakephp[3.10.0, ..., 3.10.5] require php >=5.6.0,<8.0.0 -> your php version (8.2.1) does not satisfy that requirement.
- Root composer.json requires cakephp/cakephp 3.10.* -> satisfiable by cakephp/cakephp[3.10.0, ..., 3.10.5].
Oh, noes đŸ˜”. Well, let me downgrade my PHP version, or better yet let brew install PHP 7.4 side-by-side (actually it was not a "better yet"). Quick Googling yield Update PHP to 7.4 macOS Catalina with brew on SO. Hmm, I'm on Venture and this is for Catalina. But there is one comment;
This solution works perfectly in MacOS BigSur.
-juanitourquiza
I took juanitourquiza's word for it, besides there's nothing to lose... Except for those irritating "libicuio.71.dylib no such file" errors. It turned out that "Xcode 7.1 changed the name of some libraries now it uses .tdb files.". Bummer!
There I was scratching my head, I thought to myself "well I'm already going to use Docker to serve the app anyway (locally), why not use Docker to create the project too?".
First and foremost you have to have Docker installed, and how is another story.
Create a CakePHP Project - Take 2
Next this little command would suffice to create a CakePHP 3.6 project;
docker run -it --rm \
--name php74_composer \
-v "$PWD":/usr/src/myapp \
-w /usr/src/myapp \
php:7.4-cli sh -c "pwd; apt-get update && apt-get install -y unzip; \
curl https://raw.githubusercontent.com/composer/getcomposer.org/76a7060ccb93902cd7576b67264ad91c8a2700e2/web/installer | php; \
php composer.phar create-project --no-install --no-interaction --no-scripts --prefer-dist cakephp/app:\"3.6.5\" my_app_name; \
cd my_app_name; \
php ../composer.phar config --no-plugins allow-plugins.cakephp/plugin-installer true; \
php ../composer.phar install --ignore-platform-reqs; \
php ../composer.phar run --no-interaction post-install-cmd; \
cd ../ && rm composer.phar; \
exit"
Although it's highly opinionated about the versions (PHP 7.4 and CakePHP 3.6.5) it does the trick! When you run it, it'll create a directory called "my_app_name" on the current working directory. After the container exit out you may move this directory anywhere as your heart desire.
Serve the App
As I mentioned I am going to use Docker as well to serve the app (locally). There are trillion tutorials out there showing how to do it, nonetheless here's my solution;
# In the "my_app_name" directory
mkdir docker && cd docker
echo "FROM php:7.4-fpm
RUN apt-get update && apt-get install -y \\
libicu-dev \\
git \\
curl \\
zip \\
unzip
RUN docker-php-ext-configure intl
RUN docker-php-ext-install intl
RUN docker-php-ext-enable intl
RUN docker-php-ext-configure pdo_mysql
RUN docker-php-ext-install pdo_mysql
RUN docker-php-ext-enable pdo_mysql
RUN curl https://raw.githubusercontent.com/composer/getcomposer.org/76a7060ccb93902cd7576b67264ad91c8a2700e2/web/installer | php && \\
mv composer.phar /usr/local/bin/composer
WORKDIR /var/www" > Dockerfile
echo "version: '3.8'
services:
app:
build:
context: ./
dockerfile: Dockerfile
container_name: johndoes-app
restart: always
working_dir: /var/www
volumes:
- ../:/var/www
nginx:
image: nginx:1.23-alpine
container_name: johndoes-nginx
restart: always
ports:
- 8000:80
volumes:
- ../:/var/www
- ./nginx:/etc/nginx/conf.d
" > docker-compose.yml
mkdir nginx && cd nginx
echo "server {
listen 80;
index index.php;
error_log /var/log/nginx/error.log;
access_log /var/log/nginx/access.log;
error_page 404 /index.php;
root /var/www/webroot;
location ~ \.php {
try_files \$uri =404;
fastcgi_pass app:9000;
fastcgi_index index.php;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME \$document_root\$fastcgi_script_name;
}
location / {
try_files \$uri \$uri/ /index.php?\$query_string;
}
}
" > nginx.conf
And then at "my_app_name/docker" run docker-compose up. After services started go to "http://localhost:8000".
Now you may proceed with configuring your app. Cheers.
Update for Xdebug
If you wish to debug your app via VSCode / VSCodium running on your host machine (Mac in my case);
Append the following to Dockerfile (in between existing lines denoted);
# existing RUN docker-php-ext-enable pdo_mysql
RUN pecl install xdebug-3.1.6 && docker-php-ext-enable xdebug
# existing RUN curl https://raw.githubusercontent.com/composer/getcomposer.org/...
Append the following to docker-compose.yml (in between existing lines denoted);
# for "app" service
# existing - ../:/var/www
- ./99-xdebug.ini:/usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini
# mind indentation since this is a YAML file, i.e.;
# dashes must be on the same level vertically
Create the INI file for Xdebug;
# at the docker directory
echo "zend_extension = xdebug
xdebug.mode = debug
xdebug.start_with_request = yes
xdebug.client_host = host.docker.internal" > 99-xdebug.ini
Install the PHP Debug extension for VSCode or for VSCodium.
As the final step have a .vscode/launch.json file including these;
{
"name": "Listen for Xdebug",
"type": "php",
"request": "launch",
"port": 9003,
"pathMappings": {
"/var/www": "${workspaceFolder}"
}
}
"pathMappings" is the crucial setting here, others were automatically generated.

Launching Docker container with Ansible gives "command not found"

I'm trying to create a container from within a playbook that would execute some terraform code, but as soon as the container is created the first command I ask of it apt update gives me the error message on exit /bin/bash: line 1: apt update && apt install -y build-essential && bundle install && terraspace check_setup && terraspace up rds -y: command not found.
The weirdest part, is that it used to work well, but since we updated from ansible 2.10.X to ansible 2.12.8 this error appeared.
Here's a the code use to create the container:
- name: Launch Docker container with Terraspace image
register: container_terraspace_apply_result
community.docker.docker_container:
name: "terraspace-test"
image: a-private-ubuntu-base-terraform-image
working_dir: /opt/terraform
volumes:
- "/home/terraform:/opt/terraform:rw"
- "/home/.aws:/root/.aws:ro"
command_handling: correct
entrypoint: ["/bin/bash"]
command: [
"-l",
"-c",
"'apt update && apt install -y build-essential && bundle install && terraspace check_setup && terraspace up rds -y'"
]
env:
REGION: "eu-west-1"
AWS_REGION: "eu-west-1"
TS_ENV: "staging"
AWS_PROFILE: "terraform-staging"
I found out that if I only do "'apt'" I do get the aptr help output you would get when typing it with no arguments, so the binary path is good, the only problems seems to be the space after it ???
Is there something I'm missing here ? I've been looking around and couldn't find a solution to my issue.
Thank you !
So it turned out it was the command_handling option that needed to be set to compatibility !

Containerized Azure Function does not see databricks odbc driver

I have below Dockerfile:
FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS installer-env
COPY . /src/dotnet-function-app
RUN cd /src/dotnet-function-app && \
mkdir -p /home/site/wwwroot && \
dotnet publish *.csproj --output /home/site/wwwroot
FROM mcr.microsoft.com/azure-functions/dotnet:4
ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
AzureFunctionsJobHost__Logging__Console__IsEnabled=true
#ODBCINI=/etc/odbc.in \
#ODBCSYSINI=/etc/odbcinst.ini \
#SIMBASPARKINI=/opt/simba/spark/lib/64/simba.sparkodbc.ini
WORKDIR ./home/site/wwwroot
COPY --from=installer-env /home/site/wwwroot /home/site/wwwroot
RUN apt update && apt install -y apt-utils odbcinst1debian2 libodbc1 odbcinst vim unixodbc unixodbc-dev freetds-dev curl tdsodbc unzip libsasl2-modules-gssapi-mit
RUN curl -sL https://databricks.com/wp-content/uploads/drivers-2020/SimbaSparkODBC-2.6.16.1019-Debian-64bit.zip -o databricksOdbc.zip && unzip databricksOdbc.zip
RUN dpkg -i SimbaSparkODBC-2.6.16.1019-Debian-64bit/simbaspark_2.6.16.1019-2_amd64.deb
RUN export ODBCINI=/etc/odbc.ini ODBCSYSINI=/etc/odbcinst.ini SIMBASPARKINI=/opt/simba/spark/lib/64/simba.sparkodbc.ini
Purpose why this azure function app is containerized is to enable using databricks odbc driver to connect to azure databricks instance and delta lake. I have read on stackoverflow, in other thread, that there is no other way for installing custom drivers if app service is not containerized. I thought it should work the same with function app if is containerized.
Unfortunatelly I get exception that:
ERROR [01000] [unixODBC][Driver Manager]Can't open lib 'Simba Spark ODBC Driver' : file not found
or
Dependency unixODBC with minimum version 2.3.1 is required. Unable to load shared library 'libodbc.so.2' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: liblibodbc.so.2: cannot open shared object file: No such file or directory
even if I point drivers in this line:
RUN export ODBCINI=/etc/odbc.ini ODBCSYSINI=/etc/odbcinst.ini SIMBASPARKINI=/opt/simba/spark/lib/64/simba.sparkodbc.ini
Looks like home/site/wwwroot can not access above folders. What interessting I also tried to copy content of /etc to /home/site/wwwroot/bin to set enrironment variable pointing from that folder, but it is not possible to copy:
WORKDIR /etc
COPY . /home/site/wwwroot/bin
Generally, I pass connection details to databricks instance in connection string, but I also tried to point /etc files by below command:
RUN gawk -i inplace '{ print } ENDFILE { print "[ODBC Drivers]" }' /etc/odbcinst.ini
but I get exception during building that:
gawk: inplace:59: warning: inplace::begin: Cannot stat '/etc/odbcinst.ini' (No such file or directory)

Rootless docker-compose cannot build timescale image

I have installed docker rootless on an ubuntu host machine. I have a Dockerfile for building timescaledb with the most important part looking like that:
# Install the tools we need for installation
RUN apt-get update && apt-get -y install gnupg2 lsb-release wget
# Add Postgres and Timescale package repository
RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -c -s)-pgdg main" | tee /etc/apt/sources.list.d/pgdg.list
RUN wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
RUN sh -c "echo 'deb https://packagecloud.io/timescale/timescaledb/debian/ `lsb_release -c -s` main' > /etc/apt/sources.list.d/timescaledb.list"
RUN wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | apt-key add -
# Install Timescale
RUN apt-get update && apt-get -y install timescaledb-2-postgresql-12=2.0.0-zz~debian10
the corresponding docker-compose file looks like this:
timescale:
tty: true
volumes:
- timescale-volume:/var/lib/postgresql/data:rw
build:
context: ./timescale
dockerfile: Dockerfile
command:
- /bin/bash
depends_on:
- cert-mounter
When I run docker-compose up with sudo it works fine, the image is built and the container is running. If I execute it rootless I get the following error:
dpkg: error processing package postgresql-12 (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of timescaledb-2-postgresql-12:
timescaledb-2-postgresql-12 depends on postgresql-12; however:
Package postgresql-12 is not configured yet.
dpkg: error processing package timescaledb-2-postgresql-12 (--configure):
dependency problems - leaving unconfigured
Setting up exim4-daemon-light (4.92-8+deb10u5) ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Initializing GnuTLS DH parameter file
Setting up libmailutils5:amd64 (1:3.5-4) ...
Setting up mailutils (1:3.5-4) ...
update-alternatives: using /usr/bin/frm.mailutils to provide /usr/bin/frm (frm) in auto mode
update-alternatives: using /usr/bin/from.mailutils to provide /usr/bin/from (from) in auto mode
update-alternatives: using /usr/bin/messages.mailutils to provide /usr/bin/messages (messages) in auto mode
update-alternatives: using /usr/bin/movemail.mailutils to provide /usr/bin/movemail (movemail) in auto mode
update-alternatives: using /usr/bin/readmsg.mailutils to provide /usr/bin/readmsg (readmsg) in auto mode
update-alternatives: using /usr/bin/dotlock.mailutils to provide /usr/bin/dotlock (dotlock) in auto mode
update-alternatives: using /usr/bin/mail.mailutils to provide /usr/bin/mailx (mailx) in auto mode
dpkg: dependency problems prevent configuration of timescaledb-2-loader-postgresql-12:
timescaledb-2-loader-postgresql-12 depends on postgresql-12; however:
Package postgresql-12 is not configured yet.
dpkg: error processing package timescaledb-2-loader-postgresql-12 (--configure):
dependency problems - leaving unconfigured
Processing triggers for libc-bin (2.28-10) ...
Processing triggers for mime-support (3.62) ...
Errors were encountered while processing:
postgresql-common
postgresql-12
timescaledb-2-postgresql-12
timescaledb-2-loader-postgresql-12
E: Sub-process /usr/bin/dpkg returned an error code (1)
The command '/bin/sh -c apt-get update && apt-get -y install timescaledb-2-postgresql-12=2.0.0-zz~debian10' returned a non-zero code: 100
ERROR: Service 'timescale' failed to build
What could be the problem? Other containers are somehow built and run rootless without problems...
So I managed to make it work. In my Dockerfile I also set the uid of a user because I share some volumes and want the UIDs of users be consistent between the containers. So on top of my Dockerfile I had the following:
RUN useradd --uid 80000 postgres
replacing the uid with the lower value solved the issue
RUN useradd --uid 18000 postgres

How do I properly use Silex 2, Doctrine, and PDO via docker containers?

So I am attempting to build a Silex 2 application; using docker service containerization as the ENV control system. However, when accessing the resource the return is a PDO Exception 'driver not found'.
docker-compose.yml
# program code
code:
build: docker/code
# env_file: .env
ports:
- "80:8080"
volumes:
- ./code:/code
# database
db:
image: mysql:latest
volumes:
- /var/lib/mysql
ports:
- "3306:3306"
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: app_db
MYSQL_USER: app_db_user
MYSQL_PASSWORD: app_db_pass
code Dockerfile
FROM php:7
WORKDIR /code
# install curl and php
RUN apt-get update -y
RUN apt-get install git zip -y
# install composer & app deps; then remove
RUN curl -sS https://getcomposer.org/installer | php
RUN mv composer.phar /usr/local/bin/composer
EXPOSE 80
CMD ["php", "-S", "0.0.0.0:80"]
server logic
<?php
# silex micro framework basic entry script
// web/index.php
require_once __DIR__.'/vendor/autoload.php';
require_once __DIR__.'/common/env.php';
$app = new Silex\Application();
$app['debug'] = true;
$app->register(new Silex\Provider\DoctrineServiceProvider(), [
'db.options' => [
'driver' => 'pdo_mysql',
'dbname' => 'app_db',
'host' => '172.17.0.2',
'user' => 'app_db_user',
'password' => 'app_db_pass',
'charset' => 'utf8mb4',
'port' => '3306',
]
]);
$app->get('/', function () use ($app) {
$sql = 'SELECT * FROM test WHERE id = 1';
$data = $app['db']->fetchAssoc($sql);
return json_encode($data);
});
$app->run();
The Error:
Whoops, looks like something went wrong.
3/3
DriverException in AbstractMySQLDriver.php line 115:
An exception occured in driver: could not find driver
2/3
PDOException in PDOConnection.php line 47:
could not find driver
1/3
PDOException in PDOConnection.php line 43:
could not find driver
Tried using mariaDB database image, check php.ini and the mysqlnd PDO driver is installed:
mysqlnd
mysqlnd enabled
Version mysqlnd 5.0.12-dev - 20150407 - $Id: d8daadaf41e3cd81d7c6ae96c6091fd15b2c9382 $
Compression supported
core SSL supported
extended SSL supported
Command buffer size 4096
Read buffer size 32768
Read timeout 31536000
Collecting statistics Yes
Collecting memory statistics No
Tracing n/a
Loaded plugins mysqlnd,debug_trace,auth_plugin_mysql_native_password,auth_plugin_mysql_clear_password,auth_plugin_sha256_password
API Extensions no value
php-m inside of the _server_ container
~
"php.ini" [New] 1L, 23C written
php -m
[PHP Modules]
Core
ctype
curl
date
dom
fileinfo
filter
ftp
hash
iconv
json
libxml
mbstring
mysqlnd
openssl
pcre
PDO
pdo_sqlite
Phar
posix
readline
Reflection
session
SimpleXML
SPL
sqlite3
standard
tokenizer
xml
xmlreader
xmlwriter
zlib
As a side note I'd like to keep the server service as abstracted as possible as the server may be one of any number of options (built in, apache, nginx, etc).
I has to be something simple Im over looking, but what?
Fixed it:
FROM php:latest
# install curl and php
RUN apt-get update -y
RUN apt-get install curl git zip -y
# call PHP images script `docker-php-ext-install` and install language extensions
RUN docker-php-ext-install pdo pdo_mysql
# install composer & app deps; then remove
RUN curl -sS https://getcomposer.org/installer | php
RUN mv composer.phar /usr/local/bin/composer
# change working dir
WORKDIR /code
CMD ["php", "-S", "0.0.0.0:80"]
Noticed RUN docker-php-ext-install pdo pdo_mysql. Apparently installing php extensions in the php docker container requires using the built in docker-php-ext-install and docker-php-ext-configure scripts that are provided with the machine.

Resources