Plugin: https://github.com/fluent/fluent-plugin-sql
I tried with MySQL and it's working perfectly fine. But with snowflake, it's giving error:
[error]: #0 Could not load the 'snowflake' Active Record adapter.
Ensure that the adapter is spelled correctly in config/database.yml
and that you've added the necessary adapter gem to your Gemfile.
It seems there's something wrong with the adapter/adapter name.
I'm using docker for fluentd.
Dockerfile:
FROM fluent/fluentd:v1.15.3-debian-1.0
USER root
RUN ["fluent-gem", "install", "fluent-plugin-sql"]
RUN apt-get update && \
apt-get -y install make && \
apt-get -y install unixodbc-dev
RUN ["fluent-gem", "install", "sequel-snowflake"]
USER fluent
Conf:
<match>
#type sql
adapter snowflake
host my_org.snowflakecomputing.com
port 443
database PROD
username <username>
password <password>
<table>
table <schema>.EVENTS_TEST
column_mapping 'name:name,value:value,labels:labels,type:type'
</table>
</match>
I've also tried adapter :snowflake as mentioned in https://github.com/Yesware/sequel-snowflake
Related
I created a custom image with the following Dockerfile:
FROM apache/airflow:2.1.1-python3.8
USER root
RUN apt-get update \
&& apt-get -y install gcc gnupg2 \
&& curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \
&& curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get update \
&& ACCEPT_EULA=Y apt-get -y install msodbcsql17 \
&& ACCEPT_EULA=Y apt-get -y install mssql-tools
RUN echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc \
&& echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc \
&& source ~/.bashrc
RUN apt-get -y install unixodbc-dev \
&& apt-get -y install python-pip \
&& pip install pyodbc
RUN echo -e “AIRFLOW_UID=$(id -u) \nAIRFLOW_GID=0” > .env
USER airflow
The image creates successfully, but when I try to run it, I get this error:
"airflow command error: the following arguments are required: GROUP_OR_COMMAND, see help above."
I have tried supplying a group ID with the --user, but I can't figure it out.
How can I start this custom Airflow Docker image?
Thanks!
First of all this line is wrong:
RUN echo -e “AIRFLOW_UID=$(id -u) \nAIRFLOW_GID=0” > .env
If you are running it with Docker Compose (I presume you took it from https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html), this is something you should run on "Host" machine, not in the image. Remove that line, it has no effect.
Secondly - it really depends what "command" you run. The "GROUP_OR_COMMAND" message you got is the output of "airflow" command. You have not copied the whole output of your command but this is a message you get when you try to run airflow without telling it what to do. When you run the image you will run by default the airflow command which has a number of subcommands that can be executed. So the "see help above" message tells you the very thing you should do - look at the help and see what subcommand you wanted to run (and possibly run it).
docker run -it apache/airflow:2.1.2
usage: airflow [-h] GROUP_OR_COMMAND ...
positional arguments:
GROUP_OR_COMMAND
Groups:
celery Celery components
config View configuration
connections Manage connections
dags Manage DAGs
db Database operations
jobs Manage jobs
kubernetes Tools to help run the KubernetesExecutor
pools Manage pools
providers Display providers
roles Manage roles
tasks Manage tasks
users Manage users
variables Manage variables
Commands:
cheat-sheet Display cheat sheet
info Show information about current Airflow and environment
kerberos Start a kerberos ticket renewer
plugins Dump information about loaded plugins
rotate-fernet-key
Rotate encrypted connection credentials and variables
scheduler Start a scheduler instance
sync-perm Update permissions for existing roles and optionally DAGs
version Show the version
webserver Start a Airflow webserver instance
optional arguments:
-h, --help show this help message and exit
airflow command error: the following arguments are required: GROUP_OR_COMMAND, see help above.
when you extend the official image, it will pass the parametor to "airflow" command which causing this problem. Check this out: https://airflow.apache.org/docs/docker-stack/entrypoint.html#entrypoint-commands
I have CircleCI pipeline setup for my test flow using Jest snapshot and one of my snapshot tests keeps failing. I use Javascript to generate a Date object (new Date("YYYY-MM-DD")) and locally it yields MM/DD/YYYY but in the docker image (node:8) it yields YYYY-MM-DD instead so the snapshot test fails. I have tried to set up locales by:
docker:
- image: circleci/node:8
environment:
TZ: "America/Los_Angeles"
LANG: en_US.UTF-8
LANGUAGE: en_US.UTF-8
LC_ALL: en_US.UTF-8
But it complains it cannot set the default locale so I added:
- run:
name: Reconfigure Locale
command: sudo dpkg-reconfigure locales
which seemed to be a solution for most of the people that had the same problem but not my case.
I also tried to have the same local docker image and test it there and it worked fine with these commands:
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y locales
RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
dpkg-reconfigure --frontend=noninteractive locales && \
update-locale LANG=en_US.UTF-8
So I tried these in circleci and sed commands complained about permissions even when it is called with sudo.
Okay just FYI, it was the node version caused the date format issue. I installed full-icu npm package which handles locale for the node application. To re-visit my problem, I had successfully installed locale and set it to be the same as the local machine but Node won't pick the locale from the system but from the browser being used. I hope this info helps.
So I am attempting to build a Silex 2 application; using docker service containerization as the ENV control system. However, when accessing the resource the return is a PDO Exception 'driver not found'.
docker-compose.yml
# program code
code:
build: docker/code
# env_file: .env
ports:
- "80:8080"
volumes:
- ./code:/code
# database
db:
image: mysql:latest
volumes:
- /var/lib/mysql
ports:
- "3306:3306"
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: app_db
MYSQL_USER: app_db_user
MYSQL_PASSWORD: app_db_pass
code Dockerfile
FROM php:7
WORKDIR /code
# install curl and php
RUN apt-get update -y
RUN apt-get install git zip -y
# install composer & app deps; then remove
RUN curl -sS https://getcomposer.org/installer | php
RUN mv composer.phar /usr/local/bin/composer
EXPOSE 80
CMD ["php", "-S", "0.0.0.0:80"]
server logic
<?php
# silex micro framework basic entry script
// web/index.php
require_once __DIR__.'/vendor/autoload.php';
require_once __DIR__.'/common/env.php';
$app = new Silex\Application();
$app['debug'] = true;
$app->register(new Silex\Provider\DoctrineServiceProvider(), [
'db.options' => [
'driver' => 'pdo_mysql',
'dbname' => 'app_db',
'host' => '172.17.0.2',
'user' => 'app_db_user',
'password' => 'app_db_pass',
'charset' => 'utf8mb4',
'port' => '3306',
]
]);
$app->get('/', function () use ($app) {
$sql = 'SELECT * FROM test WHERE id = 1';
$data = $app['db']->fetchAssoc($sql);
return json_encode($data);
});
$app->run();
The Error:
Whoops, looks like something went wrong.
3/3
DriverException in AbstractMySQLDriver.php line 115:
An exception occured in driver: could not find driver
2/3
PDOException in PDOConnection.php line 47:
could not find driver
1/3
PDOException in PDOConnection.php line 43:
could not find driver
Tried using mariaDB database image, check php.ini and the mysqlnd PDO driver is installed:
mysqlnd
mysqlnd enabled
Version mysqlnd 5.0.12-dev - 20150407 - $Id: d8daadaf41e3cd81d7c6ae96c6091fd15b2c9382 $
Compression supported
core SSL supported
extended SSL supported
Command buffer size 4096
Read buffer size 32768
Read timeout 31536000
Collecting statistics Yes
Collecting memory statistics No
Tracing n/a
Loaded plugins mysqlnd,debug_trace,auth_plugin_mysql_native_password,auth_plugin_mysql_clear_password,auth_plugin_sha256_password
API Extensions no value
php-m inside of the _server_ container
~
"php.ini" [New] 1L, 23C written
php -m
[PHP Modules]
Core
ctype
curl
date
dom
fileinfo
filter
ftp
hash
iconv
json
libxml
mbstring
mysqlnd
openssl
pcre
PDO
pdo_sqlite
Phar
posix
readline
Reflection
session
SimpleXML
SPL
sqlite3
standard
tokenizer
xml
xmlreader
xmlwriter
zlib
As a side note I'd like to keep the server service as abstracted as possible as the server may be one of any number of options (built in, apache, nginx, etc).
I has to be something simple Im over looking, but what?
Fixed it:
FROM php:latest
# install curl and php
RUN apt-get update -y
RUN apt-get install curl git zip -y
# call PHP images script `docker-php-ext-install` and install language extensions
RUN docker-php-ext-install pdo pdo_mysql
# install composer & app deps; then remove
RUN curl -sS https://getcomposer.org/installer | php
RUN mv composer.phar /usr/local/bin/composer
# change working dir
WORKDIR /code
CMD ["php", "-S", "0.0.0.0:80"]
Noticed RUN docker-php-ext-install pdo pdo_mysql. Apparently installing php extensions in the php docker container requires using the built in docker-php-ext-install and docker-php-ext-configure scripts that are provided with the machine.
Im trying to use Swift as a backend by installing it on an Ubuntu Server. I followed these instructions:
http://www.sitepoint.com/server-side-swift-with-perfect/
Unfortunately when I did the following:
git clone https://github.com/PerfectlySoft/Perfect.git
cd Perfect/PerfectLib
make
sudo make install
I get an error when running "make". The error is as follows:
<module-includes>:1:10: note: in file included from <module-includes>:1:
#include "curl_support.hpp"
^
/home/chris/Perfect/PerfectLib/linked/cURL_Linux/curl_support.hpp:30:10:
error: 'curl/curl.h' file not found
#include "curl/curl.h"
^
cURL.swift:26:8: error: could not build Objective-C module 'cURL'
import cURL
i tried with all the different Swift Snapshots here:
https://swift.org/download/#apple-platforms
but still nothing.
Please can anyone help? Im using Ubuntu 14.04
You need to install curl library in order to have it on your system :)
sudo apt-get install curl should solve your problem.
I personally use this vagrantfile, that sets up my Virtual Machine (so I don't mess up my regular linux settings by mistake)
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
## 1 get the ubuntu image
config.vm.box = "https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box"
config.vm.provision "shell", inline: <<-SHELL
## 2 Install all necessary dependencies
sudo apt-get --assume-yes install clang libssl-dev libevent-dev libsqlite3-dev libicu-dev uuid-dev libcurl4-openssl-dev git
## 3 get the swift snapshot
curl -O https://swift.org/builds/ubuntu1404/swift-2.2-SNAPSHOT-2015-12-01-b/swift-2.2-SNAPSHOT-2015-12-01-b-ubuntu14.04.tar.gz
## 4 unpack
tar zxf swift-2.2-SNAPSHOT-2015-12-01-b-ubuntu14.04.tar.gz
## 5 add swift snapshot to PATH
echo "export PATH=/home/vagrant/swift-2.2-SNAPSHOT-2015-12-01-b-ubuntu14.04/usr/bin:\"${PATH}\"" >> .profile
echo "Swift has successfully installed on Linux"
## 5.5 create tiny script that will install all the Perfect Libs and have them accessible for the non-sudo user
echo "git clone https://github.com/PerfectlySoft/Perfect.git
cd Perfect/PerfectLib
make
sudo make install
cd ../PerfectServer
make" >> compilePerfectLib.sh
chmod a+x compilePerfectLib.sh
echo "Remember to run ./compilePerfectLib.sh after logging into the VM"
SHELL
config.vm.network :public_network #, bridge: "wlan0"
config.vm.network :forwarded_port, guest: 8181, host: 8181
config.vm.boot_timeout = 300
end
Then after you do vagrant ssh and you're logged in the VM (Virtual Machine), you need to sh ./compilePerfectLib.sh and that will setup your environment :)
Good luck!
I am trying to deploy to AWS the sample Rubber test in the Quick Start wiki
https://github.com/rubber/rubber/wiki/Quick-Start
I get past the part where it prompts for my local machine password to write to the hosts file ( it creates roles, the instance, and a lot of other things; I even got through another error where it was not connecting - I renamed the private key without the .pem extension and that fixed that ), and it runs on for a while, and gives an error here:
failed: "/bin/bash -l -c 'sudo -p '\''sudo password: '\'' bash -l -c '\''export DEBIAN_FRONTEND=noninteractive; apt-get -q -o Dpkg::Options::=--force-confold -y --force-yes install postfix build-essential git-core libxslt-dev ntp mysql-client libmysqlclient15-dev subversion curl autoconf bison ruby zlib1g-dev libssl-dev libreadline6-dev libxml2-dev libyaml-dev apache2 libapache2-mod-proxy-html libcurl4-openssl-dev libapache2-mod-xsendfile apache2-mpm-prefork apache2-prefork-dev collectd libperl-dev monit mysql-server percona-toolkit openjdk-7-jdk unzip python-django python-django-tagging python-cairo python-memcache memcached uwsgi uwsgi-plugin-python uwsgi-plugin-http sqlite3 bzr zip mongodb-10gen haproxy ec2-ami-tools'\'''" on production.foo.com
I got the same failure, and looking just above that line at the output from the command, I saw that the uwsgi-plugin-http package needs to be replaced by the uwsgi-core package.
There apparently a pull request to fix this that has not yet been accepted, but you can fix it by making the same edit: on rubber 2.8.1, open config/rubber/rubber-graphite.yml, go to line 22, and it should look like this:
packages: [python-django, python-django-tagging, python-cairo, python-memcache, memcached, uwsgi, uwsgi-plugin-python, uwsgi-plugin-http, sqlite3, bzr, zip]
Replace uwsgi-plugin-http with uwsgi-core so the line will be:
packages: [python-django, python-django-tagging, python-cairo, python-memcache, memcached, uwsgi, uwsgi-plugin-python, uwsgi-core, sqlite3, bzr, zip]
Then, run cap rubber:bootstrap, and it should finish setting up your instance.