symfony/yaml backed symfony/config not parsing environment variables - environment-variables

I have recreated a simple example in this tiny github repo. I am attempting to use symfony/dependency-injection to configure monolog/monolog to write logs to php://stderr. I am using a yaml file called services.yml to configure dependency injection.
This all works fine if my yml file looks like this:
parameters:
log.file: 'php://stderr'
log.level: 'DEBUG'
services:
stream_handler:
class: \Monolog\Handler\StreamHandler
arguments:
- '%log.file%'
- '%log.level%'
log:
class: \Monolog\Logger
arguments: [ 'default', ['#stream_handler'] ]
However, my goal is to read the path of the log files and the log level from environment variables, $APP_LOG and LOG_LEVEL respectively. According to The symphony documentations on external paramaters the correct way to do that in the services.yml file is like this:
parameters:
log.file: '%env(APP_LOG)%'
log.level: '%env(LOGGING_LEVEL)%'
In my sample app I verified PHP can read these environment variables with the following:
echo "Hello World!\n\n";
echo 'APP_LOG=' . (getenv('APP_LOG') ?? '__NULL__') . "\n";
echo 'LOG_LEVEL=' . (getenv('LOG_LEVEL') ?? '__NULL__') . "\n";
Which writes the following to the browser when I use my original services.yml with hard coded values.:
Hello World!
APP_LOG=php://stderr
LOG_LEVEL=debug
However, if I use the %env(VAR_NAME)% syntax in services.yml, I get the following error:
Fatal error: Uncaught UnexpectedValueException: The stream or file "env_PATH_a61e1e48db268605210ee2286597d6fb" could not be opened: failed to open stream: Permission denied in /var/www/vendor/monolog/monolog/src/Monolog/Handler/StreamHandler.php:107 Stack trace: #0 /var/www/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php(37): Monolog\Handler\StreamHandler->write(Array) #1 /var/www/vendor/monolog/monolog/src/Monolog/Logger.php(337): Monolog\Handler\AbstractProcessingHandler->handle(Array) #2 /var/www/vendor/monolog/monolog/src/Monolog/Logger.php(532): Monolog\Logger->addRecord(100, 'Initialized dep...', Array) #3 /var/www/html/index.php(17): Monolog\Logger->debug('Initialized dep...') #4 {main} thrown in /var/www/vendor/monolog/monolog/src/Monolog/Handler/StreamHandler.php on line 107
What am I doing wrong?

Ok you need a few things here. First of all you need version 3.3 of Symfony, which is still in beta. 3.2 was the released version when I encountered this. Second you need to "compile" the environment variables.
Edit your composer.json with the following values and run composer update. You might need to update other dependencies. You can substitute ^3.3 with dev-master.
"symfony/config": "^3.3",
"symfony/console": "^3.3",
"symfony/dependency-injection": "^3.3",
"symfony/yaml": "^3.3",
You will likely have to do this for symfony/__WHATEVER__ if you have other symfony components.
Now in you're code after you load your yaml configuration into your dependency container you compile it.
So after you're lines here (perhaps in bin/console):
$container = new ContainerBuilder();
$loader = new YamlFileLoader($container, new FileLocator(__DIR__ . DIRECTORY_SEPARATOR . '..'));
$loader->load('services.yml');
Do this:
$container->compile(true);
Your IDE's intellisense might tell you compile takes no parameters. That's ok. That's because compile() grabs its args indirectly via func_get_arg().
public function compile(/*$resolveEnvPlaceholders = false*/)
{
if (1 <= func_num_args()) {
$resolveEnvPlaceholders = func_get_arg(0);
} else {
. . .
}
References
Github issue where this was discussed
Pull request to add compile(true)

Using this command after loading your services.yaml file should help.
$containerBuilder->compile(true);
given your files gets also validated by the checks for proper configurations which this method also does. The parameter is $resolveEnvPlaceholders which makes environmental variables accessible to the yaml services configuration.

Related

Error when running near indexer localnet, fail to generate config.json

So I'm trying to run the indexer on localnet following the official tutorial https://docs.near.org/docs/tutorials/near-indexer
However when I run cargo run -- init to generate the localnet json config I get this error
Finished dev [unoptimized + debuginfo] target(s) in 17.62s
Running `target/debug/example-indexer init`
thread 'main' panicked at 'Failed to deserialize config: Error("expected value", line: 1, column: 1)', /home/francois/.cargo/git/checkouts/nearcore-5bf7818cf2261fd0/a44be20/nearcore/src/config.rs:499:39
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
At some point it seems the json is not created or not created properly I guess, the function crashing in config.rf line 499 is
impl From<&str> for Config {
fn from(content: &str) -> Self {
serde_json::from_str(content).expect("Failed to deserialize config")
}
}
It's quite difficult to debug since cargo run -- init is using some inner near function (also I'm new to rust).
the config.json file is created but it seems the permission are not set properly by the script, the content of config.json is
"<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message> ... "
If anyone from the community has encountered this problem or has a hint it would be great!! thanks a lot !
In the tutorial you referenced, it mentions a similar error, and suggests the following:
Open your config.json located in the .near folder in the root of your home directory. ( ~/.near/config.json )
In this file, locate: "tracked_shards": [] and change the value to [0].
Save the file and try running your indexer again.
So I had the wrong config with download_config: false,
It should be download_config: false, for the localnet use

How to I set an ENV VAR in my WORKSPACE in Bazel

I am trying to use Bazel with Pybind, and it requires that I set the following variables:
"""Repository rule for Python autoconfiguration.
`python_configure` depends on the following environment variables:
* `PYTHON_BIN_PATH`: location of python binary.
* `PYTHON_LIB_PATH`: Location of python libraries.
"""
https://github.com/pybind/pybind11_bazel/blob/master/python_configure.bzl
I dont want to have to pass it in manually when building my libraries, how can i hardcode these env vars in my WORKSPACE?
To (always) set environmental variable for a repository rule consumption, you case use --repo_env command line option. And if you want to include those with every invocation in your workspace, you can set add these flags to your .bazelrc file therein.
Now the wisdom of doing that could be questioned. If it's actually a project (repo) and not build host configuration, it would probably make more sense, be more targeted and more explicit, if it was an attribute of the given rule which was then checked in with the rest of the build configuration.
And looking at the name, there may be another question about specifying python configuration (from outside the bazel build) instead of actually using correctly resolved python toolchain (but there I have to say have no background in what the given rule is about and what is it trying to accomplish to render judgment, this is just a general comment).
To address your comment... I don't what other factors make it "not accept" or what exactly does that actually look like, but if I have this mini-example:
.
├── BUILD
├── WORKSPACE
└── customrule.bzl
Where customrule.bzl reads:
def _run_me(repo_ctx):
repo_ctx.file(
"WORKSPACE",
'workspace(name = "{}")\n'.format(repo_ctx.name),
executable = False,
)
repo_ctx.file(
"BUILD",
'exports_files(["var.sh"], visibility=["//visibility:public"])',
executable = False,
)
repo_ctx.file(
"var.sh",
"echo {}\n".format(repo_ctx.os.environ.get("var1")),
executable = True,
)
wsrule = repository_rule(
implementation = _run_me,
environ = ["var1"],
)
The WORKSPACE is:
load(":customrule.bzl", "wsrule")
wsrule(
name = "extdep"
)
And BUILD:
sh_binary(
name = "tgt",
srcs = ["#extdep//:var.sh"],
)
Then I do get:
$ bazel run --repo_env var1=val1 tgt
val1
and:
$ bazel run --repo_env var1=val2 tgt
val2
I.e. this is a way to pass variables to a repo rule and it does (as such) work.
If you absolutely know, you must call a build with some variable set to certain value (which as mentioned above is itself a requirement that is worth closer examination) and you want these to be associated with the project / repo. You can always check in a build.sh or any such file that wraps your bazel call to be exactly what it must be. But again, this looks more likely to not be really entirely "The Right Thing" to do or want.

pyhdfs.HdfsIOException: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

I am trying to run hadoop using docker provided here:
https://github.com/big-data-europe/docker-hadoop
I use the following command:
docker-compose up -d
to up the service and am able to access it and browse file system using: localhost:9870. Problem rises whenever I try to use pyhdfs to put file on HDFS. Here is my sample code:
hdfs_client = HdfsClient(hosts = 'localhost:9870')
# Determine the output_hdfs_path
output_hdfs_path = 'path/to/test/dir'
# Does the output path exist? If not then create it
if not hdfs_client.exists(output_hdfs_path):
hdfs_client.mkdirs(output_hdfs_path)
hdfs_client.create(output_hdfs_path + 'data.json', data = 'This is test.', overwrite = True)
If test directory does not exist on HDFS, the code is able to successfully create it but when it gets to the .create part it throws the following exception:
pyhdfs.HdfsIOException: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null
What surprises me is that my code is able to create the empty directory but fails to put the file on HDFS. My docker-compose.yml file is exactly the same as the one provided in the github repo. The only change I've made is in the hadoop.env file where I change:
CORE_CONF_fs_defaultFS=hdfs://namenode:9000
to
CORE_CONF_fs_defaultFS=hdfs://localhost:9000
I have seen this other post on sof and tried the following command:
hdfs dfs -mkdir hdfs:///demofolder
which works fine in my case. Any help is much appreciated.
I would keep the default CORE_CONF_fs_defaultFS=hdfs://namenode:9000 setting.
Works fine for me after adding a forward slash to the paths
import pyhdfs
fs = pyhdfs.HdfsClient(hosts="namenode")
output_hdfs_path = '/path/to/test/dir'
if not fs.exists(output_hdfs_path):
fs.mkdirs(output_hdfs_path)
fs.create(output_hdfs_path + '/data.json', data = 'This is test.')
# check that it's present
list(fs.walk(output_hdfs_path))
[('/path/to/test/dir', [], ['data.json'])]

CraftCMS exception on first install (HTTP 503 – ServiceUnavailableHttpException)

I'm trying to install CraftCMS for the first time, and appear to have gone through all the steps on the installation guide - https://docs.craftcms.com/v3/installation.html#step-1-download-craft - yet I'm getting an Exception.
HTTP 503 – Service Unavailable – craft\web\ServiceUnavailableHttpException
Here is the line (509 in /var/www/craft/vendor/craftcms/cms/src/web/Application.php) that's throwing the exception:
// Should they be accessing the installer?
if (!$isInstalled) {
if (!$isCpRequest) {
throw new ServiceUnavailableHttpException();
}
Below is the call stack:
craft\web\ServiceUnavailableHttpException in /var/www/craft/vendor/craftcms/cms/src/web/Application.php:509
Stack trace:
#0 /var/www/craft/vendor/craftcms/cms/src/web/Application.php(184): craft\web\Application->_processInstallRequest(Object(craft\web\Request))
#1 /var/www/craft/vendor/yiisoft/yii2/base/Application.php(386): craft\web\Application->handleRequest(Object(craft\web\Request))
#2 /var/www/craft/web/index.php(21): yii\base\Application->run()
#3 {main}
I'm using v3.0.24 as far as I can see:
- Installing craftcms/cms (3.0.24): Downloading (100%)
As I haven't even got started with the CMS, I don't really know what more info to give - or where to go from here. The .env file has been copied, there really is no more instruction to do anything. Any ideas?
UPDATE
I've identified this section here (in /vendor/yiisoft/yii2/db/mysql/Schema.php) is returning an empty array:
protected function findTableNames($schema = '')
{
$sql = 'SHOW TABLES';
if ($schema !== '') {
$sql .= ' FROM ' . $this->quoteSimpleTableName($schema);
}
return $this->db->createCommand($sql)->queryColumn();
}
The table have been setup, I can see them in the MySQL console. My .env db config settings seem totally fine too.
Try the following steps for install craft3 by the terminal.
create a virtual host that point to the web directory of the project setup.
composer create-project craftcms/craft
./craft setup/security-key
./craft setup
After completing the above steps, provide the permission of storage, config, web, Modules, template folder.
Admin URL: http:///index.php/admin
For those creating a fresh install using Craft CMS Nitro and its nitro create command, don't forget to run the the Setup Wizard as a final step, as described in Step 6: Run the Setup Wizard, from the Craft Docs.
This will populate the database with Craft's tables and what not and should address the 503 error.

Angular - Error: [$injector:unpr] Unknown provider: IdleProvider

Getting the error Error: [$injector:unpr] Unknown provider: IdleProvider in my application when it is deployed to our staging server using dokku but I am not getting it when running it on my local machine. I'm using ng-idle 1.2.1
I've found this question asked a number of times but the cause was always related to the changes made in version 1.0.0 where the service names were changed. The only thing I can think of is that the minification of the code is the problem but as far as I can see the code should be ok but I am not an expert. Any help would be greatly appreciated.
It's written in Coffeescript
configuration = (RestangularProvider, $logProvider, growlProvider, IdleProvider, KeepaliveProvider) ->
.
.
.
return
configuration.$inject = [
'RestangularProvider'
'$logProvider'
'growlProvider'
'IdleProvider'
'KeepaliveProvider'
]
angular
.module 'vssApp.config', [
'restangular'
]
.config configuration
EDIT
While trying to replicate the problem on my local machine I removed the 'ngIdle' module in the modules array below. This resulted in the same behavior so I am assuming that the problem stems from the ngIdle module not being loaded correctly here. I still feel that minification could be causing the problem but, again, I'm not sure why or how to fix it.
modules = [
'ui.router'
'ui.bootstrap'
'ui.select'
'ngAnimate'
'ngMessages'
'ngSanitize'
'ngCookies'
'smart-table'
'angularMoment'
'templates'
'angular-storage'
'angular-growl'
'vssApp.core.auth'
'vssApp.core.loading'
'ngIdle'
'cgPrompt'
'vssApp.filters'
]
runBlock.$inject = [
'$templateCache'
]
angular
.module 'vssApp.core', modules
.run runBlock
EDIT 2
Here's the full output from the error message I'm getting
Error: [$injector:modulerr] Failed to instantiate module vssApp due to:
Error: [$injector:modulerr] Failed to instantiate module vssApp.config due to:
Error: [$injector:unpr] Unknown provider: IdleProvider
http://errors.angularjs.org/1.3.16/$injector/unpr?p0=IdleProvider
at https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:3:18814
at https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:4:16489
at getService (https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:4:14903)
at Object.invoke (https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:4:15466)
at runInvokeQueue (https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:4:13793)
at https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:4:14062
at forEach (https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:3:19482)
at loadModules (https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:4:13587)
at https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:4:13964
at forEach (https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:3:19482)
http://errors.angularjs.org/1.3.16/$injector/modulerr?p0=vssApp.config&p1=E…net%2Fassets%2Fapplication-85a5fd382c73380bf2a71b66e581c941.js%3A3%3A19482)
at https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:3:18814
at https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:4:14406
at forEach (https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:3:19482)
at loadModules (https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:4:13587)
at https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:4:13964
at forEach (https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:3:19482)
at loadModules (https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:4:13587)
at createInjector (https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:4:16844)
at doBootstrap (https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:3:28466)
at bootstrap (https://SERVER/assets/application-85a5fd382c73380bf2a71b66e581c941.js:3:28995)
http://errors.angularjs.org/1.3.16/$injector/modulerr?p0=vssApp&p1=Error%3A…net%2Fassets%2Fapplication-85a5fd382c73380bf2a71b66e581c941.js%3A3%3A28995)
A rule of thumb is to load modules dependencies in each place where they are used. This allows to decouple them. And this eliminates race condition with service provider injection.
If the app looks like this
angular.module('vssApp', ['vssApp.config', 'ngIdle', ...])..
angular.module('vssApp.config', ['restangular'])...
service provider for Idle service is not defined at the time when vssApp.config module is loaded.
While this
angular.module('vssApp', ['ngIdle', 'vssApp.config', ...])
angular.module('vssApp.config', ['restangular'])...
avoids race condition but still indicates code smell.
It should be
angular.module('vssApp.config', ['restangular', 'ngIdle'])...
This issue applies to service providers only and config phase. Service instances can be injected for any module order.
Finally found the cause and solution to this, it seems to have been a bower issue.
It's a Rails app, so I specified ng-idle 1.2.1 in the bower.json file but for some reason the bower file was ignored when the app was being deployed using Dokku and the last installed version 0.3.5 remained, which meant that the pre-1.0.0 ng-idle services naming convention was still being used where all service names were preceded with a $. This resulted in the Unknown provider: IdleProvider error because $IdleProvider was the actual service name.
In the end I had to connect to the docker container and remove and reinstall all bower components. Running bower update as part of the deployment was not enough for some reason. When I have more time I will investigate what caused this behavior and I will report here.

Resources