Cannot use/pass shell variables using jq on command line - environment-variables

I am fairly new to jq and I am using this tutorial to add a new Org to a hyperledger fabric network.
There is extensive use of jq throughout the tutorial, especially modifying json files.
The tutorial uses an example Org name but I am trying to make the org name dynamic. Everything works out well except when I try to pass variables to jq.
Here are the jq commands and their outputs.
jq version: 1.5.1
$ export MSPID=Org4MSP
$ echo $MSPID
Org4MSP
Trying to pass the variable using env.
Keyword: env.MSPID
$ jq -s '.[0] * {"channel_group":{"groups":{"Application":{"groups": {"env.MSPID":.[1]}}}}}' config.json org4.json
Output snippet: Instead of printing Org4MSP, it prints the literal string env.MSPID
"env.MSPID": {
"groups": {},
"mod_policy": "Admins",
"policies": {
"Admins": {
"mod_policy": "Admins",
"policy": {
"type": 1,
"value": {
"identities": [
{
Trying to pass the variable using --arg option
Keyword: "$MSP"
jq --arg MSP "$MSPID" -s '.[0] * {"channel_group":{"groups":{"Application":{"groups": {"$MSP":.[1]}}}}}' config.json org4.json
Output snippet: Instead of printing Org4MSP, it prints the literal string $MSP
"$MSP": {
"groups": {},
"mod_policy": "Admins",
"policies": {
"Admins": {
"mod_policy": "Admins",
"policy": {
"type": 1,
"value": {
"identities": [
{
Trying to pass a variable using --arg option and without using double quotes:
keyword: $MSP
$ jq --arg MSP "$MSPID" -s '.[0] * {"channel_group":{"groups":{"Application":{"groups": {$MSP:.[1]}}}}}' config.json org4.json
jq: error: syntax error, unexpected ':', expecting '}' (Unix shell quoting issues?) at , line 1:
.[0] * {"channel_group":{"groups":{"Application":{"groups": {$MSP:.[1]}}}}}
jq: 1 compile error
Trying to pass variable using env. and without double quotes:
keyword: env.MSPID
$ jq -s '.[0] * {"channel_group":{"groups":{"Application":{"groups": {env.MSPID:.[1]}}}}}' config.json org4.json
jq: error: syntax error, unexpected FIELD, expecting '}' (Unix shell quoting issues?) at , line 1:
.[0] * {"channel_group":{"groups":{"Application":{"groups": {env.MSPID:.[1]}}}}}
jq: 1 compile error
I apologize if this seems to be a trivial question but I have searched online and in docs and do not understand why the JSON key will not convert to the shell variable's value.
Thank you

Environment variables
In your sub-expression:
{"env.MSPID":.[1]}
you have quoted env.MSPID thereby making it a literal string. Since you want to invoke the env function, you should instead write:
{ (env.MSPID):.[1]}
The parentheses are needed to ensure that jq will evaluate the parenthesized expression properly.
{$MSP:.[1]}
As noted above, when an expression must be evaluated to determine the string-value of a key, the expression must be parenthesized, e.g.
{($MSP):.[1]}

Related

Cannot render groovy sh script with environmental variable that uses jq

I have this line here that is inside of a groovy function and i am trying to return a cluster ID
EMR_ID = sh(returnStdout: true, script: "aws emr list-clusters --active --profile \'${PROFILE}\' | jq -r '.Clusters[] | select (.Name=="\'${ENV}\'-emr-cluster") | .Id'")
Without the environmental variables, this works fine.
so the $PROFILE param is set to dev and the $ENV is set to aws-dev.
I keep getting the error that complains about the $ENV var, saying that an unexpected bracket is showing up
It is rarely a good idea to pass environment variables using string-interpolation. There are much better ways.
One you might consider is modifying your invocation of jq as follows:
jq -r --arg env "${ENV}" '.Clusters[] | select (.Name=="\($env)-emr-cluster") | .Id'
Here, $env is an ordinary jq variable, or more accurately perhaps, a defined constant.
After realising that the jenkins snippet generator was actually useful I used this:
sh '''jq -r --arg env "${ENV}" \'.Clusters[] | select (.Name=="\\($env)-emr-cluster") | .Id\' '''
and it worked!

Querying the "name" attribute of a nix program

Let's say I using a nix program nixProgram which is located in some nix store location like
/nix/store/5zlrw36bpq6z1m3zjawwrwaryhmqjwbr-nixProgram-75.0/bin/nixProgram
Here I'm assuming nixProgram-75.0 is the name attribute of the derivation.
Question: Is there some well-known way to query what the name attribute of the nixProgram is over the Linux shell? Something like
$ nix-get-name nixProgram
nixProgram-75.0
?
You can use nix-store -qd to get the derivation that generated the store path. This derivation might have already been garbage collected in which case you're out of luck. Otherwise you can use nix show-derivation to serialize the derivation to JSON and use jq to get the name field.
#!/bin/sh
set -e
die() {
>&2 echo "$#"
exit 1
}
# Get the store path of $1
STORE_PATH=${1%*${1#/nix/store/*/}}
# Query the derivation which generates the store path
DRV=$(nix-store -qd "$STORE_PATH")
# Check if the derivation is still there
[ -f "$DRV" ] || die "You're out of luck, the derivation is no longer there"
# Serialize to JSON and use jq to get the name
nix show-derivation "$DRV" | jq -r '.[] | .env.name'
On my system:
$ ./nix-get-name /nix/store/4xb9z8vvk3fk2ciwqh53hzp72d0hx1da-bash-interactive-4.4-p23/bin/bash
bash-interactive-4.4-p23

jq: not using environment variable correctly

I have an envs.json file with content:
{
"dev-cc1": { "url": "https://my-url.com" },
"dev-cc2": { "url": "https://my-url.com" }
}
I would like to fetch urls based on an environment variable $ENV, but every other related article I came across didn't seem to work, or I just read it wrong. I think I'm missing something super trivial...
Things I tried:
export ENV="dev-cc1"
jq --arg ENV "$ENV" -n '."env.ENV"' envs.json
Because my env variable has a dash in it, I quoted around it, and then tried to reference it inside. However jq returns back null.
The hardcoded query works (jq --arg ENV "$ENV" '."dev-cc1"' envs.json), and I've also confirmed that the environment variable is passed in correctly jq --arg ENV "$ENV" -n 'env.ENV'.
I tried a ton of different ways to substitute this env var in but none worked.. Could anyone please give this a second pair of eyes?
Try
ENV="dev-cc1" jq '.[env.ENV]' envs.json
or
export ENV="dev-cc1"
jq '.[env.ENV]' envs.json
or using a jq variable:
ENV="dev-cc1"
jq --arg ENV "$ENV" '.[$ENV]' envs.json

JQ adds single quotes while saving in environment variables

OK, this might be a silly question. I've got the test.json file:
{
"timestamp": 1234567890,
"report": "AgeReport"
}
What I want to do is to extract timestamp and report values and store them in some env variables:
export $(cat test.json | jq -r '#sh "TIMESTAMP=\(.timestamp) REPORT=\(.report)"')
and the result is:
echo $TIMESTAMP $REPORT
1234567890 'AgeReport'
The problem is that those single quotes break other commands.
How can I get rid of those single quotes?
NOTE: I'm gonna leave the accepted answer as is, but see #Inian's answer for a better solution.
Why make it convoluted with using eval and have a quoting mess? Rather simply emit the variables by joining them with NULL (\u0000) and read it back in the shell environment
{
IFS= read -r -d '' TIMESTAMP
IFS= read -r -d '' REPORT
} < <(jq -r '(.timestamp|tostring) + "\u0000" + .report + "\u0000"' test.json)
This makes your parsing more robust by making the fields joined by NULL delimiter, which can't be part of your string sequence.
From the jq man-page, the #sh command converts its input to be
escaped suitable for use in a command-line for a POSIX shell.
So, rather than attempting to splice the output of jq into the shell's export command which would require carefully removing some quoting, you can generate the entire commandline inside jq, and then execute it with eval:
eval "$(
cat test.json |\
jq -r '#sh "export TIMESTAMP=\(.timestamp) REPORT=\(.report)"'
)"

How to get match[1] with grep?

Here's my code so far...
cat config.json | grep -Po '"server"\s*:\s*"([^"]*)"'
But I just want the part within (parentheses). I can't use a look-behind because it's variable-length. What are my options?
Sample input 1:
{"debug":false,"server":"dev-dutch","env":"dev"}
Sample input 2:
{
"debug": false,
"server": "dev-dutch",
"env": "dev"
}
Desired output for both:
dev-dutch
I know there are probably safer/better ways to parse JSON, but I want to do this in shell, and it should run on both Ubuntu and FreeBSD without installing any external programs, so I'm OK with a grep hack.
With GNU grep:
grep -Po '"server": *"\K[^"]+' file

Resources