Get values from JSON in Ruby - ruby-on-rails

I am trying to get the VolumeId and State of the Volume attached to the machines using aws API .
Code
#!/usr/local/bin/ruby
require "aws-sdk"
require "rubygems"
list=Aws::EC2::Client.new(region: "us-east-1")
volume=list.describe_volumes()
volumes=%x( aws ec2 describe-volumes --region='us-east-1' )
puts volumes
Below is the sample output of the command
aws ec2 describe-volumes --region='us-east-1' .
Please help to get VolumeID and state from the below
Sample Output of API(JSON):
{
"Volumes": [
{
"AvailabilityZone": "us-east-1d",
"Attachments": [
{
"AttachTime": "2015-02-02T07:31:36.000Z",
"InstanceId": "i-bca66353",
"VolumeId": "vol-892a2acd",
"State": "attached",
"DeleteOnTermination": true,
"Device": "/dev/sda1"
}
],
"Encrypted": false,
"VolumeType": "gp2",
"VolumeId": "vol-892a2acd",
"State": "in-use",
"Iops": 100,
"SnapshotId": "snap-df910966",
"CreateTime": "2015-02-02T07:31:36.380Z",
"Size": 8
},
]
}

for getting just the volume_ids ->
JSON.parse(volumes)['Volumes'].map{|v|v["VolumeId"]}
for getting just the states ->
JSON.parse(volumes)['Volumes'].map{|v|v["state"]}
for getting a hash/map with volume-ids as keys and their states as values ->
JSON.parse(volumes)['Volumes'].map{|v| [v["VolumeId"],v["state"]] }.to_h

Related

Creating Managed Policy in CDK errors with MalformedPolicy

When I try to deploy a seemingly simple CDK stack, it fails with a strange error. I don't get this same behavior when I create a different iam.ManagedPolicy in a different file, and that one has a much more complicated policy with several actions, etc. What am I doing wrong?
import aws_cdk.core as core
from aws_cdk import aws_iam as iam
from constructs import Construct
from master_payer import ( env, myenv )
class FromStack(core.Stack):
def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
#myenv['pma'] = an account ID (12 digits)
#env = 'dev'
rolename = f"arn:aws:iam:{myenv['pma']}:role/CrossAccount{env.capitalize()}MpaAdminRole"
mpname = f"{env.capitalize()}MpaAdminPolicy"
pol = iam.ManagedPolicy(self, mpname, managed_policy_name = mpname,
document = iam.PolicyDocument(statements= [
iam.PolicyStatement(actions=["sts:AssumeRole"], effect=iam.Effect.ALLOW, resources=[rolename])
]))
grp = iam.Group(self, f"{env.capitalize()}MpaAdminGroup", managed_policies=[pol])
The cdk deploy output:
FromStack: deploying...
FromStack: creating CloudFormation changeset...
2:19:52 AM | CREATE_FAILED | AWS::IAM::ManagedPolicy | DevMpaAdminPolicyREDACTED
The policy failed legacy parsing (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: REDACTED-GUID; Proxy: null)
new ManagedPolicy (/tmp/jsii-kernel-EfRyKw/node_modules/#aws-cdk/aws-iam/lib/managed-policy.js:39:26)
\_ /tmp/tmpxl5zxf8k/lib/program.js:8432:58
\_ Kernel._wrapSandboxCode (/tmp/tmpxl5zxf8k/lib/program.js:8860:24)
\_ Kernel._create (/tmp/tmpxl5zxf8k/lib/program.js:8432:34)
\_ Kernel.create (/tmp/tmpxl5zxf8k/lib/program.js:8173:29)
\_ KernelHost.processRequest (/tmp/tmpxl5zxf8k/lib/program.js:9757:36)
\_ KernelHost.run (/tmp/tmpxl5zxf8k/lib/program.js:9720:22)
\_ Immediate._onImmediate (/tmp/tmpxl5zxf8k/lib/program.js:9721:46)
\_ processImmediate (node:internal/timers:464:21)
❌ FromStack failed: Error: The stack named FromStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE
at Object.waitForStackDeploy (/usr/local/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:307:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at prepareAndExecuteChangeSet (/usr/local/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:351:26)
at CdkToolkit.deploy (/usr/local/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:194:24)
at initCommandLine (/usr/local/lib/node_modules/aws-cdk/bin/cdk.ts:267:9)
The stack named FromStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE
And the cdk synth output, which cfn-lint is happy with (no warnings, errors, or informational violations):
{
"Resources": {
"DevMpaAdminPolicyREDACTED": {
"Type": "AWS::IAM::ManagedPolicy",
"Properties": {
"PolicyDocument": {
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Resource": "arn:aws:iam:REDACTED-ACCOUNT-ID:role/CrossAccountDevMpaAdminRole"
}
],
"Version": "2012-10-17"
},
"Description": "",
"ManagedPolicyName": "DevMpaAdminPolicy",
"Path": "/"
},
"Metadata": {
"aws:cdk:path": "FromStack/DevMpaAdminPolicy/Resource"
}
},
"DevMpaAdminGroupREDACTED": {
"Type": "AWS::IAM::Group",
"Properties": {
"ManagedPolicyArns": [
{
"Ref": "DevMpaAdminPolicyREDACTED"
}
]
},
"Metadata": {
"aws:cdk:path": "FromStack/DevMpaAdminGroup/Resource"
}
},
"CDKMetadata": {
"Type": "AWS::CDK::Metadata",
"Properties": {
"Analytics": "v2:deflate64:REDACTED-B64"
},
"Metadata": {
"aws:cdk:path": "FromStack/CDKMetadata/Default"
}
}
}
}
Environment Specs
$ cdk --version
2.2.0 (build 4f5c27c)
$ cat /etc/redhat-release
Red Hat Enterprise Linux releease 8.5 (Ootpa)
$ python --version
Python 3.6.8
$ node --version
v16.8.0
The role ARN rolename was incorrect; I was missing a colon after iam. So it's iam:: not iam:. I think I copied the single colon from a (wrong) example somewhere on the Internet. Gah...

With Google Cloud Speech-to-text, why do I get different results for the same audio file, depending on which bucket do I put it into?

I am trying to use Google Cloud Speech-to-text, using the client libraries, from a node.js environment, and I see something I don't understand: I get a different result for the same example audio file, and the same configuration, depending on whether I am using it from the original sample bucket, or from my own bucket.
There are the requests and responses:
The baseline is Google's own test data file, available here: https://storage.googleapis.com/cloud-samples-tests/speech/brooklyn.flac
Request:
{
"config": {
"encoding": "FLAC",
"languageCode": "en-US",
"sampleRateHertz": 16000,
"enableAutomaticPunctuation": true
},
"audio": {
"uri": "gs://cloud-samples-tests/speech/brooklyn.flac"
}
}
Response:
{
"results": [
{
"alternatives": [
{
"transcript": "How old is the Brooklyn Bridge?",
"confidence": 0.9831430315971375
}
]
}
]
}
So far, so good. But, if I download this audio file, re-upload it to my own bucket, and do the same, then:
Request:
{
"config": {
"encoding": "FLAC",
"languageCode": "en-US",
"sampleRateHertz": 16000,
"enableAutomaticPunctuation": true
},
"audio": {
"uri": "gs://goe-transcript-creation/brooklyn.flac"
}
}
Response:
{
"results": [
{
"alternatives": [
{
"transcript": "how old is",
"confidence": 0.8902621865272522
}
]
}
]
}
As you can see this is the same request. The re-uploaded audio data is here: https://storage.googleapis.com/goe-transcript-creation/brooklyn.flac
This the exact same file as in the first example... not a bit of difference.
Still, the results are different; I only get half of the sentence.
What am I missing here? Thanks.
Update 1:
The same thing happens with the CLI tool, too:
$ gcloud ml speech recognize gs://cloud-samples-tests/speech/brooklyn.flac --language-code=en-US
{
"results": [
{
"alternatives": [
{
"confidence": 0.98314303,
"transcript": "how old is the Brooklyn Bridge"
}
]
}
]
}
$ gcloud ml speech recognize gs://goe-transcript-creation/brooklyn.flac --language-code=en-US
ERROR: (gcloud.ml.speech.recognize) INVALID_ARGUMENT: Invalid recognition 'config': bad encoding..
$ gcloud ml speech recognize gs://goe-transcript-creation/brooklyn.flac --language-code=en-US --encoding=FLAC
ERROR: (gcloud.ml.speech.recognize) INVALID_ARGUMENT: Invalid recognition 'config': bad sample rate hertz.
$ gcloud ml speech recognize gs://goe-transcript-creation/brooklyn.flac --language-code=en-US --encoding=FLAC --sample-rate=16000
{
"results": [
{
"alternatives": [
{
"confidence": 0.8902483,
"transcript": "how old is"
}
]
}
]
}
It's also interesting that when pulling the audio from the other bucket, I need to specify encoding and sample rate, otherwise it doesn't work... but it's not necessary when I am using the original test bucket.
Update 2:
If I don't use Google Cloud Storage, but upload the data directly in the speech-to-text request, it works as intended:
$ gcloud ml speech recognize brooklyn.flac --language-code=en-US
{
"results": [
{
"alternatives": [
{
"confidence": 0.98314303,
"transcript": "how old is the Brooklyn Bridge"
}
]
}
]
}
So the problem doesn't seems to be with the recognition itself, but accessing the audio data. The obvious guess would be that maybe it's the fault of the uploading, and the data is somehow corrupted along the way?
We can verify that by pulling the data from the cloud, and comparing with the original. It doesn't seem to be broken.
So maybe it's a problem when the S-T-T service is accessing the storage service? But why with one bucket only? Or is it some kind of file metadata problem?

How to protect awsconfiguration.json data details in iOS app?

I'm using awsconfiguration.json for AWS Cognito for my iOS Application written in swift. But I'm afraid of security that awsconfiguration.json is stored in my local directory. How can I protect this json file against a third man attack?
Please see similar Github Issue https://github.com/aws-amplify/aws-sdk-ios/issues/1671
The comments talk about
the file is non-sensitive data, so resources that should be accessed by authenticated users should be configured with the approiate controls. Amplify CLI helps you with this, depending on the resources you are provisioning in AWS.
there is a way to configure it in-memory via AWSInfo.configureDefaultAWSInfo(awsConfiguration)
Configure your AWS dependencies using in-memory configuration instead of the configuration JSON file as suggested by AWS documentation.
Sample code:
func buildAuthConfiguration() -> [String:JSONValue] {
return [
"awsCognitoAuthPlugin": [
"IdentityManager": [
"Default": [:]
],
"Auth": [
"Default": [
"authenticationFlowType": "String"
]
],
"CognitoUserPool": [
"Default": [
"PoolId": "String",
"AppClientId": "String",
"Region": "String"
]
],
"CredentialsProvider": [
"CognitoIdentity": [
"Default": [
"PoolId": "String",
"Region": "String"
]
]
]
]
]
}
func buildAPIConfiguration() -> [String: JSONValue] {
return [
"awsAPIPlugin": [
"apiName" : [
"endpoint": "String",
"endpointType": "String",
"authorizationType": "String",
"region": "String"
]
]
]
}
func configureAmplify() {
let authConf = AuthCategoryConfiguration(plugins: buildAuthConfiguration())
let apiConf = APICategoryConfiguration(plugins: buildAPIConfiguration())
let config = AmplifyConfiguration(
analytics: nil,
api: apiConf,
auth: authConf,
dataStore: nil,
hub: nil,
logging: nil,
predictions: nil,
storage: nil
)
try Amplify.configure(config)
// Rest of your code
}
Source: https://github.com/aws-amplify/amplify-ios/issues/1171#issuecomment-832988756
You can provide data protection to your app files by saving it into file directory
Following documentation can help you to achieve it.
https://developer.apple.com/documentation/uikit/protecting_the_user_s_privacy/encrypting_your_app_s_files
The fix to add a new constructor has been released in 2.13.6 version of the SDK.
to allow passing a JSONObject containing the configuration from the awsconfiguration.json file. You can store the information in JSONObject in your own security mechanism and provide it at runtime through the constructor.
https://github.com/aws-amplify/aws-sdk-android/pull/1002

Having trouble accessing /dev/serial0 on a Raspi from within a Module in Azure IoT Edge

I'm trying to set up a Module which will interact with /dev/serial0 on a Raspberry Pi B+ running Raspian Stretch. I've used dtoverlay=pi3-miniuart-bt in /boot/config.txtto restore UART0/ttyAMA0 to GPIOs 14 and 15 (which is what my Raspi-based HW needs me to do).
I have attempted to make that device accessible to the Module using the following Container Create Options:
{
"HostConfig": {
"PortBindings": {
"1880/tcp": [
{
"HostPort": "1880"
}
]
},
"Privileged": true,
"Devices": [
{
"PathOnHost": "/dev/serial0",
"PathInContainer": "/dev/serial0",
"CgroupPermissions": "rwm"
},
{
"PathOnHost": "/dev/ttyAMA0",
"PathInContainer": "/dev/ttyAMA0",
"CgroupPermissions": "rwm"
},
{
"PathOnHost": "/dev/ttyS0",
"PathInContainer": "/dev/ttyS0",
"CgroupPermissions": "rwm"
}
]
}
}
I can see /dev/serial0 when I ssh in, but I can't see it from within the running Module:
pi#azure-iot-test:~ $ ls -l /dev/ser*
lrwxrwxrwx 1 root root 7 Sep 24 21:17 /dev/serial0 -> ttyAMA0
lrwxrwxrwx 1 root root 5 Sep 24 21:17 /dev/serial1 -> ttyS0
pi#azure-iot-test:~ $ sudo docker exec hub-nodered ls -l /dev/ser*
ls: /dev/serial0: No such file or directory
ls: /dev/serial1: No such file or directory
Any ideas?
Followup:
Additional things I have tried the following ideas gleaned from here:
Adding "User": "node-red" to the root of the Container Create Options
Adding "User": "root" to the root of the Container Create Options
Adding "GroupAdd": "dialout" to "HostConfig": {...} in the Container Create Options
Followup #2
While I still can't interact with /dev/serial0, I am able to interact with /dev/ttyAMA0 using the following Container Create Options:
{
"HostConfig": {
"GroupAdd": [
"dialout"
],
"PortBindings": {
"1880/tcp": [
{
"HostPort": "80"
}
]
},
"Devices": [
{
"PathOnHost": "/dev/serial0",
"PathInContainer": "/dev/serial0",
"CgroupPermissions": "rwm"
},
{
"PathOnHost": "/dev/ttyAMA0",
"PathInContainer": "/dev/ttyAMA0",
"CgroupPermissions": "rwm"
}
]
}
}
The noteworthy items appear to be:
I didn't need "Privileged": true in "HostConfig"
I don't seem to need a "User" added
I needed "GroupAdd": ["dialout"] in "HostConfig"
So, while it's satisfying that I can interact with a serial device as I wanted to, it seems odd that I can't interact with /dev/serial0, which seems like it's "the recommended way" from the reading I've done.
Thanks to help and insight from Raymond Mouthaan over on the very helpful Node-RED Slack channel, I found my way to this Container Create Options providing me access to /dev/serial0:
{
"User": "node-red:dialout",
"HostConfig": {
"PortBindings": {
"1880/tcp": [
{
"HostPort": "80"
}
]
},
"Devices": [
{
"PathOnHost": "/dev/serial0",
"PathInContainer": "/dev/serial0",
"CgroupPermissions": "rwm"
}
]
}
}
This is different than the partial solution I found in "Followup #2" above in that I now do get access to /dev/serial0 as desired.
UPDATE: I originally posted this answer using "User": "root:dialout" rather than the "User": "node-red:dialout" you currently see above.
The first working solution was with "User": "root:root", but it seemed good to me to constrain to just the devices that are called out in Devices, which root:dialout seemed to do.
But I wondered whether I should be concerned security-wise with running as root at all.
Then I tried using node-red:dialout and it seems to be working perfectly, so I've updated the Container Create Options above to be what I think is the best answer.

Youtube ContentID getting ownership info through the API using AppsScript

I am trying to get get ownership information against AssetIDs through the Youtube ContentID API.
I can see the data that I need through the API Explorer but cant seem to drill down the data using dot notation.
Here is the output from the API explorer:
{
"kind": "youtubePartner#asset",
"id": "A146063471697194",
"type": "music_video",
"ownership": {
"kind": "youtubePartner#rightsOwnership",
"general": [
{
"ratio": 100,
"owner": "Indmusic",
"type": "exclude"
}
]
},
"ownershipEffective": {
"kind": "youtubePartner#rightsOwnership",
"general": [
{
"ratio": 100,
"owner": "Indmusic",
"type": "exclude"
}
]
}
}
When accessing the "owner" I receive undefined instead of the listed value.
var url2 = _.sprintf('https://www.googleapis.com/youtube/partner/v1/assets/%s?fetchMetadata=effective&fetchOwnership=effective&key=%s',id,API_KEY);
var result2 = JSON.parse(UrlFetchApp.fetch(url2, getUrlFetchOptions()).getContentText());
Logger.log(result2.ownership.general.owner);
returns undefined
I have tried both ownershipEffective and ownership and they are both undefined.
I can log data from result2.ownership.general but nothing below that.
You can tell that general is an array by the [brackets] in:
"general": [
{
"ratio": 100,
"owner": "Indmusic",
"type": "exclude"
}
]
Try:
Logger.log(result2.ownership.general[0].owner);
general, having been declared an array, requires a position [0] even though there is only 1 item in the array.

Resources