I have declared a mapping named StageMap in my sam.yaml file:
AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Parameters:
ProjectName:
Type: String
SubProjectName:
Type: String
Stage:
Type: String
AllowedValues:
- dev
- test
- preprod
- prod
...
Mappings:
StageMap:
dev:
AuthorizerArn: !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:auth-bk-main-dev-AuthorizerFunction-1RR2YJ5STBUB6/invocations
test:
AuthorizerArn: !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:auth-bk-main-test-AuthorizerFunction-UQ1EQ2SP5W6G/invocations
preprod:
AuthorizerArn: !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:auth-bk-main-preprod-AuthorizerFunction-UQ1W6EQ2SP5G/invocations
prod:
AuthorizerArn: !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:auth-bk-main-prod-AuthorizerFunction-5STBUB61RR2YJ/invocations
I would like to use this mapping in my swagger.yaml I have tried the following:
...
x-amazon-apigateway-authorizer:
type: request
authorizerUri:
Fn::FindInMap:
- 'StageMap'
- Ref: 'Stage'
- 'AuthorizerArn
I also tried this solution but I got an error Every Mappings attribute must be a String or a List.
Can you please let me know how to access one of the values in the mapping in the swagger.yaml? Thanks!
I found the following in the AWS SAM docs:
You cannot include parameters, pseudo parameters, or intrinsic functions in the Mappings section.
So I changed:
AuthorizerArn: !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:auth-bk-main-dev-AuthorizerFunction-1RR2YJ5STBUB6/invocations
For:
AuthorizerFunctionName: auth-bk-main-dev-AuthorizerFunction-1RR2YJ5STBUB6
And in the swagger.yaml I used the following:
x-amazon-apigateway-authorizer:
type: request
authorizerUri:
Fn::Sub:
- arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${AuthorizerFunctionName}/invocations
- AuthorizerFunctionName:
Fn::FindInMap:
- 'StageMap'
- Ref: 'Stage'
- 'AuthorizerFunctionName'
Related
I deployed an envoy as a side car to manage oauth2. Everything work fine for all the resources and the client is redirected to the OIDC in order to authenticate.
Here is a part of my conf (managed in a Helm chart):
- name: envoy.filters.network.http_connection_manager
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
access_log:
- name: envoy.access_loggers.file
typed_config:
"#type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/stdout
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: my-service
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: my-service
http_filters:
- name: envoy.filters.http.oauth2
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.oauth2.v3.OAuth2
config:
token_endpoint:
cluster: {{ .Values.back.envoy.oidc.name }}
uri: https://{{ .Values.back.envoy.oidc.address }}/oidc/token
timeout: 5s
authorization_endpoint: https://{{ .Values.back.envoy.oidc.address }}/oidc/authorize
redirect_uri: "%REQ(x-forwarded-proto)%://%REQ(:authority)%/oidc/callback"
redirect_path_matcher:
path:
exact: /oidc/callback
signout_path:
path:
exact: /oidc/signout
credentials:
client_id: {{ required "back.envoy.oidc.client_id is required" .Values.back.envoy.oidc.client_id }}
token_secret:
name: token
sds_config:
resource_api_version: V3
path: "/etc/envoy/token-secret.yaml"
hmac_secret:
name: hmac
sds_config:
resource_api_version: V3
path: "/etc/envoy/hmac-secret.yaml"
forward_bearer_token: true
# (Optional): defaults to 'user' scope if not provided
auth_scopes:
- user
- openid
- email
- homelan_devices_read
- homelan_topology_read
- homelan_devices_write
# (Optional): set resource parameter for Authorization request
#resources:
#- oauth2-resource
#- http://example.com
- name: envoy.filters.http.router
typed_config: {}
Now I'd like that some of the exposed resources don't need to be authenticated.
I see in the doc the Oauth filter doc "Leave this empty to disable OAuth2 for a specific route, using per filter config." (see https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/oauth2/v3/oauth.proto#envoy-v3-api-msg-extensions-filters-http-oauth2-v3-oauth2config)
This phrase make me think that it may be possible.
I tried to manage it changing my conf throught virtual_hosts this way :
virtual_hosts:
- name: no-oauth
domains: ["*"]
typed_per_filter_config:
envoy.filters.http.oauth2:
"#type": type.googleapis.com/envoy.extensions.filters.http.oauth2.v3.OAuth2
routes:
- match:
prefix: "/api/v1/myResource1"
route:
cluster: my-service
- name: my-service
domains: ["*"]
routes:
- match:
prefix: "/api/v1/myResource2"
route:
cluster: my-service
I have the error : [critical][main] [source/server/server.cc:117] error initializing configuration '/etc/envoy/envoy.yaml': The filter envoy.filters.http.oauth2 doesn't support virtual host-specific configurations
Any idea ? Did someone implement Envoy OAuth2 filter with disabled routes ?
After looking at my envoy logs, I realized that path is know as header ":path".
The pass_through_matcher math the header.
Then only adding:
pass_through_matcher:
- name: ":path"
prefix_match: "/healthz"
- name: ":path"
prefix_match: "/api/v1/myResource1"
in my conf without the lua filter (see my previous answer) it works.
For information, I found a workaround:
I added a LUA filter before my OAuth2 one:
- name: envoy.filters.http.lua
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
inline_code: |
function envoy_on_request(request_handle)
request_handle:headers():add("X-Path", request_handle:headers():get(":path"))
end
In order to add the path in a header.
Then I can use this element of conf Oauth2:
pass_through_matcher
(repeated config.route.v3.HeaderMatcher) Any request that matches any of the provided matchers will be passed through without OAuth validation.
So I add this to my OAuth2 filter:
pass_through_matcher:
- name: "X-path"
prefix_match: "/healthz"
- name: "X-path"
prefix_match: "/api/v1/myResource1"
Then my /api/v1/myResource1 requests (and healthz also) don't need authentication (are disable from the OAuth2) while my /api/v1/myResource2 requests need it.
I still have got the unanswered question:
What do the OAuth filter doc means with :"Leave this empty to disable OAuth2 for a specific route, using per filter config."
I'm having a hard time using custom Elasticsearch ingest pipelines with Filebeat's Docker autodiscovery. I've started out with custom processors in my filebeat.yml file, however I would prefer to shift this to custom ingest pipelines I've created.
Firstly, here is my configuration using custom processors that works to provide custom grok-like processing for my Servarr app Docker containers (identified by applying a label to them in my docker-compose.yml file). The processor copies the 'message' field to 'log.original', uses dissect to extract 'log.level', 'log.logger' and overwrite 'message'. The final processor is a JavaScript function used to convert the log.level to lowercase (overkill perhaps, but humour me).
Filebeat configuration:
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 60s
filebeat.autodiscover:
providers:
- type: docker
hints.enabled: true
json.keys_under_root: true
processors:
- if:
equals:
docker.container.labels.co_elastic_logs/custom_processor: servarr
then:
- copy_fields:
fields:
- from: message
to: log.original
fail_on_error: false
ignore_missing: true
- dissect:
tokenizer: "[%{log.level}] %{log.logger}: %{message}"
field: message
target_prefix: ""
overwrite_keys: true
ignore_failure: true
- script:
lang: javascript
id: lowercase
source: >
function process(event) {
var level = event.Get("log.level");
if(level != null) {
event.Put("log.level", level.toString().toLowerCase());
}
}
output.elasticsearch:
hosts: 'elasticsearch:9200'
username: 'elastic'
password: '*************'
setup.kibana.host: 'kibana:5601'
logging.json: true
logging.metrics.enabled: false
Excerpt from docker-compose.yml file...
lidarr:
image: ghcr.io/linuxserver/lidarr:latest
container_name: lidarr
labels:
co.elastic.logs/custom_processor: "servarr"
And an example log line (in json):
{"log":"[Info] DownloadDecisionMaker: Processing 100 releases \n","stream":"stdout","time":"2021-08-07T10:10:49.125702754Z"}
This works well, and achieves my aims of extracting fields, but ideally I'd like to use Elasticsearch's (more powerful) ingest pipelines instead, and live with a cleaner filebeat.yml, so I created a working ingest pipeline "filebeat-7.13.4-servarr-stdout-pipeline" like so (ignore the fact that for now, this only does the grokking):
[
{
"grok": {
"field": "message",
"patterns": [
"\\[%{LOGLEVEL:log.level}\\] %{WORD:log.logger}: %{GREEDYDATA:message}"
],
"trace_match": true,
"ignore_missing": true
}
}
]
I tested the pipeline against existing documents (not ones that have had my custom processing applied, I should note). The pipeline worked against all the documents I tested it against in the Kibana interface.
So now I come to shift my Filebeat config to use this pipeline for containers with my custom_processor label. This is the filebeat.yml I came up with, which is apparently valid and works for the most part, but doesn't apply the grokking:
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 60s
filebeat.autodiscover:
providers:
- type: docker
hints.enabled: true
json.keys_under_root: true
appenders:
- type: config
condition.equals:
docker.container.labels.co_elastic_logs/custom_processor: servarr
config:
pipeline: filebeat-7.13.4-servarr-stdout-pipeline
output.elasticsearch:
hosts: 'elasticsearch:9200'
username: 'elastic'
password: '*************'
setup.kibana.host: 'kibana:5601'
logging.json: true
logging.metrics.enabled: false
If I use Filebeat's inbuilt modules for my other containers such as nginx, by using a label such as in this example below, the inbuild module pipelines are used:
nginx-repo:
image: nginx:latest
container_name: nginx-repo
mem_limit: 2048m
environment:
- VIRTUAL_HOST=repo.***.***.***,repo
- VIRTUAL_PORT=80
- HTTPS_METHOD=noredirect
networks:
- default
- proxy
labels:
co.elastic.logs/module: "nginx"
co.elastic.logs/fileset.stdout: "access"
co.elastic.logs/fileset.stderr: "error"
What am I doing wrong here? The logs still end up in Elasticsearch and Kibana, and are processed, but my grok isn't applied, new fields aren't created, and the 'message' field is unchanged.
EDIT: In response to one of the comments linking to a post on the elastic forums, which suggested both the path(s) and the pipeline need to be made explicit, I tried the following filebeat.yml autodiscovery excerpt, which also fails to work (but is apparently valid config):
filebeat.autodiscover:
providers:
- type: docker
hints.enabled: true
json.keys_under_root: true
appenders:
- type: config
condition:
equals:
docker.container.labels.co_elastic_logs/custom_processor: "servarr"
config:
- type: docker
containers:
ids:
- "${data.docker.container.id}"
stream: all
paths:
- /var/lib/docker/containers/${data.docker.container.id}/${data.docker.container.id}-json.log
pipeline: filebeat-7.13.4-servarr-stdout-pipeline
I tried with the docker.container.labels.co_elastic_logs/custom_processor value both quoted and unquoted. I have the same behaviour where the logs end up in Elasticsearch / Kibana, but they are processed as if they skipped my ingest pipeline.
We're using Kubernetes instead of Docker with Filebeat but maybe our config might still help you out.
We have autodiscover enabled and have all pod logs sent to a common ingest pipeline except for logs from any Redis pod which use the Redis module and send their logs to Elasticsearch via one of two custom ingest pipelines depending on whether they're normal Redis logs or slowlog Redis logs, this is configured in the following block:
All other detected pod logs get sent in to a common ingest pipeline using the following catch-all configuration in the "output" section:
Something else that we do is add the name of the ingest pipeline to ingested documents using the "set" processor:
This has proven to be really helpful when diagnosing whether or not a pipeline was actually executed when viewing an event document in Kibana.
I have a yaml file which has examples for request body. but swagger-ui springfox generated json file generated by springfox doesn't show the request body example, which makes it to show schema example.
Any workaround for displaying request body examples?
#openapi #swagger-ui #springfox
openapi: 3.0.2
info:
title: Rule APIs
description: Rule APi
version: 1.0.0
contact:
email: abc#gmail.com
servers:
- url: 'https://{server}/v1/rule'
variables:
server:
default: localhost
security:
- BasicAuth: []
paths:
#####################################################################################################
############################ Rule APIs ###############################
#####################################################################################################
/rule-set:
parameters:
- $ref: '#/components/parameters/XRequestIdHeader'
post:
summary: Create a rule set.
description: Create a rule set.
operationId: createRule
tags:
- Rule
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/RuleSet'
example:
name: My RuleSet
description: A new ruleSet
state: enabled
condition:
conditionType: ConditionAttributes
isNegate: false
dictionaryName: DEVICE
attributeName: attribute
operator: equals
attributeValue: attribute
serviceName: service
isProxy: false
responses:
'201':
$ref: '#/components/responses/BadRequest'
'400':
$ref: '#/components/responses/BadRequest'
json output
I have two serverless app which are sharing the same custom authorizer. Suddenly the import function in the second serverless.yml file stopped working.
The app is based on https://github.com/medwig/serverless-shared-authorizer
gateway.serverless
service: authorizer-stack
provider:
name: aws
runtime: nodejs12.x
region: ap-south-1
profile: xxx-dev
functions:
authorizer:
handler: handler.auth
test:
handler: handler.privateEndpoint
events:
- http:
path: /api/test
method: get
authorizer:
type: CUSTOM
authorizerId:
Ref: Authorizer
test2:
handler: handler.publicEndpoint
events:
- http:
path: /api/test/public
method: get
resources:
Resources:
AuthorizerPermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName:
Fn::GetAtt: AuthorizerLambdaFunction.Arn
Action: lambda:InvokeFunction
Principal:
Fn::Join: ["",["apigateway.", { Ref: "AWS::URLSuffix"}]]
Authorizer:
DependsOn:
- ApiGatewayRestApi
Type: AWS::ApiGateway::Authorizer
Properties:
Name: ${self:provider.stage}-Authorizer
RestApiId: { "Ref" : "ApiGatewayRestApi" }
Type: TOKEN
IdentitySource: method.request.header.Authorization
AuthorizerResultTtlInSeconds: 300
AuthorizerUri:
Fn::Join:
- ''
-
- 'arn:aws:apigateway:'
- Ref: "AWS::Region"
- ':lambda:path/2015-03-31/functions/'
- Fn::GetAtt: "AuthorizerLambdaFunction.Arn"
- "/invocations"
Outputs:
AuthorizerId:
Value:
Ref: Authorizer
Export:
Name: authorizerId
apiGatewayRestApiId:
Value:
Ref: ApiGatewayRestApi
Export:
Name: restApiId
apiGatewayRestApiRootResourceId:
Value:
Fn::GetAtt:
- ApiGatewayRestApi
- RootResourceId
Export:
Name: rootResourceId
products serverless
service: products-list
provider:
name: aws
runtime: nodejs12.x
region: ap-south-1
profile: xxx-dev
apiGateway:
restApiId:
Fn::ImportValue: authorizer-stack-dev-restApiId
restApiRootResourceId:
Fn::ImportValue: authorizer-stack-dev-rootResourceId
functions:
get-products:
handler: handler.getProducts
events:
- http:
path: /api/products
method: get
authorizer:
type: CUSTOM
authorizerId:
Fn::ImportValue: authorizer-stack-dev-authorizerId
I am getting the following errors at random
An error occurred: products-list-dev - No export named authorizer-stack-dev-restApiId found.
An error occurred: products-list-dev - No export named authorizer-stack-dev-rootResourceId found.
An error occurred: products-list-dev - No export named authorizer-stack-dev-authorizerId found.
What am I missing here?
serverless -v
Framework Core: 1.74.1
Plugin: 3.6.15
SDK: 2.3.1
Components: 2.31.10
From the shared authorizers I have configured in the past it is not necessary to go to the effort you have undergone. The documentation on the Serverless Framework site has a much simpler setup to achieve a shared authoriser and I will always go with the simplest solution possible: https://www.serverless.com/framework/docs/providers/aws/events/apigateway#share-authorizer
I would like to add an AWS resource conditionally based on presence of an env var. I tried serverless-cloudformation-parameter-setter but I get a generic error on deployment and I don't see what I need to do to fix it
I'm trying to deploy a simple lambda + SQS stack and if a env var is defined also subscribe the queue to the topic denoted by the env var - or if the var is not defined then not do that part at all, just the lambda and the queue
This is what I tried:
plugins:
- serverless-cloudformation-parameter-setter
provider:
name: aws
runtime: nodejs8.10
region: eu-west-1
functions:
update:
handler: index.update
events:
- sqs:
arn:
Fn::GetAtt:
- Queue
- Arn
custom:
cf-parameters:
SourceTopicArn: "${env:UPDATE_SNS_ARN}"
resources:
Parameters:
SourceTopicArn:
Type: string
Resources:
Queue:
Type: "AWS::SQS::Queue"
Subscription:
Type: "AWS::SNS::Subscription"
Condition: SourceTopicArn
Properties:
TopicArn:
Ref: SourceTopicArn
Endpoint:
Ref: Queue
The error I receive is: The CloudFormation template is invalid: Template format error: Unrecognized parameter type: string
If I remove all the parameter stuff it works fine
The Type has to be String, not string. See the supported parameter data types section in the docs.