Using CDK to grant permissions to Dead Letter Queue on SNS Topic Subscription - aws-cdk

I'm trying to set up an SNS topic, with a subscription to a Queue. I want to put a Dead Letter Queue onto to the SNS Subscription.
This deploys OK, however in the AWS console, when I open the subscription, I see the error "Couldn't check Amazon SQS queue permissions. Make sure that the queue exists and that your account has permission to read the attributes of the queue".
Do I need to somehow grant write permission for SNSTopic to TopicDLQ?
export class SNSToSQSConstruct extends Construct {
public readonly TopicDLQ: IQueue
public readonly SQSQueue: IQueue
public readonly SNSTopic: ITopic
constructor(scope: Construct, id: string) {
super(scope, id);
this.TopicDLQ = new Queue(this, `${id}_TopicDLQ`, {
visibilityTimeout: cdk.Duration.seconds(300),
});
this.SQSQueue = new Queue(this, `${id}_Queue`, {
visibilityTimeout: cdk.Duration.seconds(300),
});
this.SNSTopic = new Topic(this, `${id}_Topic`, {
fifo: false, // fifo support 300tps, standard support almost unlimited
topicName: id,
});
var subscription = this.SNSTopic.addSubscription(new SqsSubscription(this.SQSQueue, {
rawMessageDelivery: true,
deadLetterQueue: this.TopicDLQ
}));
// error Subscription is not IGrantable
//this.TopicDLQ.grantSendMessages(subscription);
}
}

I think you can do this with addToResourcePolicy
import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import { IQueue, Queue } from 'aws-cdk-lib/aws-sqs'
import { ITopic, Topic } from 'aws-cdk-lib/aws-sns'
import { SqsSubscription } from 'aws-cdk-lib/aws-sns-subscriptions'
import { ServicePrincipal, PolicyStatement, Effect} from 'aws-cdk-lib/aws-iam'
export class SNSToSQSConstruct extends Construct {
public readonly TopicDLQ: IQueue
public readonly SQSQueue: IQueue
public readonly SNSTopic: ITopic
constructor(scope: Construct, id: string) {
super(scope, id);
this.TopicDLQ = new Queue(this, `${id}_TopicDLQ`, {
visibilityTimeout: cdk.Duration.seconds(300),
});
this.SQSQueue = new Queue(this, `${id}_Queue`, {
visibilityTimeout: cdk.Duration.seconds(300),
});
this.SNSTopic = new Topic(this, `${id}_Topic`, {
fifo: false, // fifo support 300tps, standard support almost unlimited
topicName: id,
});
var subscription = this.SNSTopic.addSubscription(new SqsSubscription(this.SQSQueue, {
rawMessageDelivery: true,
deadLetterQueue: this.TopicDLQ
}));
this.TopicDLQ.addToResourcePolicy(
new PolicyStatement({
effect: Effect.ALLOW,
principals: [new ServicePrincipal('sns.amazonaws.com')],
actions: ["sqs:SendMessage"],
resources: [this.TopicDLQ.queueArn],
conditions: {
ArnEquals: {
"aws:SourceArn": this.SNSTopic.topicArn,
},
},
})
);
}
}

Related

type 'Null' is not a subtype of type 'Future<DataState<List<CustomerEntity>>>

I'm working on a test CRUD application using Bloc for state management and sqflite database. I was asked to implement BDD testing for the app but I have no idea about BDD testing. From what I've learned so far I tryed to implement a simple scenario for the start which is landing on the home screen, but I'm getting this error when I run the test.
Also I don't know exactly how to mock my database to test the four main Create, Read, Update, and Delete functionalities.
I'm using getIt for dependency injection, Mocktail, and bdd_widget_test.
It is the scenario that I wrote in the .feature file:
Feature: Add Feature
Scenario: Landing on the home screen
Given the app is running
Then I see enabled elevated button
And it's the logic for the test.dart file where I get the mentioned exception:
class MockGetAllCustomers extends Mock implements GetAllCustomersUsecase {}
class MockAddCustomer extends Mock implements AddCustomerUsecase {}
class MockUpdateCustomer extends Mock implements UpdateCustomerUsecase {}
class MockDeleteCustomer extends Mock implements DeleteCustomerUsecase {}
class MockDbHelper extends Mock implements DBHelper {}
void main() {
final GetIt getIt = GetIt.instance;
late CustomersBloc bloc;
late MockGetAllCustomers mockGetAllCustomers;
late MockAddCustomer mockAddCustomer;
late MockUpdateCustomer mockUpdateCustomer;
late MockDeleteCustomer mockDeleteCustomer;
late MockDbHelper dbHelper;
late Database database;
CustomerEntity customer = CustomerEntity(
firstName: 'firstName',
lastName: 'lastName',
dateOfBirth: 'dateOfBirth',
phoneNumber: 'phoneNumber',
email: 'email',
bankAccountNumber: 'bankAccountNumber');
setUpAll(() async {
// Initialize FFI
sqfliteFfiInit();
database = await databaseFactoryFfi.openDatabase(inMemoryDatabasePath);
await database.execute(
'CREATE TABLE $customersTable($colId INTEGER PRIMARY KEY AUTOINCREMENT, $colFirstName TEXT, $colLastName TEXT, $colDateOfBirth TEXT, $colPhoneNumber TEXT, $colEmail TEXT, $colAccountNum TEXT)');
dbHelper = MockDbHelper();
dbHelper.database = database;
databaseFactory = databaseFactoryFfi;
});
setUp(() {
mockGetAllCustomers = MockGetAllCustomers();
mockAddCustomer = MockAddCustomer();
mockUpdateCustomer = MockUpdateCustomer();
mockDeleteCustomer = MockDeleteCustomer();
bloc = CustomersBloc(mockGetAllCustomers, mockAddCustomer,
mockUpdateCustomer, mockDeleteCustomer);
getIt.registerFactory(() => bloc);
});
group('''Add Feature''', () {
testWidgets('''Landing on the home screen''', (tester) async {
await theAppIsRunning(tester);
await iSeeEnabledElevatedButton(tester);
});
});
}
It's a part of the exception message I'm receiving:
type 'Null' is not a subtype of type 'Future<DataState<List<CustomerEntity>>>'
package:mc_crud_test/features/customer_feature/domain/usecases/get_all_customers_usecase.dart 11:43 MockGetAllCustomers.execute
package:mc_crud_test/features/customer_feature/presentation/bloc/bloc/customers_bloc.dart 43:60 new CustomersBloc.<fn>
package:bloc/src/bloc.dart 226:26 Bloc.on.<fn>.handleEvent
package:bloc/src/bloc.dart 235:9 Bloc.on.<fn>
dart:async
And it's my database class:
String customersTable = 'customers_table';
String colId = 'id';
String colFirstName = 'firstName';
String colLastName = 'lastName';
String colDateOfBirth = 'dateOfBirth';
String colPhoneNumber = 'phoneNumber';
String colEmail = 'email';
String colAccountNum = 'bankAccountNumber';
//
class DBHelper {
//
Database database;
DBHelper({required this.database});
//Database initialization and creation
static Future<Database> initDatabase() async {
//
final dbPath = await sql.getDatabasesPath();
return await sql.openDatabase(
path.join(dbPath, 'customers.db'),
onCreate: (db, version) {
return db.execute(
'CREATE TABLE $customersTable($colId INTEGER PRIMARY KEY AUTOINCREMENT, $colFirstName TEXT, $colLastName TEXT, $colDateOfBirth TEXT, $colPhoneNumber TEXT, $colEmail TEXT, $colAccountNum TEXT)');
},
version: 1,
);
}
//
//find a customer by firstName, lastName, dateOfBirth, and email
Future<bool> findCustomer(CustomerEntity customer) async {
//
List<Map<String, dynamic>> map = await database.query(customersTable,
columns: [colFirstName, colLastName, colDateOfBirth, colEmail],
where:
'$colFirstName = ? OR $colLastName = ? OR $colDateOfBirth = ? OR $colEmail = ?',
whereArgs: [
customer.firstName,
customer.lastName,
customer.dateOfBirth,
customer.email
]);
if (map.isEmpty) {
return false;
} else {
return true;
}
}
// Add a customer to the Database
Future<int> insertCustomer(CustomerEntity customer) async {
//
final id = await database.insert(
customersTable,
customer.toMap(),
conflictAlgorithm: ConflictAlgorithm.replace,
);
return id;
}
// Get the list of customers
Future<List<CustomerEntity>> getAllCustomers() async {
//
try {
final List<Map<String, Object?>> queryResult = await database.query(
customersTable,
orderBy: colId,
);
return queryResult.isEmpty
? []
: queryResult.map((e) => CustomerEntity.fromMapObject(e)).toList();
} catch (e) {
print('Error reading database : $e');
return [];
}
}
//
// Delete a Customer
Future<int> deleteCustomer(int id) async {
return await database
.delete(customersTable, where: '$colId = ?', whereArgs: [id]);
}
//
//Update a Customer
Future<bool> updateCustomer(CustomerEntity customer) async {
try {
final count = await database.update(customersTable, customer.toMap(),
where: '$colId = ?', whereArgs: [customer.id]);
if (count == 1) {
return true;
} else {
return false;
}
} catch (e) {
print('Failed to update the customer: $e');
return false;
}
}
}

NestJS - inject service into typeorm migration

I would like to inject a service into a typeorm migration, so that I can perform data migration based on some logic within a service:
import { MigrationInterface, QueryRunner, Repository } from 'typeorm';
import { MyService } from '../../services/MyService.service';
import { MyEntity } from '../../entities/MyEntity.entity';
export class MyEntityMigration12345678
implements MigrationInterface
{
name = 'MyEntityMigration12345678';
constructor(
private readonly myService: MyService,
) {}
public async up(queryRunner: QueryRunner): Promise<void> {
const myEntityRepository: Repository<MyEntity> =
queryRunner.connection.getRepository<MyEntity>(MyEntity);
const entities = await myEntityRepository.findBy({
myColumn: '',
});
for (const entity of entities) {
const columnValue = this.myService.getColumnValue(myEntity.id);
await myEntityRepository.save({
...entity,
myColumn: columnValue,
});
}
}
// ...
}
Nevertheless
myService is undefined, and
myEntityRepository.findBy(.) gets stuck.
How can I do a migration based on business logic?
Thanks!
One option would be to write whatever query myService.getColumn value does inside your migration. If you're hell bent on using Nest's DI inside your migration then you could do something like this:
import { NestFactory } from '#nestjs/core';
import { MigrationInterface, QueryRunner, Repository } from 'typeorm';
import { AppModule } from '../../app.module'; // assumed path
import { MyService } from '../../services/MyService.service';
import { MyEntity } from '../../entities/MyEntity.entity';
export class MyEntityMigration12345678
implements MigrationInterface
{
name = 'MyEntityMigration12345678';
public async up(queryRunner: QueryRunner): Promise<void> {
const myEntityRepository: Repository<MyEntity> =
queryRunner.connection.getRepository<MyEntity>(MyEntity);
const entities = await myEntityRepository.findBy({
myColumn: '',
});
const appCtx = await NestFactory.createApplicationContext(AppModule);
const myService = app.get(MyService, { strict: false });
for (const entity of entities) {
const columnValue = myService.getColumnValue(myEntity.id);
await myEntityRepository.save({
...entity,
myColumn: columnValue,
});
}
await app.close();
}
// ...
}
You can't use injection inside the migration because the class itself is managed by TypeORM. You can, as shown above, create a NestApplicationContext instance and get the MyService instance from that. This only works, by the way, if MyService is REQUEST scoped

AWS CDK | Create a REST API spanning multiple CDK Stacks

We are using AWS CDK to create our Serverless REST API. However, there are a large number of endpoints and sometimes we have to destroy and redeploy our stack. To prevent the REST API URL from changing with each deployment, I am planning to create the API GATEWAY in one stack and add methods and resources in a separate stack. How can I refer the created rest API in a separate stack?
Tried to implement something from https://github.com/aws/aws-cdk/issues/3705, but all of the resources(API Gateway, resource and methods) are being pushed in a single stack instead of API Gateway in one stack and the resources in other stack.
Relevant codes snippets are provided below:
bts-app-cdk.ts
const first = new FirstStack(app, 'FirstStack', {
env: {
region: 'us-east-1',
account: '1234567890',
}
});
const second = new SecondStack(app, 'SecondStack', {
apiGateway: first.apiGateway,
env: {
region: 'us-east-1',
account: '1234567890',
}
});
second.addDependency(first)
first-stack.ts
export class FirstStack extends cdk.Stack {
public readonly apiGateway: apig.IResource;
constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const apiGateway = new apig.RestApi(this, 'BooksAPI', {
restApiName:'Books API',
})
apiGateway.root.addMethod('GET');
this.apiGateway = apiGateway.root;
}
}
second-stack
export interface SecondStackProps extends cdk.StackProps {
readonly apiGateway: apig.IResource;
}
export class SecondStack extends cdk.Stack {
constructor(scope: cdk.Construct, id: string, props: SecondStackProps) {
super(scope, id, props);
props.apiGateway.addMethod('ANY')
}
}
Seems that there is no way at the moment except using Cfn constructs, here's the github issue to track https://github.com/aws/aws-cdk/issues/1477
You have to set the deploy prop to false when instantiating your api-gw.
Then you can pass the restApiId and rootResourceId as enviroment variables to stacks.
Finally you can use a Deployment to deploy everything.
interface ResourceNestedStackProps extends NestedStackProps {
readonly restApiId: string;
readonly rootResourceId: string;
}
export class FirstStack extends cdk.Stack {
public readonly methods: Method[];
constructor(scope: cdk.Construct, props: ResourceNestedStackProps) {
super(scope, 'first-stack', props);
// get a hold of the rest api from attributes
const api = RestApi.fromRestApiAttributes(this, 'RestApi', {
restApiId: props.restApiId,
rootResourceId: props.rootResourceId,
})
// REPEAT THIS FOR METHODS
const method = api.root.addMethod('GET');
this.methods.push(method)
// ---
}
}
Then in the root stack file,
class RootStack extends Stack {
constructor(scope: Construct) {
const restApi = new RestApi(this, 'RestApi', {
deploy: false,
});
restApi.root.addMethod('ANY');
const firstStack = new FirstStack(this, {
restApiId: restApi.restApiId,
rootResourceId: restApi.restApiRootResourceId,
});
new DeployStack(this, {
restApiId: restApi.restApiId,
methods: [...firstStack.methods],
})
}
}
The deploy stack is where you deploy your API:
interface DeployStackProps extends NestedStackProps {
readonly restApiId: string;
readonly methods?: Method[];
}
class DeployStack extends NestedStack {
constructor(scope: Construct, props: DeployStackProps) {
super(scope, 'deploy-stack', props);
const deployment = new Deployment(this, 'Deployment', {
api: RestApi.fromRestApiId(this, 'RestApi', props.restApiId),
});
if (props.methods) {
for (const method of props.methods) {
deployment.node.addDependency(method);
}
}
new Stage(this, 'Stage', { deployment });
}
}
Refer this for more details: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.NestedStack.html

Background Thread that uses ApplicaitonDBContext

I am trying to wire up a background thread that will update the database once an hour from Active Directory. I am not sure how to pass the current
public void ConfigureServices(IServiceCollection services)
{
services.Configure<CookiePolicyOptions>(options =>
{
// This lambda determines whether user consent for non-essential cookies is needed for a given request.
options.CheckConsentNeeded = context => false;
options.MinimumSameSitePolicy = SameSiteMode.None;
});
// Add framework services.
services.AddDbContext<ApplicationDbContext>(options =>
options.UseSqlServer(Configuration.GetConnectionString("Connection")));
services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_1).AddSessionStateTempDataProvider();
services.AddSession();
services.AddHttpContextAccessor();
services.AddSingleton(Configuration);
services.AddScoped<IAppDbRepository, AppDbRepository>();
services.AddScoped<IActiveDirectoryUtility, ActiveDirectoryUtility>();
services.AddScoped<IActiveDirectoryManager, ActiveDirectoryManager>();
services.AddHostedService<LdapManager>();
services.AddScoped<ILdapManager, LdapManager>();
}
In the LdapManager class I would like to call the UpdateUsers method every hour:
public class LdapManager : ILdapManager, IHostedService
{
private IConfiguration _configuration = null;
private Logging _logger;
private List<string> ldapConnectorForDirectoryEntries = new List<string>();
public LdapManager(IConfiguration configuration)
{
_configuration = configuration;
UpdateUsers();
SyncActiveDirectoryUsers();
}
public void SyncActiveDirectoryUsers()
{
try
{
using (var waitHandle = new AutoResetEvent(false))
{
ThreadPool.RegisterWaitForSingleObject(waitHandle, (state, timeout) => { UpdateUsers(); }, null, TimeSpan.FromHours(1), false);
}
}
catch
{
throw;
}
}
}
The UpdateUsers() method should be able to call the applicationDBContext.SaveChanges() method.
How can I ensure that the LDAP manger class can use the Application DB context?
You probably want class LdapManager : BackgroundService, ILdapManager
BackgroundService is .NET Core 2.1, there is a code sample available for core 2.0
Inject IServiceScopeFactory and override Task ExecuteAsync( ), run a while loop there.
while(!stoppingToken.IsCancellationRequested)
{
using (var scope = _serviceScopeFactory.CreateScope())
{
var context = scope.ServiceProvider.GetRequiredService<ApplicationDbContext>();
...; // do your stuff
}
await Task.Delay(myConfig.BackgroundDelay, stoppingToken);
}
And here is a good read about this on MSDN, including the code sample for 2.0
For accessing ApplicationDbContext from HostedService.
DbHostedService
public class DbHostedService : IHostedService
{
private readonly ILogger _logger;
public DbHostedService(IServiceProvider services,
ILogger<DbHostedService> logger)
{
Services = services;
_logger = logger;
}
public IServiceProvider Services { get; }
public Task StartAsync(CancellationToken cancellationToken)
{
_logger.LogInformation("Consume Scoped Service Hosted Service is starting.");
DoWork();
return Task.CompletedTask;
}
private void DoWork()
{
_logger.LogInformation("Consume Scoped Service Hosted Service is working.");
using (var scope = Services.CreateScope())
{
var context = scope.ServiceProvider.GetRequiredService<ApplicationDbContext>();
var user = context.Users.LastOrDefault();
_logger.LogInformation(user?.UserName);
}
}
public Task StopAsync(CancellationToken cancellationToken)
{
_logger.LogInformation("Consume Scoped Service Hosted Service is stopping.");
return Task.CompletedTask;
}
}
Register DbHostedService
services.AddHostedService<DbHostedService>();

How to handle TypeORM entity field unique validation error in NestJS?

I've set a custom unique validator decorator on my TypeORM entity field email. NestJS has dependency injection, but the service is not injected.
The error is:
TypeError: Cannot read property 'findByEmail' of undefined
Any help on implementing a custom email validator?
user.entity.ts:
#Column()
#Validate(CustomEmail, {
message: "Title is too short or long!"
})
#IsEmail()
email: string;
My CustomEmail validator is
import {ValidatorConstraint, ValidatorConstraintInterface,
ValidationArguments} from "class-validator";
import {UserService} from "./user.service";
#ValidatorConstraint({ name: "customText", async: true })
export class CustomEmail implements ValidatorConstraintInterface {
constructor(private userService: UserService) {}
async validate(text: string, args: ValidationArguments) {
const user = await this.userService.findByEmail(text);
return !user;
}
defaultMessage(args: ValidationArguments) {
return "Text ($value) is too short or too long!";
}
}
I know I could set unique in the Column options
#Column({
unique: true
})
but this throws a mysql error and the ExceptionsHandler that crashes my app, so I can't handle it myself...
Thankx!
I can propose 2 different approaches here, the first one catches the constraint violation error locally without additional request, and the second one uses a global error filter, catching such errors in the entire application. I personally use the latter.
Local no-db request solution
No need to make additional database request. You can catch the error violating the unique constraint and throw any HttpException you want to the client. In users.service.ts:
public create(newUser: Partial<UserEntity>): Promise<UserEntity> {
return this.usersRepository.save(newUser).catch((e) => {
if (/(email)[\s\S]+(already exists)/.test(e.detail)) {
throw new BadRequestException(
'Account with this email already exists.',
);
}
return e;
});
}
Which will return:
Global error filter solution
Or even create a global QueryErrorFilter:
#Catch(QueryFailedError)
export class QueryErrorFilter extends BaseExceptionFilter {
public catch(exception: any, host: ArgumentsHost): any {
const detail = exception.detail;
if (typeof detail === 'string' && detail.includes('already exists')) {
const messageStart = exception.table.split('_').join(' ') + ' with';
throw new BadRequestException(
exception.detail.replace('Key', messageStart),
);
}
return super.catch(exception, host);
}
}
Then in main.ts:
async function bootstrap() {
const app = await NestFactory.create(/**/);
/* ... */
const { httpAdapter } = app.get(HttpAdapterHost);
app.useGlobalFilters(new QueryErrorFilter(httpAdapter));
/* ... */
await app.listen(3000);
}
bootstrap();
This will give generic $table entity with ($field)=($value) already exists. error message. Example:
I have modified my code. I am checking the uniqueness of username/email in the user service (instead of a custom validator) and return an HttpExcetion in case the user is already inserted in the DB.
The easiest solution!
#Entity()
export class MyEntity extends BaseEntity{
#Column({unique:true}) name:string;
}
export abstract class BaseDataService<T> {
constructor(protected readonly repo: Repository<T>) {}
private async isUnique(t: any) {
const uniqueColumns = this.repo.metadata.uniques.map(
(e) => e.givenColumnNames[0]
);
for (const u of uniqueColumns) {
const count = await this.repo.count({ where: { [u]: ILike(t[u]) } });
if (count > 0) {
throw new UnprocessableEntityException(`${u} must be unique!`);
}
}
}
async save(body: DeepPartial<T>) {
await this.isUnique(body);
try {
return await this.repo.save(body);
} catch (err) {
throw new UnprocessableEntityException(err.message);
}
}
async update(id: number, updated: QueryDeepPartialEntity<T>) {
await this.isUnique(updated)
try {
return await this.repo.update(id, updated);
} catch (err) {
throw new UnprocessableEntityException(err.message);
}
}
}
An approach that works for modern version of NestJS which is based in Daniel Kucal's answer and actually returns the error to the frontend when calling the JSON API is the following:
import {
Catch,
ArgumentsHost,
BadRequestException,
HttpException,
} from '#nestjs/common';
import { BaseExceptionFilter } from '#nestjs/core';
import { QueryFailedError } from 'typeorm';
type ExceptionType = { detail: string; table: string };
#Catch(QueryFailedError)
export class QueryErrorFilter extends BaseExceptionFilter<
HttpException | ExceptionType
> {
public catch(exception: ExceptionType, host: ArgumentsHost): void {
const { detail = null } = exception || {};
if (
!detail ||
typeof detail !== 'string' ||
// deepcode ignore AttrAccessOnNull: <False positive>
!detail.includes('already exists')
) {
return super.catch(exception, host);
} // else
/**
* this regex transform the message `(phone)=(123)` to a more intuitive `with phone: "123"` one,
* the regex is long to prevent mistakes if the value itself is ()=(), for example, (phone)=(()=())
*/
const extractMessageRegex =
/\((.*?)(?:(?:\)=\()(?!.*(\))(?!.*\))=\()(.*?)\)(?!.*\)))(?!.*(?:\)=\()(?!.*\)=\()((.*?)\))(?!.*\)))/;
const messageStart = `${exception.table.split('_').join(' ')} with`;
/** prevent Regex DoS, doesn't treat messages longer than 200 characters */
const exceptionDetail =
exception.detail.length <= 200
? exception.detail.replace(extractMessageRegex, 'with $1: "$3"')
: exception.detail;
super.catch(
new BadRequestException(exceptionDetail.replace('Key', messageStart)),
host,
);
}
}
Also, not forgetting main.ts:
async function bootstrap() {
const app = await NestFactory.create(/**/);
/* ... */
const { httpAdapter } = app.get(HttpAdapterHost);
app.useGlobalFilters(new QueryErrorFilter(httpAdapter));
/* ... */
await app.listen(3000);
}
bootstrap();

Resources