Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 3 years ago.
Improve this question
So I've been trying to setup a BI tool with my Serverless backend i.e. Aurora MySQL.
Architecture as below:
Implementation on these lines: tutorial-link
Now, I'm exploring options for connecting this backend with a BI/Analytics tool for quick querying and visualisation purposes.
Since, our backend is a Aurora MySQL Serverless, the limitation would be to not be able to connect directly to the endpoint like how one could do with a provisioned one.
As stated here: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless.html
On similar lines, I'm trying to setup a AWS Direct Connect, which is currently in requested stage:
I need assistance whether this is right approach, if so, are there any tutorials for using Direct Connect with Aurora Serverless?
On the other hand, are there any bi tools anyone has successfully used with Serverless backends.
Need is to be able to get up and running in the shortest amount of time, to get quick inferences. And then move to ETLs or more complex use cases.
Any assistance much appreciated!
Thanks,
Arnab
Figured this out.
One needs to setup an Enterprise QuickSight Account to be able to connect to Private VPCs. The regular QuickSight Account expects the endpoints to be publicly available. While Serverless endpoints aren't.
Limitations: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless.html
Prerequisites: Enterprise QuickSight Account, Aurora Serverless cluster and make sure your cluster has active capacity units when creating the connection.
Step 1: In QuickSight, choose your profile icon at the top right of the screen, then choose "Manage QuickSight". From the menu at left, choose "Manage VPC connections".
Step 2: Choose "Add VPC connection" to add a new VPC connection.
Step 3: For "VPC connection name", type a unique descriptive name. This name doesn't need to be an actual VPC ID or name.
Step 4: Type the subnet ID for Subnet ID, and type the group ID for Security group ID. Make sure that the subnet and the security group are in the same VPC. Also, make sure you are accessing a VPC that is in the same AWS Region where you are creating Amazon QuickSight analyses. You can't use Amazon QuickSight in one AWS Region to connect to a subnet and security group that are in a different AWS Region.
Important note: Please make sure the Security group you use for the VPC connection has access to the Security group used by your Aurora Serverless cluster and vice versa.
Step 5: Review your choices, then choose "Create".
Step 6: Create a new Aurora Data source
Step 7: For the "Connection Type" option choose your newly created VPC connection from the drop down.
Step 8 Fill out the rest of the details on for the connection and click "Create Data Source".
You should now be able to select the table you wish to use in your analysis or write a custom SQL query to get the data.
Maybe helps someone in future.
Related
I've had a little dig through azure documentation but couldn't find a definitive answer.
I have an app service and an azure db sitting in the same resource group, and I am finding the site takes a long time to connect and get responses back from the database only in the hosted environment.
Is it possible to specify a localhost equivalent as they are in the same resource group, and would this make things any quicker?
Resource Group does not have any impact on the connectivity or latency of the application and the database. It is just to group the Azure resources together based on a Project/Envrionment.
There is no equivalent for resourcegroup or even appservice unless if you want to run your application in IIS or any other server.
If you really want to see what is causing the connectivity issue, i will recommend you to monitor the request and response using Azure Monitor.
I think you need to understand the cloud concepts first before trying out anything.
some days ago I was able to set up one of my apps to be connected to one of my database instances from the google cloud run service configuration form. However lately I notice two things:
I'm no longer able to select the database instance my service is/will be connected to.
On a service that is connected using this method I no longer see the database connection name. at the bottom on the details panel.
Is this a symptom that the database connections feature will disappear from the Google CloudRun settings?.
This seems like a useful case to use the Cloud SDK to confirm your Cloud Run service is able to communicate with Cloud SQL. This will help confirm if you have a UI problem or something deeper. This is especially important given the documentation states that the Console instructions are not available yet.
Cloud Run supports Cloud SQL via gcloud management using a special flag to associate a Cloud SQL instance with an individual service.
Once this is done, the Cloud SQL instance will be available to the Cloud Run service until it is explicitly removed.
You can verify this connection is in place by looking at the service description:
gcloud beta run services describe [SERVICE-NAME]
in the response, you should see the property run.googleapis.com/cloudsql-instances inside spec.runLatest.configuration.revisionTemplate.metadata.annotations.
As long as that annotation is present and contains your Cloud SQL instance connection name, your service should be able to connect to the SQL instance as documented (assuming your service has authorization to connect to the Cloud SQL instance)
i have several databases running in RDS Service.
I'd like to know the best pratice to grant access to developers to these DB.
I tought a solution using jenkins but i dont think this is the best option.
I am trying to avoid give some password to developers.
Hope you can help me.
As #ceejayoz mentioned you can create a few users with restricted privileges, for example an user who only can run selects on few schemas, another user who can update registers in a few tables.
I can share what we do and what I've seen. We do A and use B where it is easy.
A) Standard Users
For all databases, we have 3 standard users with the following suffixes (_dba, _rw, _ro). Those all have their own passwords using a strong password generator.
_dba is used to deploy scheme and has all rights
_rw is used by the application (CRUD on all tables, but can't modify scheme)
_ro only has R on all tables and generally given to developers
Note: Developers have access to a bastion used for port forwarding and proxycap. They can query the RDS endpoints from their own machines (DB Tools) going through socks proxy and bastion.
This is lazy method - since creation of users is done programmatically and we feel comfortable giving some developers read only access. They could write a bad query and slow down system, but they could do that with a specific user so not much different and the bastion logs tell me who really was in if I had to investigate.
B) UI
Simple web app with login (ideally MFA) - that provides a way to run queries. If only for reporting, ideally against R/O copy of system. Stackoverflow offers one themselves (https://data.stackexchange.com/).
What would be nice is if RDS offered this themselves (linked to your IAM roles). They offer this on RDS Serverless (https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/query-editor.html) and it may be a feature in other RDS versions. That allows fine control or even lazy control (IAM groups).
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 9 months ago.
Improve this question
I've created several RESTful microservices and dockerized them. Now I want to have a web-based UI for them and the ability to create users and grant permissions to them to use some of the APIs.
I know that I need some kind of API gateway. My first thought was that I always could do that bruteforce way: create some django app that would serve UI and proxy all request to APIs by hand, but this seems very dull. Maybe there are some alternatives? I've ready about Tyk, but can't find any information about the ability to add users and grant permissions to them.
I probably could create an application that would serve as API gateway and automate proxying of requests by writing some code that would model that. So for example I basically need a mapping between external urls to actual api urls and some authorization logic. Maybe there are already something like that?
I was looking for something similar, including support for rate limiting, UI console, etc. It boils down to a few freemium tools like:
apigee
mashape
apiary
3scale.net
and a few open source ones:
tyk
kong
ApiAxle
WSO2
API Umbrella
I've decided on tyk since it has a nice UI console and solid docs. All of them were mentioned on Quora, which is nice when you want to go shopping :)
If you like getting your hands dirty, you could quite easily implement your own simplified API Gateway. I believe this approach perfectly fits into microservice paradigm - implement simple service with limited functionality that does only one thing, but does it well.
I've written a tutorial on this subject (implementing simple API Gateway for Dockerized microservices with Node.js). My example is about 100 lines of JavaScript code, it uses node-docker-monitor to listen to Docker events and http-proxy to handle HTTP requests from the clients.
https://memz.co/api-gateway-microservices-docker-node-js/
or alternative solution with SkyDNS and Nginx
https://memz.co/reverse-proxy-nginx-docker-microservices/
Unless I'm mistaken (I'm a bit new to containers) - I think all of these solutions from Amazon AWS (with AWS API Gateway itself in front of any of them for public access) would also fall in the category of freemium solutions for running / orchestrating Docker Container(s) behind a public API Gateway:
AWS ECS - Elastic Container Service
AWS EKS - Elastic Kubernetes Service
AWS Fargate
Here are AWS blogs on using AWS API Gateway with them:
Using Amazon API Gateway with microservices deployed on Amazon ECS
API Gateway as an Ingress Controller for Amazon EKS
And depending on your requirements, either of these (also from Amazon AWS) should also meet requirements for users, accounts, permissions, authorization, etc:
AWS Identity Access Management
AWS Cognito
By the way, I don't think any intermediary management / compute / server should be necessary for architecting with any of those (although compute from AWS EC2 and/or AWS Lambda could still be inserted for any purpose), as they are all AWS services [that manage their functionality] unto themselves.
When working with .net Core webapi or other services in the .net world: Ocelot would be a choice.
https://learn.microsoft.com/en-us/dotnet/architecture/microservices/multi-container-microservice-net-applications/implement-api-gateways-with-ocelot
I've been coding on iOS for awhile. I'm getting started to use AWS as a cloud backend to store my user's data for my iOS app as well as a server that can handle real time event in multiplayer mode. I understand that there are many services like Amazon Cognito that allows user login via public providers, Dynamo DB that provides noSQL storage, EC2 that help me create server instances. However, I'm really confuse on how I can actually get started.
Here's some stuff that I really need help on:
I don't really want to use public providers like facebook for my users to sign in/sign up. I'd really prefer it if there could be relational database that is similar to how I cache data on local sqlite files using core data.
And I also need some help on getting my ID tokens for those services if I choose not to use cognito, or must I?
I would really appreciate it if you guys can give me some clues on how I can get started on these. Thank you so much! :D
Depending on how you expect your app to evolve there are several approaches you can take.
Option 1: Minimal scalability / low cost
All you really need is a single free EC2 micro instance. On the instance you can build a full LAMP stack in seconds. Once you have built it you can start writing an application to handle your app in your preferred language. Ruby is a la mode but any language will work. Your database will be stored on the actual instance. If you go with this route, one thing you should do is use an elastic IP address so that if your server ever goes down you can point the elastic IP to another instance. You should also periodically backup your server.
Option 2: Maximum scalability / variable cost
Use RDS to store your database. This will mean that any EC2 instance will be able to reach your data so you can have an unlimited number of servers. Then build an EC2 instance just like in option 1 but point your application to the RDS instance. Use a load balancer in front of your EC2 instance to scale up in response to changes in utilisation. And the elastic IP address should point to the load balancer.
Building a LAMP stack on EC2
Open the EC2 console
Select launch instance
Pick the Linux AMI offered by Amazon and a micro instance
Create a .pem key (keep is safe on your system or you will not be able to access your instance)
Select the default security group
Open your terminal window and type: ssh -i path_to_pem ec2-user#your_instance_public_address
In the EC2 instance type sudo yum update
Then type sudo yum install httpd24 php55 php55-mysqlnd mysql55
sudo apachectl start
You now can navigate to the public address of your instance in safari and assuming I did not forget any steps you should see the apache welcome page.