I am trying to write a Heroku auto scaling script. I want to base the auto scaling on the memory usage, dyno load and response time metrics that are available on the dashboard.
However I am not able to find an API endpoint in their documentation that will give me metrics data.
The closest thing that I was able to find is a log-runtime-metrics, a Heroku Labs module, that writes the CPU and memory usage in the syslog every 20 seconds.
I then need to fetch and parse the log in order to get some meaningful data. See this answer.
Is there an API for this? Or a better way of doing it?
Related
I'm using Locust to load test my web servers. I'm running Locust in distributed mode. The worker nodes are written in Java, and use the Locust/Java port using locust4j. The master node and the worker nodes are containerized, our orchestrator is Kubernetes. When I want to spin up more workers, I'm doing it from there.
The problem that I'm running into is that no matter how many users I add, or worker nodes I add, I can't seem to generate more than ~8000 RPM. This is confirmed by the Locust web frontend, as well as the metrics I'm collecting from my web server.
Does anyone have any ideas why this is happening?
I've attached an image of timings I've collected. The snapshots are from running the load test for 60 seconds, I'm timing it from a stopwatch.
The usual culprit in these kinds of situations is your servers can't handle more than that. In my experience, the behavior you'll see client side as the servers get overwhelmed is you'll start to see a slow but steady increase in response times. This is one big reason why Locust includes those in the metrics it shows you.
Based on what I'm seeing in your screenshots, this is most likely the case for you. You have some very low minimum times but your average, median, and 90%iles are a lot higher than your minimums; your maximums are very significantly higher than those. Without seeing your charts I can't know for sure but that's a big red flag.
For more things to look out for, check out this question in the FAQ (especially see the list of server stats to investigate):
https://github.com/locustio/locust/wiki/FAQ#increase-my-request-raterps
Are there any recommended Load Test tools / services that are able to cycle through AWS Application Load Balancer logs stored in S3 preferably utilising the time stamps to perform piano roll type functionality?
aws-log-replay seems to be something you're looking for, it can replay requests with defined concurrency.
With regards to more or less popular load testing tools I can only think of Apache JMeter with Access Log Sampler which support out of box access log files from Tomcat, Weblogic, Reisin and SunOne, however you can come up with your own implementation of Generator class or dynamically populate HTTP Request sampler fields using JSR223 PreProcessor like it's described in Stop Making Assumptions! Learn How to Replay Your Production Traffic With JMeter guide.
Actually I don't think you will be able to produce realistic load by replaying your access logs, it might work for something simple like static content, however if your application assumes authentication, sessions, complex workflows, etc. - I'm afraid your "replay" attempt will got stuck at login page.
So instead of trying to replay complex scenarios from the logs I would suggest sticking to the load testing tool of your choice and create it from scratch. Access logs can be used to identify workload distribution (like X % of users are normally doing this, Y % are doing that, etc.) and anticipated concurrency (like at X time we had Y online users).
I am working on a Rails app that pulls up to 100 Instagram posts at once with the media/search endpoint and displays them on a page. The AJAX call that loads the photos takes a very long time on localhost, but once deployed to Heroku, takes much less time (10s versus 1s). Can anyone explain why Heroku is faster? I might not need to worry as much about caching my results.
Thanks!!
One major reason will be Heroku's phsyical hosting location -- I believe Instagram hosts with Amazon's AWS service (this may have changed after the Facebook acquisition):
Here at Instagram, we run our infrastructure on Amazon Web Services,
running instances on their Elastic Compute Cloud (EC2)
Heroku basically hosts through Amazon's cloud too, meaning they are ostensibly running on the same network. This will obviously cut latency down to a minimum, as well as the fact that Heroku's services are optimized for efficiency -- high speed Internet etc
Cache
Your question is really "should I be creating a cache for Instagram data in my system?"
The answer is "yes" - it's my experience you should never rely on a third party entirely, as apart from obvious latency issues, you'll also have to contend with a multitude of other problems (API outages, client bandwidth etc)
I'd personally look at storing as much data as possible in my own system. This doesn't mean to keep all in your main DB - you could utilize a Redis instance to store the third-party data you need
So the context is that I have a client application that generates logs and I want to occasionally upload this data to a backend. The backend will function as an analytics server, storing, processing and displaying this data - so as you can imagine there will be some querying involved.
In terms of data collection peak load, I expect to have about 5k clients, each generating about 50 - 100 lines per day, and I'd like the solution I'm tackling to be able to process that kind of data. If you do the math, thats upwards of 1 million log lines / month.
In terms of data analytics load, it will be fairly low - I expect a couple of us (admins) to run queries to harvest some info once a week or so from all the logs.
My application is currently running RoR + Postgres, though I'm open to using a different dB if it maps better to my needs. Current contenders in my head are MongoDB & Cassandra, but I don't really want to leave Postgres if it can scale to get the job done.
I'd recommend a purpose built tool like logstash for this:
http://logstash.net/
Another alternative would be Apache Flume:
http://flume.apache.org/
For my experiences, you will need an search engine to do troubleshooting and analysis when you have a lot of logs, instead of using database. (Search engine will more faster than database.)
For now, I am using logstash+Elasticsearch+Kibana total solution to build up my Log system.
Logstash is a tool can parse the logs and make it more human
readable.
Elasticsearch is a search engine to do indexing and
searching on your logs.
Kibana is a webUI that you can use it
to communicate with your Elasticsearch.
This is an Kibana Demo website. You can visit it. http://demo.kibana.org/ .
It provides the search interface and analysis tools such as Pie chart, Table, etc.
In my project, My application generates over 1.5 million logs per day. This Log system can handle all this logs.
Enjoy it.
If you are looking for a database solution that will grow with requests, then I would recommend looking beyond Postgres.
Cassandra is really well-suited for time-series data, though key-value stores are not suited for ad-hoc analytics. One idea could be to store your logs in Cassandra, and then roll them up into a different system later.
For straightforward storing-and-displaying of data, take a look at Graphite, a realtime graphing project.
You can create your own custom graphs with Graphite, and save them as dashboards.
I am new to DynamoDB. I am very much confused about provisioned throughput. I am creating a iPhone game in which the users can chat within the game. I am having a Chat table. The Chat table contains GameID, UserID and Message. How do I find the size of the item to calculate throughput. The size of the item entirely depends on the Message right? How to calculate the size of an item?
Amazon tells that we can either modify the throughput by using UpdateTable API or by manually from the console. If I want to change it form code, how will I know that the provisioned throughput has been exceeded for a certain table? How to check that from code?
I am also confused about the CloudWatch. How to understand this?
Could anyone please help me? Please don't point me to the documentation.
Thanks.
I will do my best to help with the confusion.
DynamoDB is a key:value database
CloudWatch is Amazon's products monitoring tool
Provisioned throughput is roughly the number Items KB you plan to Read/Write per seconds
Whenever you exceed your provisioned throughput,
DynamoDB answers with ProvisionedThroughputExceededException
DynamoDB notifies CloudWatch
What Cloudwatch does is basically record and aggregates data-points. For most applications, it will only keep track of aggregated data over each consecutive 5min periods.
You can then access these data for "manual" monitoring or set up "alarms".
There was a really interesting question on SO a couple of weeks earlier on DynamoDB auto-scaling using alarms. You might be interested in reading it: http://docs.amazonwebservices.com/amazondynamodb/latest/developerguide/ErrorHandling.html
Knowing this, you can start building your application.
As for every DynamoDB services, one needs credentials to access it. Even though they can be restricted to a specific table or set of action, it is very dangerous to bundle them in an application. Would you give MySQL or MongoDB or credentials, even Read Only to any untrusted people ?
May I suggest you do build your application to rely on a server of your own ? This server being trusted and build by you, you could safely perform any authorization check there and grant it full access to your table.
I hope this helps. Feel free to ask for more precisions.