For this BlazeMeter (BM) load test the number of virtual users (VU) is not held for the configured and expected duration of 20 minutes. The defined concurrency/number VU's reaches its peak of 5000 VU at 09:14 but begins to drop threads around 09:20.
The combined test duration is correct of ramp-up (2 minutes) + hold-for (20 minutes) but BM only holds the concurrency at the expected level for about 6-8 minutes (not 20 minutes as configured).
Do you know why BM might drop the VU's earlier than requested? I have included the Taurus configuration below (scenario details like request URL's are redacted).
[taurus.yml]
---
aggregator: aggregator
execution:
- concurrency: 5000
executor: jmeter
hold-for: 20m
locations:
ap-northeast-1: 1
ap-south-1: 1
eu-central-1: 1
eu-west-1: 1
sa-east-1: 1
us-east-1: 1
ramp-up: 2m
scenario: myscenario
local-bzt-version: 1.16.3
modules:
cloud:
account-id: '[CUT]'
default-location: us-central1-a
detach: true
project-id: '[CUT]'
test: [CUT]
workspace-id: '[CUT]'
consolidator:
percentiles:
- 0.0
- 50.0
- 90.0
- 95.0
- 99.0
- 99.9
- 100.0
jmeter:
plugins:
- jpgc-casutg
- jpgc-dummy
- jpgc-ffw
- jpgc-fifo
- jpgc-functions
- jpgc-json
- jpgc-perfmon
- jpgc-prmctl
- jpgc-tst
properties:
jmeter.save.saveservice.autoflush: 'true'
jmeter.save.saveservice.connect_time: 'true'
jmeterengine.force.system.exit: 'true'
summariser.name: ''
reporting:
- module: final-stats
- module: console
- module: final-stats
scenarios:
myscenario:
[CUT (default-address, some headers, list of request URL's)]
schema: https://json.schemastore.org/taurus.json
services:
- local:
- metrics:
- cpu
- mem
- bytes-sent
- bytes-recv
- disk-read
- disk-write
- disk-space
- engine-loop
- conn-all
module: monitoring
settings:
default-executor: jmeter
env:
OBJC_DISABLE_INITIALIZE_FORK_SAFETY: 'YES'
TAURUS_ARTIFACTS_DIR: [CUT]
[bzt.log - Module config: jmeter]
{
'class': 'bzt.modules.jmeter.JMeterExecutor',
'protocol-handlers': {
'http': 'bzt.jmx.http.HTTPProtocolHandler',
'mqtt': 'bzt.jmx.mqtt.MQTTProtocolHandler'
},
'plugins': ['jpgc-casutg', 'jpgc-dummy', 'jpgc-ffw', 'jpgc-fifo', 'jpgc-functions', 'jpgc-json', 'jpgc-perfmon', 'jpgc-prmctl', 'jpgc-tst'],
'properties': {
'jmeter.save.saveservice.autoflush': 'true',
'jmeter.save.saveservice.connect_time': 'true',
'jmeterengine.force.system.exit': 'true',
'summariser.name': ''
}
We don't know because you have [CUT] few quite important parts of the configuration file.
I can think of the following possible reasons:
CSV Data Set Config with Stop thread on EOF set to True
Flow Control Action sampler with action to Stop the Current Thread
A JSR223 Test Element which conditionally stops the thread
So check your test plan and ensure that none of the above listed options is your case.
In addition you might want to check the Artifacts Directory and check jmeter.log, jmeter.err and jmeter.out files as well az bzt.log file.
And last but not the least as a BlazeMeter customer you can reach out to BlazeMeter Support, they are supposed to know the product much better than random people from the internet
Related
I am trying to design load tests using Artillery on a computation-heavy API which typically requires at least a few seconds to send a response.
Starting from examples found in the docs, I was able to run some tests such as this one:
config:
target: "https://example.com/api"
phases:
- duration: 60
arrivalRate: 1
name: Base case
I'd now like to send requests even slower (e.g. 1 every 5 seconds) but it seems that this cannot be done using the arrivalRate parameter. Is there any way to do it that the docs do not mention?
Thanks in advance !
How can I customize interval time of virtual user creation?
You can do that with arrivalCount which spreads the creation of
virtual users evenly over a period of time (whereas arrivalRate is
always per-second). E.g.:
config:
phases:
- duration: 60
arrivalCount: 12
I have a systemd service evil, with state described in init.sls. This service requires the presence of a file /etc/evil/wicked.txt. This file pulls in updates from a python function fetch_darkness that contacts a server, the response of which changes very infrequently.
evil:
pkg.installed:
- name: {{ package_name }}
- watch:
- file: /etc/evil/wicked.txt
- pkg: evil
/etc/evil/wicked.txt:
file.managed:
- contents: {{ salt.evil.fetch_darkness(grains['id'], pillar.node.hostname }} # this changes infrequently
- show_changes: false
- allow_empty: false
- user: evil
- group: evil
- mode: 600
- makedirs: true
- require:
- user: evil
- watch_in:
- service: evil
This creates a problem, because every highstate, it causes the service evil to be restarted, even if the actual contents of the file /etc/evil/wicked.txt haven't changed, which is 99% of the time.
So the next solution was creating a temporary file /etc/evil/wicked-temp.txt that wasn't watched by evil. /etc/evil/wicked.txt pulls its contents from wicked-temp.txt when that is changed, this way evil is only restarted when the file actually updates instead of every highstate:
evil:
pkg.installed:
- name: {{ package_name }}
- watch:
- file: /etc/evil/wicked.txt
- pkg: evil
/etc/evil/wicked-temp.txt:
file.managed:
- contents: {{ salt.evil.fetch_darkness(grains['id'], pillar.node.hostname }} # this changes infrequently
- show_changes: false
- allow_empty: false
- user: evil
- group: evil
- mode: 600
- makedirs: true
- require:
- user: evil
/etc/evil/wicked.txt:
file.managed:
- source: /etc/evil/wicked-temp.txt
- show_changes: false
- allow_empty: false
- user: evil
- group: evil
- mode: 600
- makedirs: true
- onchanges:
- file: /etc/evil/wicked-temp.txt
- require:
- user: evil
- watch_in:
- service: evil
"rm /etc/evil/wicked-temp.txt":
cmd.run:
- onfail:
- file: /etc/evil/wicked.txt
However, now the issue is, there are several servers that each get highstated, and there's many times that fetch_darkness isn't able to reach the main server, causing the highstate to fail as wicked.txt/wicked-temp.txt can't get populated.
This isn't great, because >99% of the time, the contents of the server response doesn't change. If wicked.txt doesn't exist, then sure, the highstate should fail. But if it does, then I'd like to keep using whatever already exists on the file wicked.txt and not have a failed highstate.
Is there a way to solve this problem:
Service must not restart unless file contents change
File must pull updates
Highstate must not fail if file already exists, but is unable to pull updates
EDIT: I should've mentioned that the file's contents do change every time. The file is a randomly generated key that is fetched from the central server, with a certain label attached, so something like 1-xxxxxxx, or 2-yyyyyy. However, 1-xxxxx is identical to 1-zzzzzz, even if the actual contents differ, which is why evil should not restart when 1-xxxxx changes to 1-zzzzz, but only when it changes to 2-yyyyyyy.
you already have a custom module that does the fetch, why not create a custom state module that handles the rest? this way you can determine if the file actually changed, handle it when it does and correct for errors when the fetch doesn't work.
in the first example however it shouldn't do anything that triggers that the file changed unless something with the file actually is changing. you might want to double check what is changing in the file.
I am using Sidekiq in my Rails application to process background jobs.
sidekiq.yml
:verbose: false
:pidfile: ./tmp/pids/sidekiq.pid
:logfile: ./log/sidekiq.log
:concurrency: 5
:queues:
- ["queue1", 1]
- ["queue2", 1]
- ["queue3", 1]
- ["queue4", 1]
- ["queue5", 1]
- critical
- default
- low
max_retries: 1
I want to run one only task per queue at a time.
For example: I am starting same worker 3 times in queue1, I want to process the worker1 and then worker2 and then worker3.
I have added "1" with queue name (e.g ["queue1", 1]) in sidekiq.yml. I thought this configuration will run only one worker in queue1. But that is not happening.
How to achieve the above configuration in sidekiq?
First a slight correction in the terminologies. "Workers" are constantly running threads that consume "jobs" from "queues". Hence you don't process worker1, instead worker1 will process jobs from queue1.
Now coming to the solution:-
It seems that your use case is that you don't want specific type of job (one that uploads to s3 in your case) to be executed on more than one worker at a given instant.
To achieve that, you will need a locking mechanism at job level. Sidekiq doesn't provide locking on its own, hence you need to use external gems for that.
I have used redis based 'sidekiq-lock' to achieve something similar in my projects.
Gem Link
Usage is as follows:-
class Worker
include Sidekiq::Worker
include Sidekiq::Lock::Worker
# static lock that expires after one second
sidekiq_options lock: { timeout: 1000, name: 'unique-lock-name' }
def perform
# your code to upload to s3
end
end
This will ensure that your code is running only one instance of given job on one worker at a time.
Sidekiq explicitly does not allow what you want to do.
https://github.com/mperham/sidekiq/wiki/Best-Practices#3-embrace-concurrency
Just use the built in ActiveJob because it sounds like you want FIFO queue. Sidekiq is for idempotent concurrent backgrounding
I think it that you want a feature
Add it to your worker class
class YourWorker
include Sidekiq::Worker
sidekiq_options queue: :queue1
But there are a caveat which worker only could queueing by queue1.
queues setting controls how often queue will be checked for jobs to control it's priority related to others, not limiting parallel jobs to that queue.
Also you can spin up multiple workers with limited number of threads (that's what sidekiq documentation suggests for this case), but this may not be practical for many queues with low load, when most of the workers will idle.
To limit particular queue parallel jobs within one worker - you can use gem sidekiq-limit_fetch and settings like:
:limits:
queue1: 1
queue2: 1
You can use sidekiq-limit_fetch to use advanced options for sidekiq. To limit concurrency on individual queue, you can do the following.
Step 1. Add gem 'sidekiq-limit_fetch' in your Gemfile
Step 2. Add limits for each queue in your sidekiq.yml
:limits:
queue1: 1
queue2: 1
queue3: 1
queue4: 1
Set your Sidekiq concurrency to 1. This way you'll have only 1 thread working in your process which would give you the desired output: only 1 job is done at a time.
Just change your configuration file:
:verbose: false
:pidfile: ./tmp/pids/sidekiq.pid
:logfile: ./log/sidekiq.log
:concurrency: 1 # Change Here
:queues:
- ["queue1", 1]
- ["queue2", 1]
- ["queue3", 1]
- ["queue4", 1]
- ["queue5", 1]
- critical
- default
- low
max_retries: 1
I have a bunch of micro-services hosted on AWS. I am using StatsD, Graphite and Grafana to monitor them. Now I want to expand it to monitor the queues (SQS) through which these micro-services are talking to each other. How can I leverage Graphite/ Grafana to do this? Or a better approach if there aint any support/ plugin for the same. Thanks :)
PS : If it's gotta be Zipkin, please tell me they can co-exist or is there a catch to using multiple tracers.
Alright, so I'm going to answer this based on what you said here:
Or a better approach if there aint any support/ plugin for the same.
The way that I do it us through Prometheus, in combination with cloudwatch_exporter, and alertmanager.
The configuration for cloudwatch_exporter to monitor SQS is going to be something like (this is only two metrics, you'll need to add more based on what you're looking to monitor):
tasks:
- name: ec2_cloudwatch
default_region: us-west-2
metrics:
- aws_namespace: "AWS/SQS"
aws_dimensions: [QueueName]
aws_metric_name: NumberOfMessagesReceived
aws_statistics: [Sum]
range_seconds: 600
- aws_namespace: "AWS/SQS"
aws_dimensions: [QueueName]
aws_metric_name: ApproximateNumberOfMessagesDelayed
aws_statistics: [Sum]
You'll then need to configure prometheus to scrape the cloudwatch_exporter endpoint at an interval, for ex what I do:
- job_name: 'somename'
scrape_timeout: 60s
dns_sd_configs:
- names:
- "some-endpoint"
metrics_path: /scrape
params:
task: [ec2_cloudwatch]
region: [us-east-1]
relabel_configs:
- source_labels: [__param_task]
target_label: task
- source_labels: [__param_region]
target_label: region
You would then configure alertmanager to alert based on those scraped metrics; I do not alert on those metrics so I cannot give you an example. But, to give you an idea how of this architecture, a diagram is below:
If you need to use something like statsd you can use statsd_exporter. And, just in-case you were wondering, yes Grafana supports prometheus.
With making reverse proxy on Docker and Traefik, I want to dispatch several paths on the same host into two different backend servers like these,
1. traefik.test/ -> app1/
2. traefik.test/post/blabla -> app1/post/blabla
3. traefik.test/user/blabla -> app2/user/blabla
If the rules are only #2 and #3, I could do like this in docker-compose.yml
app1:
image: akky/app1
labels:
- "traefik.backend=app1"
- "traefik.frontend.rule=Host:traefik.test;PathPrefix:/post,/comment"
app2:
image: akky/app2
labels:
- "traefik.backend=app2"
- "traefik.frontend.rule=Host:traefik.test;PathPrefix:/user,/group"
However, adding the root '/' into the first PathPrefix seems to cloak /user on app2. The following does not work, and everything goes to app1 backend.
- "traefik.frontend.rule=Host:traefik.test;PathPrefix:/,/post,/group"
The rules "Host:" and "PathPrefix" seems working as 'AND', but I wanted to use 'OR' ( exact /, OR starting with /post ). I searched and came to know that multiple rules can be directed since version 1.3.0, according to pull request #1257 by making multiple lines with adding service names.
By knowing that, what I did is like this,
app1:
image: akky/app1
labels:
- "traefik.app1_subfolder.backend=app1"
- "traefik.app1_subfolder.frontend.rule=Host:traefik.test;PathPrefix:/post,/group"
- "traefik.app1_rootfolder.backend=app1"
- "traefik.app1_rootfolder.frontend.rule=Host:traefik.test;Path:/"
app2:
image: akky/app2
labels:
- "traefik.backend=app2"
- "traefik.frontend.rule=Host:traefik.test;PathPrefix:/user"
Now it works as required, the root access is dispatched to app1/ .
My question is, is this the proper way? It does not look like so for me, as this root and subfolder dispatch should be a typical use case.
You might consider adding priority labels so the app2 rules take precedence over app1 rules. Then you should be able to simplify the app1 config.
app1:
image: akky/app1
labels:
- "traefik.backend=app1"
- "traefik.frontend.priority=10"
- "traefik.frontend.rule=Host:traefik.test;PathPrefix:/,/post,/group"
app2:
image: akky/app2
labels:
- "traefik.backend=app2"
- "traefik.frontend.priority=50"
- "traefik.frontend.rule=Host:traefik.test;PathPrefix:/user"
Update: I had the priorities in the wrong order. Larger priority values take precedence over smaller priority values. According to the docs, it's based on (priority + rule length), and the larger value wins.