I'm naively passing along some variable test metadata to some py_test targets to inject that metadata into some test result artifacts that later get uploaded to the cloud. I'm doing so using either the --test_env or --test_arg values at the bazel test invocation.
Would this variable data negatively affect the way test results are cached such that running the same test back to back would effectively disturb the bazel cache?
Command Line Inputs
Command line inputs can indeed disturb cache hits. Consider the following set of executions
BUILD file
py_test(
name = "test_inputs",
srcs = ["test_inputs.py"],
deps = [
":conftest",
"#pytest",
],
)
py_library(
name = "conftest",
srcs = ["conftest.py"],
deps = [
"#pytest",
],
)
Test module
import sys
import pytest
def test_pass():
assert True
def test_arg_in(request):
assert request.config.getoption("--metadata")
if __name__ == "__main__":
args = sys.argv[1:]
ret_code = pytest.main([__file__, "--log-level=ERROR"] + args)
sys.exit(ret_code)
First execution
$ bazel test //bazel_check:test_inputs --test_arg --metadata=abc
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
INFO: 2 processes: 1 internal (50.00%), 1 local (50.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.40s
INFO: Critical path 0.57s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 0.72s (preparation 0.12s, execution 0.60s)
INFO: Build completed successfully, 2 total actions
//bazel_check:test_inputs PASSED in 0.4s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions
Second execution: same argument value, cache hit!
$ bazel test //bazel_check:test_inputs --test_arg --metadata=abc
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
INFO: 1 process: 1 internal (100.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.00s
INFO: Critical path 0.47s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 0.61s (preparation 0.12s, execution 0.49s)
INFO: Build completed successfully, 1 total action
//bazel_check:test_inputs (cached) PASSED in 0.4s
Executed 0 out of 1 test: 1 test passes.
INFO: Build completed successfully, 1 total action
Third execution: new argument value, no cache hit
$ bazel test //bazel_check:test_inputs --test_arg --metadata=kk
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 93 targets configured).
INFO: Found 1 test target...
INFO: 2 processes: 1 internal (50.00%), 1 local (50.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.30s
INFO: Critical path 0.54s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 0.71s (preparation 0.14s, execution 0.57s)
INFO: Build completed successfully, 2 total actions
//bazel_check:test_inputs PASSED in 0.3s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions
Fourth execution: reused same argument as first two runs
Interestingly enough there is no cache hit despite the result being cached earlier. Somehow it did not persist.
$ bazel test //bazel_check:test_inputs --test_arg --metadata=abc
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
INFO: 2 processes: 1 internal (50.00%), 1 local (50.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.34s
INFO: Critical path 0.50s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 0.71s (preparation 0.17s, execution 0.55s)
INFO: Build completed successfully, 2 total actions
//bazel_check:test_inputs PASSED in 0.3s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions
Environment Inputs
The same exact behavior applies for --test_env inputs
import os
import sys
import pytest
def test_pass():
assert True
def test_env_in():
assert os.environ.get("META_ENV")
if __name__ == "__main__":
args = sys.argv[1:]
ret_code = pytest.main([__file__, "--log-level=ERROR"] + args)
sys.exit(ret_code)
First execution
$ bazel test //bazel_check:test_inputs --test_env META_ENV=33
INFO: Build option --test_env has changed, discarding analysis cache.
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 7285 targets configured).
INFO: Found 1 test target...
INFO: 2 processes: 1 internal (50.00%), 1 local (50.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.29s
INFO: Critical path 0.66s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 1.26s (preparation 0.42s, execution 0.84s)
INFO: Build completed successfully, 2 total actions
//bazel_check:test_inputs PASSED in 0.3s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions
Second execution: same env value, cache hit!
$ bazel test //bazel_check:test_inputs --test_env META_ENV=33
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
INFO: 1 process: 1 internal (100.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.00s
INFO: Critical path 0.49s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 0.67s (preparation 0.15s, execution 0.52s)
INFO: Build completed successfully, 1 total action
//bazel_check:test_inputs (cached) PASSED in 0.3s
Executed 0 out of 1 test: 1 test passes.
INFO: Build completed successfully, 1 total action
Third execution: new env value, no cache hit
$ bazel test //bazel_check:test_inputs --test_env META_ENV=44
INFO: Build option --test_env has changed, discarding analysis cache.
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 7285 targets configured).
INFO: Found 1 test target...
INFO: 2 processes: 1 internal (50.00%), 1 local (50.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.29s
INFO: Critical path 0.62s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 1.22s (preparation 0.39s, execution 0.83s)
INFO: Build completed successfully, 2 total actions
//bazel_check:test_inputs PASSED in 0.3s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions
Fourth execution: reused same env value as first two runs
$ bazel test //bazel_check:test_inputs --test_env META_ENV=33
INFO: Build option --test_env has changed, discarding analysis cache.
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 7285 targets configured).
INFO: Found 1 test target...
INFO: 2 processes: 1 internal (50.00%), 1 local (50.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.28s
INFO: Critical path 0.66s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 1.25s (preparation 0.40s, execution 0.85s)
INFO: Build completed successfully, 2 total actions
//bazel_check:test_inputs PASSED in 0.3s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions
I'm trying to do performance test using the https://github.com/thingsboard/gatling-mqtt project.
Although current versions of the tools are a bit different, I managed to have it running.
I've tested a couple of basic scripts with Gatling using HTTP and Gatling worked as expected.
But when using the scripts provided with the tool, that use the plugin for MQTT it does not work.
In fact it runs but it doesn't do anything. No connections, no logs, no reports, no errors.
On the other tests it increments the global OK=0 as the test progresses, and generates logs and reports.
But as you can see below, when running with the MQTT plugin it doesn't increment the count
.
Simulation MqttSimulation_localhost started...
================================================================================
2021-10-14 17:55:02 5s elapsed
---- Requests ------------------------------------------------------------------
> Global (OK=0 KO=0 )
---- MQTT Test -----------------------------------------------------------------
[--------------------------------------------------------------------------] 0%
waiting: 0 / active: 10 / done:0
================================================================================
================================================================================
2021-10-14 17:55:07 10s elapsed
---- Requests ------------------------------------------------------------------
> Global (OK=0 KO=0 )
---- MQTT Test -----------------------------------------------------------------
[--------------------------------------------------------------------------] 0%
waiting: 0 / active: 10 / done:0
================================================================================
================================================================================
2021-10-14 17:55:12 15s elapsed
---- Requests ------------------------------------------------------------------
> Global (OK=0 KO=0 )
---- MQTT Test -----------------------------------------------------------------
[--------------------------------------------------------------------------] 0%
waiting: 0 / active: 10 / done:0
================================================================================
================================================================================
2021-10-14 17:55:17 20s elapsed
---- Requests ------------------------------------------------------------------
> Global (OK=0 KO=0 )
... and it goes on forever
Any ideas on what could be happening?
I would certainly appreciate any help on this!
Thank you!
After SonarQube Scanner was executed by a Jenkins job, the analysis result is not being uploaded to the SonarQube server:
00:15:23.616 INFO: ANALYSIS SUCCESSFUL
00:15:23.621 DEBUG: Post-jobs : GitHub Pull Request Issue Publisher (wrapped)
00:15:23.621 INFO: Executing post-job GitHub Pull Request Issue Publisher (wrapped)
In one of my other Jenkins jobs, the result is actually uploaded to SonarQube with this log which is not found for above case:
INFO: Sensor CPD Block Indexer (done) | time=0ms
INFO: 20 files had no CPD blocks
INFO: Calculating CPD for 46 files
INFO: CPD calculation finished
INFO: Analysis report generated in 312ms, dir size=1 MB
INFO: Analysis reports compressed in 227ms, zip size=561 KB
INFO: Analysis report uploaded in 256ms
Is there anything I can do to fix this?
All,
I'm learning to do benchmark testing using the Siege tool against our Rails app. I'm running Siege on my OS X box against a website hosted on another server. When I run it I always get zero hits and zero failed transactions no matter which site I run it against.
Because of the limitations of OS X, I've configured more ports with sudo sysctl -w net.inet.tcp.msl=1000 and more open files with launchctl limit maxfiles 10000 10000. I've also configured proxy variables in .siegerc since I'm running behind a proxy. Sample command line is:
siege -c1 -b -t10S 'www.google.com'.
But no matter which website I hit, I always get zero hits, zero failures and zero successful transactions. The Siege log file does not show any errors. What am I doing wrong?
siege -c1 -b -t10S 'www.google.com'
** SIEGE 3.0.5
** Preparing 1 concurrent users for battle.
The server is now under siege...
Lifting the server siege... done.
Transactions: 0 hits
Availability: 0.00 %
Elapsed time: 9.84 secs
Data transferred: 0.00 MB
Response time: 0.00 secs
Transaction rate: 0.00 trans/sec
Throughput: 0.00 MB/sec
Concurrency: 0.00
Successful transactions: 0
Failed transactions: 0
Longest transaction: 0.00
Shortest transaction: 0.00
I am using a rails performance test run as rake test:benchmark. The result gives me the warmup time.
I can't find the meaning of the 211 ms warm up time. Some of the test take longer warmup
time. I know what wall_time, user_time and etc.
.ApiTest#test_license_pool (211 ms warmup)
wall_time: 167 ms
user_time: 47 ms
memory: 6.2 MB
gc_runs: 0
gc_time: 0 ms