I'm naively passing along some variable test metadata to some py_test targets to inject that metadata into some test result artifacts that later get uploaded to the cloud. I'm doing so using either the --test_env or --test_arg values at the bazel test invocation.
Would this variable data negatively affect the way test results are cached such that running the same test back to back would effectively disturb the bazel cache?
Command Line Inputs
Command line inputs can indeed disturb cache hits. Consider the following set of executions
BUILD file
py_test(
name = "test_inputs",
srcs = ["test_inputs.py"],
deps = [
":conftest",
"#pytest",
],
)
py_library(
name = "conftest",
srcs = ["conftest.py"],
deps = [
"#pytest",
],
)
Test module
import sys
import pytest
def test_pass():
assert True
def test_arg_in(request):
assert request.config.getoption("--metadata")
if __name__ == "__main__":
args = sys.argv[1:]
ret_code = pytest.main([__file__, "--log-level=ERROR"] + args)
sys.exit(ret_code)
First execution
$ bazel test //bazel_check:test_inputs --test_arg --metadata=abc
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
INFO: 2 processes: 1 internal (50.00%), 1 local (50.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.40s
INFO: Critical path 0.57s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 0.72s (preparation 0.12s, execution 0.60s)
INFO: Build completed successfully, 2 total actions
//bazel_check:test_inputs PASSED in 0.4s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions
Second execution: same argument value, cache hit!
$ bazel test //bazel_check:test_inputs --test_arg --metadata=abc
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
INFO: 1 process: 1 internal (100.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.00s
INFO: Critical path 0.47s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 0.61s (preparation 0.12s, execution 0.49s)
INFO: Build completed successfully, 1 total action
//bazel_check:test_inputs (cached) PASSED in 0.4s
Executed 0 out of 1 test: 1 test passes.
INFO: Build completed successfully, 1 total action
Third execution: new argument value, no cache hit
$ bazel test //bazel_check:test_inputs --test_arg --metadata=kk
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 93 targets configured).
INFO: Found 1 test target...
INFO: 2 processes: 1 internal (50.00%), 1 local (50.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.30s
INFO: Critical path 0.54s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 0.71s (preparation 0.14s, execution 0.57s)
INFO: Build completed successfully, 2 total actions
//bazel_check:test_inputs PASSED in 0.3s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions
Fourth execution: reused same argument as first two runs
Interestingly enough there is no cache hit despite the result being cached earlier. Somehow it did not persist.
$ bazel test //bazel_check:test_inputs --test_arg --metadata=abc
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
INFO: 2 processes: 1 internal (50.00%), 1 local (50.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.34s
INFO: Critical path 0.50s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 0.71s (preparation 0.17s, execution 0.55s)
INFO: Build completed successfully, 2 total actions
//bazel_check:test_inputs PASSED in 0.3s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions
Environment Inputs
The same exact behavior applies for --test_env inputs
import os
import sys
import pytest
def test_pass():
assert True
def test_env_in():
assert os.environ.get("META_ENV")
if __name__ == "__main__":
args = sys.argv[1:]
ret_code = pytest.main([__file__, "--log-level=ERROR"] + args)
sys.exit(ret_code)
First execution
$ bazel test //bazel_check:test_inputs --test_env META_ENV=33
INFO: Build option --test_env has changed, discarding analysis cache.
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 7285 targets configured).
INFO: Found 1 test target...
INFO: 2 processes: 1 internal (50.00%), 1 local (50.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.29s
INFO: Critical path 0.66s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 1.26s (preparation 0.42s, execution 0.84s)
INFO: Build completed successfully, 2 total actions
//bazel_check:test_inputs PASSED in 0.3s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions
Second execution: same env value, cache hit!
$ bazel test //bazel_check:test_inputs --test_env META_ENV=33
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
INFO: 1 process: 1 internal (100.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.00s
INFO: Critical path 0.49s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 0.67s (preparation 0.15s, execution 0.52s)
INFO: Build completed successfully, 1 total action
//bazel_check:test_inputs (cached) PASSED in 0.3s
Executed 0 out of 1 test: 1 test passes.
INFO: Build completed successfully, 1 total action
Third execution: new env value, no cache hit
$ bazel test //bazel_check:test_inputs --test_env META_ENV=44
INFO: Build option --test_env has changed, discarding analysis cache.
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 7285 targets configured).
INFO: Found 1 test target...
INFO: 2 processes: 1 internal (50.00%), 1 local (50.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.29s
INFO: Critical path 0.62s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 1.22s (preparation 0.39s, execution 0.83s)
INFO: Build completed successfully, 2 total actions
//bazel_check:test_inputs PASSED in 0.3s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions
Fourth execution: reused same env value as first two runs
$ bazel test //bazel_check:test_inputs --test_env META_ENV=33
INFO: Build option --test_env has changed, discarding analysis cache.
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 7285 targets configured).
INFO: Found 1 test target...
INFO: 2 processes: 1 internal (50.00%), 1 local (50.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.28s
INFO: Critical path 0.66s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 1.25s (preparation 0.40s, execution 0.85s)
INFO: Build completed successfully, 2 total actions
//bazel_check:test_inputs PASSED in 0.3s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions
I am trying to learn the basics of sonarqube, I have set it up on my jenkins pipeline (which uses MSBuild to compile the project) and the set a step to execute SonarQube but after reading and watching a few tutorials and the documentation itself I think I got a bit lost as when my pipeline finishes executing I go to my SonarQube server and I my project says The main branch has no lines of code.. I get the following log when running the sonarqube step on my jenkins:
00:10:43.369 Injecting SonarQube environment variables using the configuration: SonarQube
[Pipeline] {
[Pipeline] bat
00:10:43.653
00:10:43.653 C:\Users\anton\.jenkins\workspace\Boston>C:\Users\anton\.jenkins\tools\hudson.plugins.sonar.MsBuildSQRunnerInstallation\Sonnar_scanner\sonar-scanner-4.4.0.2170\bin\sonar-scanner.bat
00:10:43.653 INFO: Scanner configuration file: C:\Users\anton\.jenkins\tools\hudson.plugins.sonar.MsBuildSQRunnerInstallation\Sonnar_scanner\sonar-scanner-4.4.0.2170\bin\..\conf\sonar-scanner.properties
00:10:43.653 INFO: Project root configuration file: NONE
00:10:43.653 INFO: SonarScanner 4.4.0.2170
00:10:43.653 INFO: Java 1.8.0_261 Oracle Corporation (64-bit)
00:10:43.653 INFO: Windows 10 10.0 amd64
00:10:44.230 INFO: User cache: C:\WINDOWS\system32\config\systemprofile\.sonar\cache
00:10:44.230 INFO: Scanner configuration file: C:\Users\anton\.jenkins\tools\hudson.plugins.sonar.MsBuildSQRunnerInstallation\Sonnar_scanner\sonar-scanner-4.4.0.2170\bin\..\conf\sonar-scanner.properties
00:10:44.230 INFO: Project root configuration file: NONE
00:10:44.230 INFO: Analyzing on SonarQube server 8.5.1
00:10:44.230 INFO: Default locale: "es_ES", source code encoding: "UTF-8"
00:10:44.783 WARN: SonarScanner will require Java 11 to run starting in SonarQube 8.x
00:10:44.784 INFO: Load global settings
00:10:44.784 INFO: Load global settings (done) | time=40ms
00:10:44.784 INFO: Server id: BF41A1F2-AXWnoupqHjnRwWgEnmvK
00:10:44.784 INFO: User cache: C:\WINDOWS\system32\config\systemprofile\.sonar\cache
00:10:44.784 INFO: Load/download plugins
00:10:44.784 INFO: Load plugins index
00:10:44.784 INFO: Load plugins index (done) | time=26ms
00:10:44.784 INFO: Load/download plugins (done) | time=48ms
00:10:45.037 INFO: Process project properties
00:10:45.037 INFO: Process project properties (done) | time=6ms
00:10:45.037 INFO: Execute project builders
00:10:45.037 INFO: Execute project builders (done) | time=1ms
00:10:45.037 INFO: Project key: Boston
00:10:45.037 INFO: Base dir: C:\Users\anton\.jenkins\workspace\Boston
00:10:45.037 INFO: Working dir: C:\Users\anton\.jenkins\workspace\Boston\.scannerwork
00:10:45.301 INFO: Load project settings for component key: 'Boston'
00:10:45.301 INFO: Load project settings for component key: 'Boston' (done) | time=10ms
00:10:45.301 INFO: Load quality profiles
00:10:45.301 INFO: Load quality profiles (done) | time=27ms
00:10:45.301 INFO: Auto-configuring with CI 'Jenkins'
00:10:45.301 INFO: Load active rules
00:10:46.241 INFO: Load active rules (done) | time=787ms
00:10:46.241 INFO: Indexing files...
00:10:46.241 INFO: Project configuration:
00:10:46.241 INFO: 0 files indexed
00:10:46.241 INFO: ------------- Run sensors on module Boston
00:10:46.241 INFO: Load metrics repository
00:10:46.241 INFO: Load metrics repository (done) | time=12ms
00:10:46.807 INFO: Sensor CSS Rules [cssfamily]
00:10:46.807 INFO: No CSS, PHP, HTML or VueJS files are found in the project. CSS analysis is skipped.
00:10:46.807 INFO: Sensor CSS Rules [cssfamily] (done) | time=1ms
00:10:46.807 INFO: Sensor JaCoCo XML Report Importer [jacoco]
00:10:46.807 INFO: 'sonar.coverage.jacoco.xmlReportPaths' is not defined. Using default locations: target/site/jacoco/jacoco.xml,target/site/jacoco-it/jacoco.xml,build/reports/jacoco/test/jacocoTestReport.xml
00:10:46.807 INFO: No report imported, no coverage information will be imported by JaCoCo XML Report Importer
00:10:46.807 INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=1ms
00:10:46.807 INFO: Sensor C# Properties [csharp]
00:10:46.807 INFO: Sensor C# Properties [csharp] (done) | time=1ms
00:10:46.807 INFO: Sensor JavaXmlSensor [java]
00:10:46.807 INFO: Sensor JavaXmlSensor [java] (done) | time=0ms
00:10:46.807 INFO: Sensor VB.NET Properties [vbnet]
00:10:46.807 INFO: Sensor VB.NET Properties [vbnet] (done) | time=0ms
00:10:46.807 INFO: ------------- Run sensors on project
00:10:46.807 INFO: Sensor Zero Coverage Sensor
00:10:46.807 INFO: Sensor Zero Coverage Sensor (done) | time=0ms
00:10:46.807 INFO: SCM Publisher is disabled
00:10:46.807 INFO: CPD Executor Calculating CPD for 0 files
00:10:46.807 INFO: CPD Executor CPD calculation finished (done) | time=0ms
00:10:46.807 INFO: Analysis report generated in 48ms, dir size=82 KB
00:10:46.807 INFO: Analysis report compressed in 10ms, zip size=10 KB
00:10:47.060 INFO: Analysis report uploaded in 26ms
00:10:47.060 INFO: ANALYSIS SUCCESSFUL, you can browse http://localhost:9000/dashboard?id=Boston
00:10:47.060 INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
00:10:47.060 INFO: More about the report processing at http://localhost:9000/api/ce/task?id=AXWt8eSC9o0F9NdVJGJP
00:10:47.060 INFO: Executing post-job 'Final report'
00:10:47.060 INFO: Turn debug info on to get more details (sonar-scanner -X -Dsonar.verbose=true ...).
00:10:47.060 INFO: Analysis total time: 2.118 s
00:10:47.060 INFO: ------------------------------------------------------------------------
00:10:47.060 INFO: EXECUTION SUCCESS
00:10:47.060 INFO: ------------------------------------------------------------------------
00:10:47.060 INFO: Total time: 3.274s
00:10:47.060 INFO: Final Memory: 21M/549M
00:10:47.060 INFO: ------------------------------------------------------------------------
[Pipeline] }
[Pipeline] // withSonarQubeEnv
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
and I think the main issue is the line where it says Project root configuration file: NONE I have created a sonar-project.properties file in the root folder of my project and it has the following configuration:
sonar.projectKey=Boston
sonar.projectName=Boston
sonar.dotnet.visualStudio.solution=Age3DE.sln
sonar.scm.disabled=true
sonar.sources=./Source
sonar.language=c++
sonar.sourceEncoding=UTF-8
But yet I cannot make it work. I do not know what I am doing wrong, should the sonar-project.properties be set anywhere else?
If the source files are present in Sources directory, then your sonar.sources should be like this
sonar.sources=**/Sources/*
Or
sonar.sources=Sources/*
sonar.exclusions parameter is not working for Sonarqube, the sonar.exclusions parameter defined in Jenkins or project setting in SonarQube Server it is not taking effect and scanning all files.
I want to exclude CSS,JS and HTML file scanning from the project for that I defined
sonar.exclusions=**/*.css, **/*.js ,**/*.html
also defined the path with root path with Jenkins workspace like
sonar.exclusions=$WORKSPACE/src/**/*.css, $WORKSPACE/src/**/*.js
still no positive result .
Also defined exclusions in sonarQube sever also
Software Stack
SonarQube:7.9.4-community
Jenkins: 2.235.1
Jenkins-plugin: SonarQubeScanner for Jenkins 2.11
SonarQube Scanner:- 3.0.0.702
The Jenkins console output is the following . what mistake I am making here can someone guide me?
Dsonar.coverage.jacoco.xmlReportPaths=/opt/cloudhost/apps/.jenkins/workspace/TEMP_SONARQUBE_CI_CD_LEMS_BUILD_EXTRA/build/reports/jacoco/test/jacocoTestReport.xml
Dsonar.sources=/opt/cloudhost/apps/.jenkins/workspace/TEMP_SONARQUBE_CI_CD_LEMS_BUILD_EXTRA/src
Dsonar.surefire.reportsPath=/opt/cloudhost/apps/.jenkins/workspace/TEMP_SONARQUBE_CI_CD_LEMS_BUILD_EXTRA/
build/test-results/test "-Dsonar.exclusions=**/*.css, **/*.js , **/*.html" -Dsonar.projectVersion=Octane-137020-SonarQube-Integrations -
Dsonar.java.binaries=/opt/cloudhost/apps/.jenkins/workspace/TEMP_SONARQUBE_CI_CD_LEMS_BUILD_EXTRA/build/c
lasses -Dsonar.projectKey=LEMS_X_Exclude -
Dsonar.junit.reportPaths=/opt/cloudhost/apps/.jenkins/workspace/TEMP_SONARQUBE_CI_CD_LEMS_BUILD_EXTRA/bui
ld/test-results/test
Dsonar.projectBaseDir=/opt/cloudhost/apps/.jenkins/workspace/TEMP_SONARQUBE_CI_CD_LEMS_BUILD_EXTRA
INFO: Scanner configuration file: /opt/cloudhost/apps/.jenkins/tools/hudson.plugins.sonar.SonarRunnerInstallation/SonarQubeScanner/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarQube Scanner 3.0.0.702
INFO: Java 1.8.0_252 Oracle Corporation (64-bit)
INFO: Linux 3.10.0-1127.10.1.el7.x86_64 amd64
INFO: User cache: /home/stlpdapi/.sonar/cache
INFO: SonarQube server 7.9.4
INFO: Default locale: "en_US", source code encoding: "UTF-8"
WARN: SonarScanner will require Java 11 to run starting in SonarQube 8.x
INFO: Load global settings
INFO: Load global settings (done) | time=119ms
INFO: Server id: E2281BAA-AXPWxgc9dGdpV1spT8iU
INFO: User cache: /home/stlpdapi/.sonar/cache
INFO: Load/download plugins
INFO: Load plugins index
INFO: Load plugins index (done) | time=79ms
INFO: Load/download plugins (done) | time=134ms
INFO: Process project properties
INFO: Execute project builders
INFO: Execute project builders (done) | time=3ms
INFO: Project key: LEMS_X_Exclude
INFO: Base dir: /opt/cloudhost/apps/.jenkins/workspace/TEMP_SONARQUBE_CI_CD_LEMS_BUILD_EXTRA
INFO: Working dir: /opt/cloudhost/apps/.jenkins/workspace/TEMP_SONARQUBE_CI_CD_LEMS_BUILD_EXTRA/.scannerwork
INFO: Load project settings for component key: 'LEMS_X_Exclude'
INFO: Load project settings for component key: 'LEMS_X_Exclude' (done) | time=22ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=75ms
INFO: Detected Jenkins
INFO: Load active rules
INFO: Load active rules (done) | time=747ms
INFO: Indexing files...
INFO: Project configuration:
INFO: Excluded sources: **/*.css, **/*.js, **/*.html
INFO: Load project repositories
INFO: Load project repositories (done) | time=44ms
INFO: 1221 files indexed
INFO: 0 files ignored because of inclusion/exclusion patterns
INFO: 0 files ignored because of scm ignore settings
INFO: Quality profile for css: Sonar way
INFO: Quality profile for java: Sonar way
INFO: Quality profile for jsp: Sonar way
INFO: Quality profile for xml: Sonar way
INFO: ------------- Run sensors on module LEMS_X_Exclude
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=25ms
INFO: Sensor JavaSquidSensor [java]
INFO: Configured Java source version (sonar.java.source): none
INFO: JavaClasspath initialization
WARN: Bytecode of dependencies was not provided for analysis of source files, you might end up with less precise results. Bytecode can be provided using sonar.java.libraries property.
INFO: JavaClasspath initialization (done) | time=12ms
INFO: JavaTestClasspath initialization
INFO: JavaTestClasspath initialization (done) | time=2ms
INFO: Java Main Files AST scan
INFO: 405 source files to be analyzed
INFO: 28/405 files analyzed, current file: src/main/java/com/mf/pdapi/orm/pojo/SoftwareLicenseStructure.java
INFO: 100/405 files analyzed, current file: src/main/java/com/mf/pdapi/services/pdapi/mappers/response/APGetEntitlementResponseMapper.java
INFO: 168/405 files analyzed, current file: src/main/java/com/mf/pdapi/web/controller/IndexController.java
INFO: 206/405 files analyzed, current file: src/main/java/com/mf/pdapi/web/controller/licensePanination/LicenseController.java
INFO: 258/405 files analyzed, current file: src/main/java/com/mf/pdapi/web/models/ProductVO.java
WARN: Invalid character encountered in file /opt/cloudhost/apps/.jenkins/workspace/TEMP_SONARQUBE_CI_CD_LEMS_BUILD_EXTRA/src/test/java/com/mf/pdapi/services/utils/SecurityHelperUtilTest.java at line 34 for encoding UTF-8. Please fix file content or configure the encoding to be used using property 'sonar.sourceEncoding'.
INFO: 405/405 source files have been analyzed
INFO: Java Main Files AST scan (done) | time=57784ms
INFO: Java Test Files AST scan
INFO: 0 source files to be analyzed
INFO: Java Test Files AST scan (done) | time=2ms
INFO: 0/0 source files have been analyzed
INFO: Sensor JavaSquidSensor [java] (done) | time=58443ms
INFO: Sensor SonarCSS Metrics [cssfamily]
INFO: Sensor SonarCSS Metrics [cssfamily] (done) | time=125ms
INFO: Sensor SonarCSS Rules [cssfamily]
ERROR: CSS rules were not executed. Error when running: 'node -v'. Is Node.js available during analysis?
org.sonarsource.nodejs.NodeCommandException: Error when running: 'node -v'. Is Node.js available during analysis?
at org.sonarsource.nodejs.NodeCommand.start(NodeCommand.java:83)
at org.sonarsource.nodejs.NodeCommandBuilderImpl.getVersion(NodeCommandBuilderImpl.java:196)
at org.sonarsource.nodejs.NodeCommandBuilderImpl.checkNodeCompatibility(NodeCommandBuilderImpl.java:169)
at org.sonarsource.nodejs.NodeCommandBuilderImpl.build(NodeCommandBuilderImpl.java:144)
at org.sonar.css.plugin.server.CssAnalyzerBridgeServer.initNodeCommand(CssAnalyzerBridgeServer.java:133)
at org.sonar.css.plugin.server.CssAnalyzerBridgeServer.startServer(CssAnalyzerBridgeServer.java:101)
at org.sonar.css.plugin.server.CssAnalyzerBridgeServer.startServerLazily(CssAnalyzerBridgeServer.java:153)
at org.sonar.css.plugin.CssRuleSensor.execute(CssRuleSensor.java:99)
at org.sonar.scanner.sensor.AbstractSensorWrapper.analyse(AbstractSensorWrapper.java:48)
at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:85)
at org.sonar.scanner.sensor.ModuleSensorsExecutor.lambda$execute$1(ModuleSensorsExecutor.java:59)
at org.sonar.scanner.sensor.ModuleSensorsExecutor.withModuleStrategy(ModuleSensorsExecutor.java:77)
at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:59)
at org.sonar.scanner.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:82)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:136)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:122)
at org.sonar.scanner.scan.ProjectScanContainer.scan(ProjectScanContainer.java:400)
at org.sonar.scanner.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:395)
at org.sonar.scanner.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:358)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:136)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:122)
at org.sonar.scanner.bootstrap.GlobalContainer.doAfterStart(GlobalContainer.java:141)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:136)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:122)
at org.sonar.batch.bootstrapper.Batch.doExecute(Batch.java:73)
at org.sonar.batch.bootstrapper.Batch.executeTask(Batch.java:99)
at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
at com.sun.proxy.$Proxy0.execute(Unknown Source)
at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:233)
at org.sonarsource.scanner.api.EmbeddedScanner.runAnalysis(EmbeddedScanner.java:151)
at org.sonarsource.scanner.cli.Main.runAnalysis(Main.java:123)
at org.sonarsource.scanner.cli.Main.execute(Main.java:77)
at org.sonarsource.scanner.cli.Main.main(Main.java:61)
Caused by: java.io.IOException: Cannot run program "node": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.sonarsource.nodejs.NodeCommand$ProcessWrapperImpl.start(NodeCommand.java:175)
at org.sonarsource.nodejs.NodeCommand.start(NodeCommand.java:79)
... 37 common frames omitted
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 39 common frames omitted
INFO: Sensor SonarCSS Rules [cssfamily] (done) | time=938ms
INFO: Sensor JaCoCo XML Report Importer [jacoco]
INFO: Importing 1 report(s). Turn your logs in debug mode in order to see the exhaustive list.
INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=253ms
INFO: Sensor SurefireSensor [java]
INFO: parsing [/opt/cloudhost/apps/.jenkins/workspace/TEMP_SONARQUBE_CI_CD_LEMS_BUILD_EXTRA/build/test-results/test]
INFO: Sensor SurefireSensor [java] (done) | time=35ms
INFO: Sensor JaCoCoSensor [java]
INFO: Sensor JaCoCoSensor [java] (done) | time=1ms
INFO: Sensor JavaXmlSensor [java]
INFO: 41 source files to be analyzed
INFO: Sensor JavaXmlSensor [java] (done) | time=526ms
INFO: Sensor HTML [web]
INFO: 41/41 source files have been analyzed
INFO: Sensor HTML [web] (done) | time=1841ms
INFO: Sensor XML Sensor [xml]
INFO: 52 source files to be analyzed
INFO: Sensor XML Sensor [xml] (done) | time=562ms
INFO: 52/52 source files have been analyzed
INFO: ------------- Run sensors on project
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=22ms
INFO: Sensor Java CPD Block Indexer
INFO: Sensor Java CPD Block Indexer (done) | time=640ms
INFO: 78 files had no CPD blocks
INFO: Calculating CPD for 427 files
INFO: CPD calculation finished
INFO: Analysis report generated in 378ms, dir size=9 MB
INFO: Analysis report compressed in 1574ms, zip size=3 MB
INFO: Analysis report uploaded in 263ms
The error is:
ERROR: CSS rules were not executed. Error when running: 'node -v'. Is Node.js available during analysis?
It means what you do not have nodejs installed. Just configure it in jenkins global tools menu, add it to your job and run it again.
I am fairly new with jenkins and sonar scanner. I have created a simple hello world gradle project in Intellij. After that was done, i added the Jenkinsfile to the root folder of the project and started working on it. When i try to build it in jenkins, i get an error message looking like this:
05:57:51.485 INFO: Scanner configuration file: /usr/lib/sonar-scanner/conf/sonar-scanner.properties
05:57:51.491 INFO: Project root configuration file: /root/src/sonar-project.properties
05:57:51.506 INFO: SonarQube Scanner 3.3.0.1492
05:57:51.506 INFO: Java 1.8.0_121 Oracle Corporation (64-bit)
05:57:51.506 INFO: Linux 3.10.0-957.5.1.el7.x86_64 amd64
05:57:51.702 DEBUG: keyStore is :
05:57:51.702 DEBUG: keyStore type is : jks
05:57:51.702 DEBUG: keyStore provider is :
05:57:51.702 DEBUG: init keystore
05:57:51.703 DEBUG: init keymanager of type SunX509
05:57:51.877 DEBUG: Create: /root/.sonar/cache
05:57:51.879 INFO: User cache: /root/.sonar/cache
05:57:51.879 DEBUG: Create: /root/.sonar/cache/_tmp
05:57:51.881 DEBUG: Extract sonar-scanner-api-batch in temp...
05:57:51.889 DEBUG: Get bootstrap index...
05:57:51.889 DEBUG: Download: http://10.221.4.33:9000/batch/index
05:57:51.968 DEBUG: Get bootstrap completed
05:57:51.986 DEBUG: Download http://10.221.4.33:9000/batch/file?name=sonar-scanner-engine-shaded-7.7-all.jar to /root/.sonar/cache/_tmp/fileCache3570509839777067796.tmp
05:57:52.323 DEBUG: Create isolated classloader...
05:57:52.333 DEBUG: Start temp cleaning...
05:57:52.338 DEBUG: Temp cleaning done
05:57:52.338 DEBUG: Execution getVersion
05:57:52.345 INFO: SonarQube server 7.7.0
05:57:52.346 DEBUG: Execution execute
05:57:52.801 INFO: Load global settings
05:57:52.879 DEBUG: GET 200 http://10.221.4.33:9000/api/settings/values.protobuf | time=58ms
05:57:52.936 INFO: Load global settings (done) | time=138ms
05:57:52.944 INFO: Server id: C9A263D1-AWnZoCq1phaxFFDSMo0l
05:57:52.966 DEBUG: Create : /root/.sonar/_tmp
05:57:52.967 INFO: User cache: /root/.sonar/cache
05:57:52.971 INFO: Load/download plugins
05:57:52.972 INFO: Load plugins index
05:57:52.981 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/installed | time=9ms
05:57:53.025 INFO: Load plugins index (done) | time=52ms
05:57:53.027 DEBUG: Download plugin 'scmgit' to '/root/.sonar/_tmp/fileCache4395394311979981807.tmp'
05:57:53.034 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=scmgit&acceptCompressions=pack200 | time=7ms
05:57:53.112 DEBUG: Download plugin 'jacoco' to '/root/.sonar/_tmp/fileCache6933684684880586527.tmp'
05:57:53.118 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=jacoco&acceptCompressions=pack200 | time=6ms
05:57:53.119 DEBUG: Download plugin 'ldap' to '/root/.sonar/_tmp/fileCache7133022802065402857.tmp'
05:57:53.128 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=ldap&acceptCompressions=pack200 | time=8ms
05:57:53.136 DEBUG: Download plugin 'csharp' to '/root/.sonar/_tmp/fileCache9082420825692938062.tmp'
05:57:53.150 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=csharp&acceptCompressions=pack200 | time=13ms
05:57:53.241 DEBUG: Download plugin 'cssfamily' to '/root/.sonar/_tmp/fileCache293189218006487968.tmp'
05:57:53.246 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=cssfamily&acceptCompressions=pack200 | time=5ms
05:57:53.301 DEBUG: Download plugin 'clojure' to '/root/.sonar/_tmp/fileCache948669080401350606.tmp'
05:57:53.307 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=clojure&acceptCompressions=pack200 | time=6ms
05:57:53.379 DEBUG: Download plugin 'flex' to '/root/.sonar/_tmp/fileCache3889788857842376851.tmp'
05:57:53.386 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=flex&acceptCompressions=pack200 | time=7ms
05:57:53.401 DEBUG: Download plugin 'go' to '/root/.sonar/_tmp/fileCache4205858474210720715.tmp'
05:57:53.408 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=go&acceptCompressions=pack200 | time=7ms
05:57:53.462 DEBUG: Download plugin 'web' to '/root/.sonar/_tmp/fileCache783608628579300027.tmp'
05:57:53.470 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=web&acceptCompressions=pack200 | time=8ms
05:57:53.502 DEBUG: Download plugin 'javascript' to '/root/.sonar/_tmp/fileCache895874836559468181.tmp'
05:57:53.523 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=javascript&acceptCompressions=pack200 | time=20ms
05:57:53.641 DEBUG: Download plugin 'java' to '/root/.sonar/_tmp/fileCache2653809152725468294.tmp'
05:57:53.649 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=java&acceptCompressions=pack200 | time=8ms
05:57:53.724 DEBUG: Download plugin 'kotlin' to '/root/.sonar/_tmp/fileCache4849792320171587582.tmp'
05:57:53.731 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=kotlin&acceptCompressions=pack200 | time=7ms
05:57:53.819 DEBUG: Download plugin 'php' to '/root/.sonar/_tmp/fileCache6204606293784138971.tmp'
05:57:53.826 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=php&acceptCompressions=pack200 | time=7ms
05:57:53.877 DEBUG: Download plugin 'python' to '/root/.sonar/_tmp/fileCache288964667021647401.tmp'
05:57:53.885 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=python&acceptCompressions=pack200 | time=7ms
05:57:53.910 DEBUG: Download plugin 'ruby' to '/root/.sonar/_tmp/fileCache5281102871587928818.tmp'
05:57:53.915 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=ruby&acceptCompressions=pack200 | time=4ms
05:57:53.993 DEBUG: Download plugin 'sonarscala' to '/root/.sonar/_tmp/fileCache1960676723457785292.tmp'
05:57:53.998 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=sonarscala&acceptCompressions=pack200 | time=5ms
05:57:54.070 DEBUG: Download plugin 'typescript' to '/root/.sonar/_tmp/fileCache7183740303261272847.tmp'
05:57:54.076 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=typescript&acceptCompressions=pack200 | time=6ms
05:57:54.095 DEBUG: Download plugin 'vbnet' to '/root/.sonar/_tmp/fileCache7315453031927695364.tmp'
05:57:54.100 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=vbnet&acceptCompressions=pack200 | time=5ms
05:57:54.170 DEBUG: Download plugin 'xml' to '/root/.sonar/_tmp/fileCache226747213883935028.tmp'
05:57:54.177 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=xml&acceptCompressions=pack200 | time=6ms
05:57:54.198 DEBUG: Download plugin 'scmsvn' to '/root/.sonar/_tmp/fileCache1337467676129254645.tmp'
05:57:54.203 DEBUG: GET 200 http://10.221.4.33:9000/api/plugins/download?plugin=scmsvn&acceptCompressions=pack200 | time=5ms
05:57:54.257 INFO: Load/download plugins (done) | time=1286ms
05:57:54.464 DEBUG: Plugins:
05:57:54.464 DEBUG: * SonarPython 1.13.0.2922 (python)
05:57:54.464 DEBUG: * SonarCSS 1.0.3.724 (cssfamily)
05:57:54.464 DEBUG: * SonarClojure 1.8.1 (clojure)
05:57:54.464 DEBUG: * JaCoCo 1.0.1.143 (jacoco)
05:57:54.464 DEBUG: * SonarGo 1.1.0.1612 (go)
05:57:54.464 DEBUG: * SonarKotlin 1.5.0.315 (kotlin)
05:57:54.464 DEBUG: * Svn 1.9.0.1295 (scmsvn)
05:57:54.464 DEBUG: * SonarJS 5.1.1.7506 (javascript)
05:57:54.464 DEBUG: * SonarRuby 1.5.0.315 (ruby)
05:57:54.465 DEBUG: * SonarScala 1.5.0.315 (sonarscala)
05:57:54.465 DEBUG: * SonarC# 7.14.0.8411 (csharp)
05:57:54.465 DEBUG: * SonarJava 5.12.1.17771 (java)
05:57:54.465 DEBUG: * LDAP 2.2.0.608 (ldap)
05:57:54.465 DEBUG: * SonarHTML 3.1.0.1615 (web)
05:57:54.465 DEBUG: * Git 1.8.0.1574 (scmgit)
05:57:54.465 DEBUG: * SonarFlex 2.4.0.1222 (flex)
05:57:54.465 DEBUG: * SonarXML 2.0.1.2020 (xml)
05:57:54.465 DEBUG: * SonarPHP 3.0.0.4537 (php)
05:57:54.465 DEBUG: * SonarTS 1.9.0.3766 (typescript)
05:57:54.465 DEBUG: * SonarVB 7.14.0.8411 (vbnet)
05:57:54.493 INFO: ------------------------------------------------------------------------
05:57:54.493 INFO: EXECUTION FAILURE
05:57:54.493 INFO: ------------------------------------------------------------------------
05:57:54.493 INFO: Total time: 3.042s
05:57:54.591 INFO: Final Memory: 9M/597M
05:57:54.591 INFO: ------------------------------------------------------------------------
05:57:54.591 ERROR: Error during SonarQube Scanner execution
05:57:54.591 ERROR: Tasks support was removed in SonarQube 7.6.
I have tried searching for the error (Tasks support was removed in Sonarqube 7.6) which really havent helped me at all so far. I've tried seperate ways of constructing the Jenkinsfile but seems to be no luck at all.
This is my jenkinsfile:
pipeline {
agent any
stages {
stage('List folder') {
steps {
sh 'ls'
}
}
stage('Make folder for docker') {
steps {
sh 'docker run -v $(pwd):/root/src newtmitch/sonar-scanner sonar-scanner -X \
-Dsonar.host.url=http://XXXX:9000 \
-Dsonar.jdbc.url=jdbc:h2:tcp://sonarqube/sonar \
-Dsonar.projectKey=MyProjectKey \
-Dsonar.projectName="My Project Name" \
-Dsonar.projectVersion=1 \
-Dsonar.projectBaseDir=/root/src \
-Dsonar.sources=/root/src'
}
}
}
}
It seems to be able to get the right docker image from sonar scanner, but why is it failing afterwards?
The steps should be somewhat simple:
1. Get image.
2. Build Gradle project
3. Scan project code
4. Profit.
In case anyone runs into this, i solved it by removing "sonar-scanner -X" from the docker run command.
unsure why it changed anything but now it can analyze the project code correctly.
thanks
Does anyone have any experience in this area?
I've tried so far with ffmpeg (using libx264 and libaacplus) muxing into mpegts over udp, but that mpegts muxer is obviously broken (confirmed from several different sources).
I've also tried with vlc, but it can only encode AAC-LC and not HE-AAC v2 (aacplus).
Anyway, the problem I need to solve is to have several different geographic locations, covered with webcams and I need udp to be able to have incoming streams all the time and not to worry about networks ups and downs (udp will simply continue sending packets when the network is up again). So, did any of you have any experience using streaming live media using h264 with aacplus over udp protocol and if yes, can you please give me any links or directions how to accomplish it.
Thanks a lot in advance.
I'm developing a system which is kind of a DVR that must periodically record a h264 file from a video device and at the same time provide a local preview to allow for adjusting the video parameters and camera view. Although I am no expert at all in this field, I had a relative success streaming h264 over UDP, so I'll try to share what didn't and what did work out for me, which was all based on ffmpeg as a server (no audio in my case).
Initially I had set up my application to simultaneously record the video to a file and feed it to a .ffm file for ffserver to stream it up using RTP/UDP for the camera preview. The problem with that approach was exactly that when the feeding ffmpeg process stopped to change the video file, the preview would stop and never resume although the subsequent ffmpeg process would have already started feeding ffserver again. With RTP ffserver apparently complains the frames timestamp started back at 0 instead of continuing where they have stopped. Then I realized that if I were able to send the h264 packets over pure UDP I would get exactly the same effect you described, with preview resuming as soon as the next ffmpeg process would take over.
While trying to understand ffmpeg documentation I also tried using the mpegts format, but doing that I was getting MPEG2 video at the player on the other end (this shows a multicast address, but it also worked for a specific target):
$ ffmpeg -y -f video4linux2 -i /dev/video2 -vcodec libx264 -preset ultrafast /mnt/hd/video.mp4 -an -f mpegts udp://224.124.0.1:5000
ffmpeg version N-35860-g62adc60, Copyright (c) 2000-2011 the FFmpeg developers
built on Dec 16 2011 09:47:41 with gcc 4.5.3
configuration: --prefix=/usr --libdir=/usr/lib --shlibdir=/usr/lib --mandir=/usr/man --disable-debug --enable-shared --disable-static --enab
le-pthreads --enable-libtheora --enable-libvorbis --enable-gpl --enable-version3 --enable-postproc --enable-swscale --enable-avfilter --enable
-libx264 --enable-libvpx --enable-librtmp --disable-indev='v4l,dv1394'
libavutil 51. 32. 0 / 51. 32. 0
libavcodec 53. 46. 0 / 53. 46. 0
libavformat 53. 26. 0 / 53. 26. 0
libavdevice 53. 4. 0 / 53. 4. 0
libavfilter 2. 53. 0 / 2. 53. 0
libswscale 2. 1. 0 / 2. 1. 0
libpostproc 51. 2. 0 / 51. 2. 0
[video4linux2,v4l2 # 0x8a96b00] Estimating duration from bitrate, this may be inaccurate
Input #0, video4linux2,v4l2, from '/dev/video2':
Duration: N/A, start: 1325538250.366878, bitrate: 27620 kb/s
Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240, 27620 kb/s, 29.97 tbr, 1000k tbn, 29.97 tbc
[buffer # 0x8a9d8c0] w:320 h:240 pixfmt:yuv420p tb:1/1000000 sar:0/1 sws_param:
[buffer # 0x8a9c860] w:320 h:240 pixfmt:yuv420p tb:1/1000000 sar:0/1 sws_param:
[libx264 # 0x8a97780] using cpu capabilities: MMX2 Cache64
[libx264 # 0x8a97780] profile Constrained Baseline, level 1.3
[libx264 # 0x8a97780] 264 - core 120 - H.264/MPEG-4 AVC codec - Copyleft 2003-2011 - http://www.videolan.org/x264.html - options: cabac=0 ref=
1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11
fast_pskip=1 chroma_qp_offset=0 threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=
0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
[mpegts # 0x8a98100] muxrate VBR, pcr every 2 pkts, sdt every 200, pat/pmt every 40 pkts
Output #0, mp4, to '/mnt/hd/video.mp4':
Metadata:
encoder : Lavf53.26.0
Stream #0:0: Video: h264 (![0][0][0] / 0x0021), yuv420p, 320x240, q=-1--1, 30k tbn, 29.97 tbc
Output #1, mpegts, to 'udp://224.124.0.1:5000':
Metadata:
encoder : Lavf53.26.0
Stream #1:0: Video: mpeg2video, yuv420p, 320x240, q=2-31, 200 kb/s, 90k tbn, 29.97 tbc
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo -> libx264)
Stream #0:0 -> #1:0 (rawvideo -> mpeg2video)
Press [q] to stop, [?] for help
On the client PC I was able to watch the video with ffplay, and it was indeed in MPEG2 format:
$ ffplay -f mpegts udp://224.124.0.1:5000
ffplay version N-35860-g62adc60, Copyright (c) 2003-2011 the FFmpeg developers
built on Dec 16 2011 09:47:41 with gcc 4.5.3
configuration: --prefix=/usr --libdir=/usr/lib --shlibdir=/usr/lib --mandir=/usr/man --disable-debug --enable-shared --disable-static --enable-pthreads --enable-libtheora --enable-libvorbis --enable-gpl --enable-version3 --enable-postproc --enable-swscale --enable-avfilter --enable-libx264 --enable-libvpx --enable-librtmp --disable-indev='v4l,dv1394'
libavutil 51. 32. 0 / 51. 32. 0
libavcodec 53. 46. 0 / 53. 46. 0
libavformat 53. 26. 0 / 53. 26. 0
libavdevice 53. 4. 0 / 53. 4. 0
libavfilter 2. 53. 0 / 2. 53. 0
libswscale 2. 1. 0 / 2. 1. 0
libpostproc 51. 2. 0 / 51. 2. 0
[mpegts # 0x80f02e0] Unable to seek back to the start
[mpeg2video # 0x8111a00] mpeg_decode_postinit() failure
Last message repeated 6 times
[mpegts # 0x80f02e0] max_analyze_duration 5000000 reached at 5005000
[mpegts # 0x80f02e0] Estimating duration from bitrate, this may be inaccurate
Input #0, mpegts, from 'udp://224.124.0.1:5000':
Duration: N/A, start: 255.420433, bitrate: 104857 kb/s
Program 1
Metadata:
service_name : Service01
service_provider: FFmpeg
Stream #0:0[0x100]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p, 320x240 [SAR 1:1 DAR 4:3], 104857 kb/s, 30.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
VLC also worked to play the stream, but only after specifying the switch --demux ffmpeg (thanks to this):
vlc -vv --demux ffmpeg udp://#224.124.0.1:5000
Since I also wanted the preview to be in h264 but I already had it encoding the file, I tried to use the copy codec for the UDP streaming, but ffmpeg failed with a segfault (versions included for reference):
ffmpeg -y -f video4linux2 -i /dev/video2 -vcodec libx264 -preset ultrafast /mnt/hd/video.mp4 -an -vcodec cop
y -f h264 udp://224.124.0.1:5000
ffmpeg version N-35860-g62adc60, Copyright (c) 2000-2011 the FFmpeg developers
built on Dec 16 2011 09:47:41 with gcc 4.5.3
configuration: --prefix=/usr --libdir=/usr/lib --shlibdir=/usr/lib --mandir=/usr/man --disable-debug --enable-shared --disable-static --enab
le-pthreads --enable-libtheora --enable-libvorbis --enable-gpl --enable-version3 --enable-postproc --enable-swscale --enable-avfilter --enable
-libx264 --enable-libvpx --enable-librtmp --disable-indev='v4l,dv1394'
libavutil 51. 32. 0 / 51. 32. 0
libavcodec 53. 46. 0 / 53. 46. 0
libavformat 53. 26. 0 / 53. 26. 0
libavdevice 53. 4. 0 / 53. 4. 0
libavfilter 2. 53. 0 / 2. 53. 0
libswscale 2. 1. 0 / 2. 1. 0
libpostproc 51. 2. 0 / 51. 2. 0
[video4linux2,v4l2 # 0x92c7b00] Estimating duration from bitrate, this may be inaccurate
Input #0, video4linux2,v4l2, from '/dev/video2':
Duration: N/A, start: 1325539132.411691, bitrate: 27620 kb/s
Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240, 27620 kb/s, 29.97 tbr, 1000k tbn, 29.97 tbc
[buffer # 0x92ce860] w:320 h:240 pixfmt:yuv420p tb:1/1000000 sar:0/1 sws_param:
[libx264 # 0x92c8780] using cpu capabilities: MMX2 Cache64
[libx264 # 0x92c8780] profile Constrained Baseline, level 1.3
[libx264 # 0x92c8780] 264 - core 120 - H.264/MPEG-4 AVC codec - Copyleft 2003-2011 - http://www.videolan.org/x264.html - options: cabac=0 ref=
1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11
fast_pskip=1 chroma_qp_offset=0 threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=
0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
Output #0, mp4, to '/mnt/hd/video.mp4':
Metadata:
encoder : Lavf53.26.0
Stream #0:0: Video: h264 (![0][0][0] / 0x0021), yuv420p, 320x240, q=-1--1, 30k tbn, 29.97 tbc
Output #1, h264, to 'udp://224.124.0.1:5000':
Metadata:
encoder : Lavf53.26.0
Stream #1:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240, q=2-31, 27620 kb/s, 90k tbn, 29.97 tbc
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo -> libx264)
Stream #0:0 -> #1:0 (copy)
Press [q] to stop, [?] for help
Segmentation fault
Although less than ideal, specifying the format h264 for the UDP streaming part led to a second concurrent h264 conversion, but it worked:
$ ffmpeg -y -f video4linux2 -i /dev/video2 -vcodec libx264 -preset ultrafast /mnt/hd/video.mp4 -an -f h264 -preset ultrafast udp://224.124.0.1:5000
ffmpeg version N-35860-g62adc60, Copyright (c) 2000-2011 the FFmpeg developers
built on Dec 16 2011 09:47:41 with gcc 4.5.3
configuration: --prefix=/usr --libdir=/usr/lib --shlibdir=/usr/lib --mandir=/usr/man --disable-debug --enable-shared --disable-static --enab
le-pthreads --enable-libtheora --enable-libvorbis --enable-gpl --enable-version3 --enable-postproc --enable-swscale --enable-avfilter --enable
-libx264 --enable-libvpx --enable-librtmp --disable-indev='v4l,dv1394'
libavutil 51. 32. 0 / 51. 32. 0
libavcodec 53. 46. 0 / 53. 46. 0
libavformat 53. 26. 0 / 53. 26. 0
libavdevice 53. 4. 0 / 53. 4. 0
libavfilter 2. 53. 0 / 2. 53. 0
libswscale 2. 1. 0 / 2. 1. 0
libpostproc 51. 2. 0 / 51. 2. 0
[video4linux2,v4l2 # 0x913ab00] Estimating duration from bitrate, this may be inaccurate
Input #0, video4linux2,v4l2, from '/dev/video2':
Duration: N/A, start: 1325539689.729735, bitrate: 27620 kb/s
Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240, 27620 kb/s, 29.97 tbr, 1000k tbn, 29.97 tbc
[buffer # 0x9141840] w:320 h:240 pixfmt:yuv420p tb:1/1000000 sar:0/1 sws_param:
[buffer # 0x913e480] w:320 h:240 pixfmt:yuv420p tb:1/1000000 sar:0/1 sws_param:
[libx264 # 0x913b780] using cpu capabilities: MMX2 Cache64
[libx264 # 0x913b780] profile Constrained Baseline, level 1.3
[libx264 # 0x913b780] 264 - core 120 - H.264/MPEG-4 AVC codec - Copyleft 2003-2011 - http://www.videolan.org/x264.html - options: cabac=0 ref=
1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11
fast_pskip=1 chroma_qp_offset=0 threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=
0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
[libx264 # 0x913c820] using cpu capabilities: MMX2 Cache64
[libx264 # 0x913c820] profile Constrained Baseline, level 1.3
Output #0, mp4, to '/mnt/hd/video.mp4':
Metadata:
encoder : Lavf53.26.0
Stream #0:0: Video: h264 (![0][0][0] / 0x0021), yuv420p, 320x240, q=-1--1, 30k tbn, 29.97 tbc
Output #1, h264, to 'udp://224.124.0.1:5000':
Metadata:
encoder : Lavf53.26.0
Stream #1:0: Video: h264, yuv420p, 320x240, q=-1--1, 90k tbn, 29.97 tbc
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo -> libx264)
Stream #0:0 -> #1:0 (rawvideo -> libx264)
Press [q] to stop, [?] for help
On the client side ffplay repeatedly complained about some missing information, but after some seconds it finally showed up the video, which was ok but a bit choppy:
$ ffplay -f h264 udp://224.124.0.1:5000
[h264 # 0xa0be740] non-existing PPS referenced
[h264 # 0xa0be740] non-existing PPS 0 referenced
[h264 # 0xa0be740] decode_slice_header error
[h264 # 0xa0be740] no frame!
(...)
[h264 # 0xa0be740] non-existing PPS referenced
[h264 # 0xa0be740] non-existing PPS 0 referenced
[h264 # 0xa0be740] decode_slice_header error
[h264 # 0xa0be740] no frame!
[h264 # 0xa0e72e0] max_analyze_duration 5000000 reached at 5013967
[h264 # 0xa0e72e0] Estimating duration from bitrate, this may be inaccurate
Input #0, h264, from 'udp://224.124.0.1:5000':
Duration: N/A, bitrate: N/A
Stream #0:0: Video: h264 (Constrained Baseline), yuv420p, 320x240, 47.27 fps, 29.97 tbr, 1200k tbn, 59.94 tbc
[h264 # 0xa0be740] Missing reference picture
[h264 # 0xa0be740] decode_slice_header error
[h264 # 0xa0be740] concealing 300 DC, 300 AC, 300 MV errors
[h264 # 0xa0be740] Missing reference picture 0KB sq= 0B f=0/0 0/0
[h264 # 0xa0be740] decode_slice_header error
[h264 # 0xa0be740] mmco: unref short failure
[h264 # 0xa0be740] concealing 300 DC, 300 AC, 300 MV errors
10.78 A-V: 0.000 fd= 0 aq= 0KB vq= 0KB sq= 0B f=0/0
I then tried VLC with the same ''--demux ffmpeg'' switch; it also complained about SPS/PPS (don't know what it is about yet) but in the end it played the video really smoothly:
$ vlc -v --demux ffmpeg udp/h264://#224.124.0.1:5000
VLC media player 1.1.12 The Luggage (revision exported)
Blocked: call to unsetenv("DBUS_ACTIVATION_ADDRESS")
Blocked: call to unsetenv("DBUS_ACTIVATION_BUS_TYPE")
[0x943346c] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
Blocked: call to setlocale(6, "")
Blocked: call to setlocale(6, "")
[0x94cde8c] qt4 interface error: Unable to load extensions module
[0x96720e4] h264 demux error: this doesn't look like a H264 ES stream, continuing anyway
[0x963989c] access_udp access warning: unimplemented query in control
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x94be644] main input error: Invalid PCR value in ES_OUT_SET_(GROUP_)PCR !
[0xb2a325f4] avcodec decoder warning: disabling direct rendering
[0x96dfac4] main video output warning: vlc_object_find_name(postproc) is not safe!
[0x94c2474] signals interface warning: signal 17 overridden (0xb6f31030)
[0x94c2474] signals interface warning: /usr/lib/qt/lib/libQtCore.so.4(?)[(nil)]
[0x96dfac4] main video output warning: late picture skipped (32703 > -4)
I'm still trying to figure out what is missing, but the results are pleasing already. After VLC is playing I can stop the feeding ffmpeg that playback pauses and resumes right away when I restart ffmpeg on the other end.
Hope this can be of any help, and please let me know if you have any additional info on using h264 as the chosen format in place of mpegts - I suspect this missing SPS/PPS info might have something to do with it.