Related
Instead of doing a CV and train the Random Forest multiple times I would like to use the OOB Error as and unbiased estimation of the generalized error.
And for a few data points (in the low thousands), does it make sense to use the OOB error instead of CV, since it might be possible that only a few data points are oob?
So far I could only find something about it from this issue thread https://github.com/mlr-org/mlr/issues/338 from mlr. I think it is suggested to use a hould out split with almost only training data.
I found the insample resampling method https://mlr3.mlr-org.com/reference/mlr_resamplings_insample.html which uses the same data for training and testing.
This is my code:
learner = as_learner(
po("select", selector=selector_name(selection)) %>>% po("learner", learner=lrn("regr.ranger"))
)
sp = ps(
regr.ranger.mtry.ratio = p_dbl(0, 1),
regr.ranger.replace = p_fct(c(TRUE, FALSE)),
regr.ranger.sample.fraction = p_dbl(0.1, 1),
regr.ranger.num.trees = p_int(1, 2000)
)
at = auto_tuner(
resampling = rsmp("insample"),
method = "random_search",
learner = learner,
measure = msr("oob_error"),
term_evals = 5,
search_space=sp
)
learners = c(at)
resamplings = rsmp("cv", folds = 5)
design = benchmark_grid(task, learners, resamplings)
bmr = benchmark(design)
But when running the code above, I get the error: Error in learner$oob_error() : attempt to apply non-function
The problem is that the resulting GraphLearner does not have the method oob_error() anymore. This is similar to the issues here:
https://github.com/mlr-org/mlr3pipelines/issues/291
Edit: Add workaround.
This suggestion should be seen as a workaround.
The idea is that it is possible to write custom measures as mentioned in the comments. A tutorial on that can be found in the mlr3 book
This custom measure only works in this specific case, because it is tailored to the specific structure of the GraphLearner. For a different learner, the measure would have to be adjusted.
library(mlr3verse)
#> Loading required package: mlr3
task = tsk("mtcars")
selection = c("mpg", "cyl")
learner = as_learner(
po("select", selector = selector_name(selection)) %>>% po("learner", learner = lrn("regr.ranger"))
)
sp = ps(
regr.ranger.mtry.ratio = p_dbl(0, 1),
regr.ranger.replace = p_fct(c(TRUE, FALSE)),
regr.ranger.sample.fraction = p_dbl(0.1, 1),
regr.ranger.num.trees = p_int(1, 2000)
)
MyMeasure = R6::R6Class(
"MyMeasure",
inherit = MeasureRegr,
public = list(
initialize = function() {
super$initialize(
id = "MyMeasure",
range = c(-Inf, Inf),
minimize = TRUE,
predict_type = "response",
properties = "requires_learner"
)
}
),
private = list(
.score = function(prediction, learner, ...) {
model = learner$state$model$regr.ranger
if (is.null(model)) stop("Set store_models = TRUE.")
model$model$prediction.error
}
)
)
at = auto_tuner(
resampling = rsmp("insample"),
method = "random_search",
learner = learner,
measure = MyMeasure$new(),
term_evals = 1,
search_space = sp,
store_models = TRUE
)
learners = c(at)
resamplings = rsmp("cv", folds = 5)
design = benchmark_grid(task, learners, resamplings)
lgr::get_logger("mlr3")$set_threshold(NULL)
lgr::get_logger("mlr3tuning")$set_threshold(NULL)
bmr = benchmark(design)
#> INFO [23:28:45.638] [mlr3] Running benchmark with 5 resampling iterations
#> INFO [23:28:45.740] [mlr3] Applying learner 'select.regr.ranger.tuned' on task 'mtcars' (iter 1/5)
#> INFO [23:28:47.112] [bbotk] Starting to optimize 4 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=1, k=0]'
#> INFO [23:28:47.158] [bbotk] Evaluating 1 configuration(s)
#> INFO [23:28:47.201] [mlr3] Running benchmark with 1 resampling iterations
#> INFO [23:28:47.209] [mlr3] Applying learner 'select.regr.ranger' on task 'mtcars' (iter 1/1)
#> INFO [23:28:47.346] [mlr3] Finished benchmark
#> INFO [23:28:47.419] [bbotk] Result of batch 1:
#> INFO [23:28:47.424] [bbotk] regr.ranger.mtry.ratio regr.ranger.replace regr.ranger.sample.fraction
#> INFO [23:28:47.424] [bbotk] 0.5708216 TRUE 0.4830289
#> INFO [23:28:47.424] [bbotk] regr.ranger.num.trees MyMeasure warnings errors runtime_learners
#> INFO [23:28:47.424] [bbotk] 1209 11.39842 0 0 0.124
#> INFO [23:28:47.424] [bbotk] uhash
#> INFO [23:28:47.424] [bbotk] abfcaa2f-8b01-4821-8e8b-1d209fbe2229
#> INFO [23:28:47.444] [bbotk] Finished optimizing after 1 evaluation(s)
#> INFO [23:28:47.445] [bbotk] Result:
#> INFO [23:28:47.447] [bbotk] regr.ranger.mtry.ratio regr.ranger.replace regr.ranger.sample.fraction
#> INFO [23:28:47.447] [bbotk] 0.5708216 TRUE 0.4830289
#> INFO [23:28:47.447] [bbotk] regr.ranger.num.trees learner_param_vals x_domain MyMeasure
#> INFO [23:28:47.447] [bbotk] 1209 <list[6]> <list[4]> 11.39842
#> INFO [23:28:47.616] [mlr3] Applying learner 'select.regr.ranger.tuned' on task 'mtcars' (iter 2/5)
#> INFO [23:28:47.733] [bbotk] Starting to optimize 4 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=1, k=0]'
#> INFO [23:28:47.758] [bbotk] Evaluating 1 configuration(s)
#> INFO [23:28:47.799] [mlr3] Running benchmark with 1 resampling iterations
#> INFO [23:28:47.807] [mlr3] Applying learner 'select.regr.ranger' on task 'mtcars' (iter 1/1)
#> INFO [23:28:47.900] [mlr3] Finished benchmark
#> INFO [23:28:47.969] [bbotk] Result of batch 1:
#> INFO [23:28:47.971] [bbotk] regr.ranger.mtry.ratio regr.ranger.replace regr.ranger.sample.fraction
#> INFO [23:28:47.971] [bbotk] 0.9683787 FALSE 0.4303312
#> INFO [23:28:47.971] [bbotk] regr.ranger.num.trees MyMeasure warnings errors runtime_learners
#> INFO [23:28:47.971] [bbotk] 112 9.594568 0 0 0.084
#> INFO [23:28:47.971] [bbotk] uhash
#> INFO [23:28:47.971] [bbotk] 4bb2742b-49e2-4b02-adc4-ffaa70aef8d4
#> INFO [23:28:47.984] [bbotk] Finished optimizing after 1 evaluation(s)
#> INFO [23:28:47.984] [bbotk] Result:
#> INFO [23:28:47.986] [bbotk] regr.ranger.mtry.ratio regr.ranger.replace regr.ranger.sample.fraction
#> INFO [23:28:47.986] [bbotk] 0.9683787 FALSE 0.4303312
#> INFO [23:28:47.986] [bbotk] regr.ranger.num.trees learner_param_vals x_domain MyMeasure
#> INFO [23:28:47.986] [bbotk] 112 <list[6]> <list[4]> 9.594568
#> INFO [23:28:48.116] [mlr3] Applying learner 'select.regr.ranger.tuned' on task 'mtcars' (iter 3/5)
#> INFO [23:28:48.241] [bbotk] Starting to optimize 4 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=1, k=0]'
#> INFO [23:28:48.266] [bbotk] Evaluating 1 configuration(s)
#> INFO [23:28:48.308] [mlr3] Running benchmark with 1 resampling iterations
#> INFO [23:28:48.316] [mlr3] Applying learner 'select.regr.ranger' on task 'mtcars' (iter 1/1)
#> INFO [23:28:48.413] [mlr3] Finished benchmark
#> INFO [23:28:48.480] [bbotk] Result of batch 1:
#> INFO [23:28:48.483] [bbotk] regr.ranger.mtry.ratio regr.ranger.replace regr.ranger.sample.fraction
#> INFO [23:28:48.483] [bbotk] 0.4089994 TRUE 0.1780138
#> INFO [23:28:48.483] [bbotk] regr.ranger.num.trees MyMeasure warnings errors runtime_learners
#> INFO [23:28:48.483] [bbotk] 620 38.86261 0 0 0.089
#> INFO [23:28:48.483] [bbotk] uhash
#> INFO [23:28:48.483] [bbotk] 9b47bdb0-15dc-421d-9091-db2e6c41cbee
#> INFO [23:28:48.495] [bbotk] Finished optimizing after 1 evaluation(s)
#> INFO [23:28:48.496] [bbotk] Result:
#> INFO [23:28:48.498] [bbotk] regr.ranger.mtry.ratio regr.ranger.replace regr.ranger.sample.fraction
#> INFO [23:28:48.498] [bbotk] 0.4089994 TRUE 0.1780138
#> INFO [23:28:48.498] [bbotk] regr.ranger.num.trees learner_param_vals x_domain MyMeasure
#> INFO [23:28:48.498] [bbotk] 620 <list[6]> <list[4]> 38.86261
#> INFO [23:28:48.646] [mlr3] Applying learner 'select.regr.ranger.tuned' on task 'mtcars' (iter 4/5)
#> INFO [23:28:48.763] [bbotk] Starting to optimize 4 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=1, k=0]'
#> INFO [23:28:48.788] [bbotk] Evaluating 1 configuration(s)
#> INFO [23:28:48.829] [mlr3] Running benchmark with 1 resampling iterations
#> INFO [23:28:48.837] [mlr3] Applying learner 'select.regr.ranger' on task 'mtcars' (iter 1/1)
#> INFO [23:28:48.959] [mlr3] Finished benchmark
#> INFO [23:28:49.027] [bbotk] Result of batch 1:
#> INFO [23:28:49.030] [bbotk] regr.ranger.mtry.ratio regr.ranger.replace regr.ranger.sample.fraction
#> INFO [23:28:49.030] [bbotk] 0.3449179 FALSE 0.344375
#> INFO [23:28:49.030] [bbotk] regr.ranger.num.trees MyMeasure warnings errors runtime_learners
#> INFO [23:28:49.030] [bbotk] 1004 11.96155 0 0 0.112
#> INFO [23:28:49.030] [bbotk] uhash
#> INFO [23:28:49.030] [bbotk] d14754c3-ab73-4777-84bd-10daa10318f0
#> INFO [23:28:49.043] [bbotk] Finished optimizing after 1 evaluation(s)
#> INFO [23:28:49.044] [bbotk] Result:
#> INFO [23:28:49.046] [bbotk] regr.ranger.mtry.ratio regr.ranger.replace regr.ranger.sample.fraction
#> INFO [23:28:49.046] [bbotk] 0.3449179 FALSE 0.344375
#> INFO [23:28:49.046] [bbotk] regr.ranger.num.trees learner_param_vals x_domain MyMeasure
#> INFO [23:28:49.046] [bbotk] 1004 <list[6]> <list[4]> 11.96155
#> INFO [23:28:49.203] [mlr3] Applying learner 'select.regr.ranger.tuned' on task 'mtcars' (iter 5/5)
#> INFO [23:28:49.327] [bbotk] Starting to optimize 4 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=1, k=0]'
#> INFO [23:28:49.352] [bbotk] Evaluating 1 configuration(s)
#> INFO [23:28:49.393] [mlr3] Running benchmark with 1 resampling iterations
#> INFO [23:28:49.401] [mlr3] Applying learner 'select.regr.ranger' on task 'mtcars' (iter 1/1)
#> INFO [23:28:49.537] [mlr3] Finished benchmark
#> INFO [23:28:49.614] [bbotk] Result of batch 1:
#> INFO [23:28:49.616] [bbotk] regr.ranger.mtry.ratio regr.ranger.replace regr.ranger.sample.fraction
#> INFO [23:28:49.616] [bbotk] 0.4485645 FALSE 0.4184389
#> INFO [23:28:49.616] [bbotk] regr.ranger.num.trees MyMeasure warnings errors runtime_learners
#> INFO [23:28:49.616] [bbotk] 1931 12.59067 0 0 0.127
#> INFO [23:28:49.616] [bbotk] uhash
#> INFO [23:28:49.616] [bbotk] 295d1dc0-810d-4351-9bb4-7255fca38be3
#> INFO [23:28:49.629] [bbotk] Finished optimizing after 1 evaluation(s)
#> INFO [23:28:49.630] [bbotk] Result:
#> INFO [23:28:49.631] [bbotk] regr.ranger.mtry.ratio regr.ranger.replace regr.ranger.sample.fraction
#> INFO [23:28:49.631] [bbotk] 0.4485645 FALSE 0.4184389
#> INFO [23:28:49.631] [bbotk] regr.ranger.num.trees learner_param_vals x_domain MyMeasure
#> INFO [23:28:49.631] [bbotk] 1931 <list[6]> <list[4]> 12.59067
#> INFO [23:28:49.806] [mlr3] Finished benchmark
Created on 2023-01-30 with reprex v2.0.2
I want to tune an xgboost learner and set the parameter early_stopping_rounds to 10% of the parameter nrounds (whichever is generated each time).
Should be a simple thing to do in general (i.e. tuning a parameter relative to another) but I can't make it work, see example below:
library(mlr3verse)
#> Loading required package: mlr3
learner = lrn('surv.xgboost', nrounds = to_tune(50, 5000),
early_stopping_rounds = to_tune(ps(
a = p_int(10,5000), # had to put something in here, `early_stopping_rounds` also doesn't work
.extra_trafo = function(x, param_set) {
list(early_stopping_rounds = ceiling(0.1 * x$nrounds))
}, .allow_dangling_dependencies = TRUE)))
#> Error in self$assert(xs): Assertion on 'xs' failed: early_stopping_rounds: tune token invalid: to_tune(ps(a = p_int(10, 5000), .extra_trafo = function(x, param_set) { list(early_stopping_rounds = ceiling(0.1 * x$nrounds)) }, .allow_dangling_dependencies = TRUE)) generates points that are not compatible with param early_stopping_rounds.
#> Bad value:
#> numeric(0)
#> Parameter:
#> id class lower upper levels default
#> 1: early_stopping_rounds ParamInt 1 Inf .
# this works though:
pam = ps(z = p_int(-3,3), x = p_int(0,10),
.extra_trafo = function(x, param_set) {
x$z = 2*(x$x) # overwrite z as 2*x
x
})
dplyr::bind_rows(generate_design_random(pam, 5)$transpose())
#> # A tibble: 5 × 2
#> z x
#> <dbl> <int>
#> 1 2 1
#> 2 14 7
#> 3 8 4
#> 4 12 6
#> 5 20 10
Created on 2022-08-29 by the reprex package (v2.0.1)
The reason why your solution is not working is that you are using x$nrounds from the paramset in which it does not exist.
You can use this as a workaround.
library(mlr3verse)
#> Loading required package: mlr3
search_space = ps(
nrounds = p_int(lower = 50, upper = 5000),
.extra_trafo = function(x, param_set) {
x$early_stopping_rounds = as.integer(ceiling(0.1 * x$nrounds))
x
}
)
task = tsk("iris")
learner = lrn("classif.xgboost")
terminator = trm("evals", n_evals = 10)
tuner = tnr("random_search")
at = AutoTuner$new(
learner = learner,
resampling = rsmp("holdout"),
measure = msr("classif.ce"),
search_space = search_space,
terminator = terminator,
tuner = tuner
)
at$train(task)
#> INFO [13:12:50.316] [bbotk] Starting to optimize 1 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=10, k=0]'
#> INFO [13:12:50.351] [bbotk] Evaluating 1 configuration(s)
#> INFO [13:12:50.406] [mlr3] Running benchmark with 1 resampling iterations
#> INFO [13:12:50.441] [mlr3] Applying learner 'classif.xgboost' on task 'iris' (iter 1/1)
#> INFO [13:12:51.837] [mlr3] Finished benchmark
#> INFO [13:12:51.865] [bbotk] Result of batch 1:
#> INFO [13:12:51.867] [bbotk] nrounds classif.ce warnings errors runtime_learners
#> INFO [13:12:51.867] [bbotk] 3497 0 0 0 1.387
#> INFO [13:12:51.867] [bbotk] uhash
#> INFO [13:12:51.867] [bbotk] 8a8e7d03-3166-4c03-8e06-78fe9f4e8a35
#> INFO [13:12:51.870] [bbotk] Evaluating 1 configuration(s)
#> INFO [13:12:51.918] [mlr3] Running benchmark with 1 resampling iterations
#> INFO [13:12:51.926] [mlr3] Applying learner 'classif.xgboost' on task 'iris' (iter 1/1)
#> INFO [13:12:53.650] [mlr3] Finished benchmark
#> INFO [13:12:53.680] [bbotk] Result of batch 2:
#> INFO [13:12:53.681] [bbotk] nrounds classif.ce warnings errors runtime_learners
#> INFO [13:12:53.681] [bbotk] 4197 0 0 0 1.718
#> INFO [13:12:53.681] [bbotk] uhash
#> INFO [13:12:53.681] [bbotk] 85c94228-4419-4e7e-8f4b-6e289a2d2900
#> INFO [13:12:53.684] [bbotk] Evaluating 1 configuration(s)
#> INFO [13:12:53.725] [mlr3] Running benchmark with 1 resampling iterations
#> INFO [13:12:53.730] [mlr3] Applying learner 'classif.xgboost' on task 'iris' (iter 1/1)
#> INFO [13:12:54.648] [mlr3] Finished benchmark
#> INFO [13:12:54.683] [bbotk] Result of batch 3:
#> INFO [13:12:54.685] [bbotk] nrounds classif.ce warnings errors runtime_learners
#> INFO [13:12:54.685] [bbotk] 2199 0 0 0 0.911
#> INFO [13:12:54.685] [bbotk] uhash
#> INFO [13:12:54.685] [bbotk] cd33357f-13bf-4851-8da3-f3c1b58755a6
#> INFO [13:12:54.687] [bbotk] Evaluating 1 configuration(s)
#> INFO [13:12:54.727] [mlr3] Running benchmark with 1 resampling iterations
#> INFO [13:12:54.732] [mlr3] Applying learner 'classif.xgboost' on task 'iris' (iter 1/1)
#> INFO [13:12:56.651] [mlr3] Finished benchmark
#> INFO [13:12:56.679] [bbotk] Result of batch 4:
#> INFO [13:12:56.681] [bbotk] nrounds classif.ce warnings errors runtime_learners
#> INFO [13:12:56.681] [bbotk] 4679 0 0 0 1.909
#> INFO [13:12:56.681] [bbotk] uhash
#> INFO [13:12:56.681] [bbotk] 4efe832d-9163-4447-9e4c-5a41190de74c
#> INFO [13:12:56.684] [bbotk] Evaluating 1 configuration(s)
#> INFO [13:12:56.722] [mlr3] Running benchmark with 1 resampling iterations
#> INFO [13:12:56.727] [mlr3] Applying learner 'classif.xgboost' on task 'iris' (iter 1/1)
#> INFO [13:12:57.850] [mlr3] Finished benchmark
#> INFO [13:12:57.875] [bbotk] Result of batch 5:
#> INFO [13:12:57.877] [bbotk] nrounds classif.ce warnings errors runtime_learners
#> INFO [13:12:57.877] [bbotk] 2422 0 0 0 1.116
#> INFO [13:12:57.877] [bbotk] uhash
#> INFO [13:12:57.877] [bbotk] 8db417a2-0b6e-4844-9c07-4c83e899964e
#> INFO [13:12:57.880] [bbotk] Evaluating 1 configuration(s)
#> INFO [13:12:57.915] [mlr3] Running benchmark with 1 resampling iterations
#> INFO [13:12:57.920] [mlr3] Applying learner 'classif.xgboost' on task 'iris' (iter 1/1)
#> INFO [13:12:59.769] [mlr3] Finished benchmark
#> INFO [13:12:59.794] [bbotk] Result of batch 6:
#> INFO [13:12:59.795] [bbotk] nrounds classif.ce warnings errors runtime_learners
#> INFO [13:12:59.795] [bbotk] 4721 0 0 0 1.843
#> INFO [13:12:59.795] [bbotk] uhash
#> INFO [13:12:59.795] [bbotk] d37d1ec0-bd89-408b-9c29-ecf657a9bbb5
#> INFO [13:12:59.798] [bbotk] Evaluating 1 configuration(s)
#> INFO [13:12:59.833] [mlr3] Running benchmark with 1 resampling iterations
#> INFO [13:12:59.838] [mlr3] Applying learner 'classif.xgboost' on task 'iris' (iter 1/1)
#> INFO [13:13:00.336] [mlr3] Finished benchmark
#> INFO [13:13:00.369] [bbotk] Result of batch 7:
#> INFO [13:13:00.371] [bbotk] nrounds classif.ce warnings errors runtime_learners
#> INFO [13:13:00.371] [bbotk] 1323 0 0 0 0.491
#> INFO [13:13:00.371] [bbotk] uhash
#> INFO [13:13:00.371] [bbotk] 89f100b9-2f9e-4c47-8734-9165dc215277
#> INFO [13:13:00.374] [bbotk] Evaluating 1 configuration(s)
#> INFO [13:13:00.412] [mlr3] Running benchmark with 1 resampling iterations
#> INFO [13:13:00.417] [mlr3] Applying learner 'classif.xgboost' on task 'iris' (iter 1/1)
#> INFO [13:13:01.706] [mlr3] Finished benchmark
#> INFO [13:13:01.736] [bbotk] Result of batch 8:
#> INFO [13:13:01.737] [bbotk] nrounds classif.ce warnings errors runtime_learners
#> INFO [13:13:01.737] [bbotk] 3424 0 0 0 1.282
#> INFO [13:13:01.737] [bbotk] uhash
#> INFO [13:13:01.737] [bbotk] 9f754641-fa5f-420a-b09a-32fe7512bb9b
#> INFO [13:13:01.740] [bbotk] Evaluating 1 configuration(s)
#> INFO [13:13:01.784] [mlr3] Running benchmark with 1 resampling iterations
#> INFO [13:13:01.789] [mlr3] Applying learner 'classif.xgboost' on task 'iris' (iter 1/1)
#> INFO [13:13:03.160] [mlr3] Finished benchmark
#> INFO [13:13:03.189] [bbotk] Result of batch 9:
#> INFO [13:13:03.191] [bbotk] nrounds classif.ce warnings errors runtime_learners
#> INFO [13:13:03.191] [bbotk] 3432 0 0 0 1.365
#> INFO [13:13:03.191] [bbotk] uhash
#> INFO [13:13:03.191] [bbotk] 47cfe02f-fd4e-4382-9343-b4c4ac274d91
#> INFO [13:13:03.194] [bbotk] Evaluating 1 configuration(s)
#> INFO [13:13:03.232] [mlr3] Running benchmark with 1 resampling iterations
#> INFO [13:13:03.237] [mlr3] Applying learner 'classif.xgboost' on task 'iris' (iter 1/1)
#> INFO [13:13:04.387] [mlr3] Finished benchmark
#> INFO [13:13:04.413] [bbotk] Result of batch 10:
#> INFO [13:13:04.415] [bbotk] nrounds classif.ce warnings errors runtime_learners
#> INFO [13:13:04.415] [bbotk] 2991 0 0 0 1.142
#> INFO [13:13:04.415] [bbotk] uhash
#> INFO [13:13:04.415] [bbotk] a1b9d503-0dae-4c5d-ba50-ffd27a754032
#> INFO [13:13:04.421] [bbotk] Finished optimizing after 10 evaluation(s)
#> INFO [13:13:04.422] [bbotk] Result:
#> INFO [13:13:04.423] [bbotk] nrounds learner_param_vals x_domain classif.ce
#> INFO [13:13:04.423] [bbotk] 3497 <list[4]> <list[2]> 0
Created on 2022-08-29 by the reprex package (v2.0.1)
So I'm playing around with Scrapy & Splash and I'm running into some issues.
I tried running my spiders, and kept getting HTTP 502 & 504 errors. Okay, so I tried to check out Splash in my browser.
First I did "sudo docker run -p 8050:8050 -p 5023:5023 scrapinghub/splash --max-timeout 3600 -v3" to start Splash running, then I went to localhost:8050. Web UI opens up properly, and I'm able to enter code.
Here is the basic function I'm trying to run:
function main(splash, args)
assert(splash:autoload("https://code.jquery.com/jquery-3.1.1.min.js"))
splash.resource_timeout = 30.0
splash.images_enabled = false
assert(splash:go(args.url))
assert(splash:wait(0.5))
return {
html = splash:html(),
--png = splash:png(),
--har = splash:har(),
}
end
I try to render http://boingboing.net/blog, using this function, and get an 'invalid hostname' LUA error; here are the logs:
2017-08-01 18:26:28+0000 [-] Log opened.
2017-08-01 18:26:28.077457 [-] Splash version: 3.0
2017-08-01 18:26:28.077838 [-] Qt 5.9.1, PyQt 5.9, WebKit 602.1, sip 4.19.3, Twisted 16.1.1, Lua 5.2
2017-08-01 18:26:28.077900 [-] Python 3.5.2 (default, Nov 17 2016, 17:05:23) [GCC 5.4.0 20160609]
2017-08-01 18:26:28.077984 [-] Open files limit: 65536
2017-08-01 18:26:28.078046 [-] Can't bump open files limit
2017-08-01 18:26:28.180376 [-] Xvfb is started: ['Xvfb', ':1937726875', '-screen', '0', '1024x768x24', '-nolisten', 'tcp']
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
2017-08-01 18:26:28.226937 [-] proxy profiles support is enabled, proxy profiles path: /etc/splash/proxy-profiles
2017-08-01 18:26:28.301002 [-] verbosity=3
2017-08-01 18:26:28.301116 [-] slots=50
2017-08-01 18:26:28.301202 [-] argument_cache_max_entries=500
2017-08-01 18:26:28.301530 [-] Web UI: enabled, Lua: enabled (sandbox: enabled)
2017-08-01 18:26:28.302122 [-] Site starting on 8050
2017-08-01 18:26:28.302219 [-] Starting factory <twisted.web.server.Site object at 0x7ffa08390dd8>
2017-08-01 18:26:32.660457 [-] "172.17.0.1" - - [01/Aug/2017:18:26:32 +0000] "GET / HTTP/1.1" 200 7677 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
2017-08-01 18:27:18.860020 [-] "172.17.0.1" - - [01/Aug/2017:18:27:18 +0000] "GET /info?wait=0.5&images=1&expand=1&timeout=3600.0&url=http%3A%2F%2Fboingboing.net%2Fblog&lua_source=function+main%28splash%2C+args%29%0D%0A++assert%28splash%3Aautoload%28%22https%3A%2F%2Fcode.jquery.com%2Fjquery-3.1.1.min.js%22%29%29%0D%0A++splash.resource_timeout+%3D+30.0%0D%0A++splash.images_enabled+%3D+false%0D%0A++assert%28splash%3Ago%28args.url%29%29%0D%0A++assert%28splash%3Await%280.5%29%29%0D%0A++return+%7B%0D%0A++++html+%3D+splash%3Ahtml%28%29%2C%0D%0A++++--png+%3D+splash%3Apng%28%29%2C%0D%0A++++--har+%3D+splash%3Ahar%28%29%2C%0D%0A++%7D%0D%0Aend HTTP/1.1" 200 5656 "http://localhost:8050/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
2017-08-01 18:27:19.038565 [pool] initializing SLOT 0
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
process 1: D-Bus library appears to be incorrectly set up; failed to read machine uuid: UUID file '/etc/machine-id' should contain a hex string of length 32, not length 0, with no other text
See the manual page for dbus-uuidgen to correct this issue.
2017-08-01 18:27:19.066765 [render] [140711856519656] viewport size is set to 1024x768
2017-08-01 18:27:19.066964 [pool] [140711856519656] SLOT 0 is starting
2017-08-01 18:27:19.067071 [render] [140711856519656] function main(splash, args)\r\n assert(splash:autoload("https://code.jquery.com/jquery-3.1.1.min.js"))\r\n splash.resource_timeout = 30.0\r\n splash.images_enabled = false\r\n assert(splash:go(args.url))\r\n assert(splash:wait(0.5))\r\n return {\r\n html = splash:html(),\r\n --png = splash:png(),\r\n --har = splash:har(),\r\n }\r\nend
2017-08-01 18:27:19.070107 [render] [140711856519656] [lua_runner] dispatch cmd_id=__START__
2017-08-01 18:27:19.070270 [render] [140711856519656] [lua_runner] arguments are for command __START__, waiting for result of __START__
2017-08-01 18:27:19.070352 [render] [140711856519656] [lua_runner] entering dispatch/loop body, args=()
2017-08-01 18:27:19.070424 [render] [140711856519656] [lua_runner] send None
2017-08-01 18:27:19.070496 [render] [140711856519656] [lua_runner] send (lua) None
2017-08-01 18:27:19.070657 [render] [140711856519656] [lua_runner] got AsyncBrowserCommand(id=None, name='http_get', kwargs={'url': 'https://code.jquery.com/jquery-3.1.1.min.js', 'callback': '<a callback>'})
2017-08-01 18:27:19.070755 [render] [140711856519656] [lua_runner] instructions used: 70
2017-08-01 18:27:19.070834 [render] [140711856519656] [lua_runner] executing AsyncBrowserCommand(id=0, name='http_get', kwargs={'url': 'https://code.jquery.com/jquery-3.1.1.min.js', 'callback': '<a callback>'})
2017-08-01 18:27:19.071141 [network] [140711856519656] GET https://code.jquery.com/jquery-3.1.1.min.js
qt.network.ssl: QSslSocket: cannot resolve SSLv2_client_method
qt.network.ssl: QSslSocket: cannot resolve SSLv2_server_method
2017-08-01 18:27:19.082150 [pool] [140711856519656] SLOT 0 is working
2017-08-01 18:27:19.082298 [pool] [140711856519656] queued
2017-08-01 18:28:39.151814 [network-manager] Download error 3: the remote host name was not found (invalid hostname) (https://code.jquery.com/jquery-3.1.1.min.js)
2017-08-01 18:28:39.152087 [network-manager] Finished downloading https://code.jquery.com/jquery-3.1.1.min.js
2017-08-01 18:28:39.152202 [render] [140711856519656] [lua_runner] dispatch cmd_id=0
2017-08-01 18:28:39.152268 [render] [140711856519656] [lua_runner] arguments are for command 0, waiting for result of 0
2017-08-01 18:28:39.152339 [render] [140711856519656] [lua_runner] entering dispatch/loop body, args=(PyResult('return', None, 'invalid_hostname'),)
2017-08-01 18:28:39.152400 [render] [140711856519656] [lua_runner] send PyResult('return', None, 'invalid_hostname')
2017-08-01 18:28:39.152468 [render] [140711856519656] [lua_runner] send (lua) (b'return', None, b'invalid_hostname')
2017-08-01 18:28:39.152582 [render] [140711856519656] [lua_runner] instructions used: 79
2017-08-01 18:28:39.152642 [render] [140711856519656] [lua_runner] caught LuaError LuaError('[string "function main(splash, args)\\r..."]:2: invalid_hostname',)
2017-08-01 18:28:39.152816 [pool] [140711856519656] SLOT 0 finished with an error <splash.qtrender_lua.LuaRender object at 0x7ffa08477e48>: [Failure instance: Traceback: <class 'splash.exceptions.ScriptError'>: {'error': 'invalid_hostname', 'type': 'LUA_ERROR', 'source': '[string "function main(splash, args)\r..."]', 'message': 'Lua error: [string "function main(splash, args)\r..."]:2: invalid_hostname', 'line_number': 2}
/app/splash/browser_tab.py:1180:_return_reply
/app/splash/qtrender_lua.py:901:callback
/app/splash/lua_runner.py:27:return_result
/app/splash/qtrender.py:17:stop_on_error_wrapper
--- <exception caught here> ---
/app/splash/qtrender.py:15:stop_on_error_wrapper
/app/splash/qtrender_lua.py:2257:dispatch
/app/splash/lua_runner.py:195:dispatch
]
2017-08-01 18:28:39.152883 [pool] [140711856519656] SLOT 0 is closing <splash.qtrender_lua.LuaRender object at 0x7ffa08477e48>
2017-08-01 18:28:39.152944 [render] [140711856519656] [splash] clearing 0 objects
2017-08-01 18:28:39.153026 [render] [140711856519656] close is requested by a script
2017-08-01 18:28:39.153304 [render] [140711856519656] cancelling 0 remaining timers
2017-08-01 18:28:39.153374 [pool] [140711856519656] SLOT 0 done with <splash.qtrender_lua.LuaRender object at 0x7ffa08477e48>
2017-08-01 18:28:39.153997 [events] {"user-agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0", "error": {"error": 400, "info": {"error": "invalid_hostname", "type": "LUA_ERROR", "source": "[string \"function main(splash, args)\r...\"]", "message": "Lua error: [string \"function main(splash, args)\r...\"]:2: invalid_hostname", "line_number": 2}, "type": "ScriptError", "description": "Error happened while executing Lua script"}, "active": 0, "status_code": 400, "maxrss": 107916, "qsize": 0, "path": "/execute", "timestamp": 1501612119, "fds": 18, "args": {"render_all": false, "http_method": "GET", "png": 1, "url": "http://boingboing.net/blog", "wait": 0.5, "html": 1, "response_body": false, "har": 1, "load_args": {}, "lua_source": "function main(splash, args)\r\n assert(splash:autoload(\"https://code.jquery.com/jquery-3.1.1.min.js\"))\r\n splash.resource_timeout = 30.0\r\n splash.images_enabled = false\r\n assert(splash:go(args.url))\r\n assert(splash:wait(0.5))\r\n return {\r\n html = splash:html(),\r\n --png = splash:png(),\r\n --har = splash:har(),\r\n }\r\nend", "resource_timeout": 0, "uid": 140711856519656, "save_args": [], "viewport": "1024x768", "timeout": 3600, "images": 1}, "client_ip": "172.17.0.1", "rendertime": 80.11527562141418, "method": "POST", "_id": 140711856519656, "load": [0.46, 0.51, 0.54]}
2017-08-01 18:28:39.154127 [-] "172.17.0.1" - - [01/Aug/2017:18:28:38 +0000] "POST /execute HTTP/1.1" 400 325 "http://localhost:8050/info?wait=0.5&images=1&expand=1&timeout=3600.0&url=http%3A%2F%2Fboingboing.net%2Fblog&lua_source=function+main%28splash%2C+args%29%0D%0A++assert%28splash%3Aautoload%28%22https%3A%2F%2Fcode.jquery.com%2Fjquery-3.1.1.min.js%22%29%29%0D%0A++splash.resource_timeout+%3D+30.0%0D%0A++splash.images_enabled+%3D+false%0D%0A++assert%28splash%3Ago%28args.url%29%29%0D%0A++assert%28splash%3Await%280.5%29%29%0D%0A++return+%7B%0D%0A++++html+%3D+splash%3Ahtml%28%29%2C%0D%0A++++--png+%3D+splash%3Apng%28%29%2C%0D%0A++++--har+%3D+splash%3Ahar%28%29%2C%0D%0A++%7D%0D%0Aend" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
2017-08-01 18:28:39.154237 [pool] SLOT 0 is available
If I try it without loading up JQuery first, I get a 'network5' LUA error (which is some species of timeout). Logs for that are as follows:
2017-08-01 18:31:07.110255 [-] "172.17.0.1" - - [01/Aug/2017:18:31:06 +0000] "GET /info?wait=0.5&images=1&expand=1&timeout=3600.0&url=http%3A%2F%2Fboingboing.net%2Fblog&lua_source=function+main%28splash%2C+args%29%0D%0A++--assert%28splash%3Aautoload%28%22https%3A%2F%2Fcode.jquery.com%2Fjquery-3.1.1.min.js%22%29%29%0D%0A++splash.resource_timeout+%3D+30.0%0D%0A++splash.images_enabled+%3D+false%0D%0A++assert%28splash%3Ago%28args.url%29%29%0D%0A++assert%28splash%3Await%280.5%29%29%0D%0A++return+%7B%0D%0A++++html+%3D+splash%3Ahtml%28%29%2C%0D%0A++++--png+%3D+splash%3Apng%28%29%2C%0D%0A++++--har+%3D+splash%3Ahar%28%29%2C%0D%0A++%7D%0D%0Aend HTTP/1.1" 200 5658 "http://localhost:8050/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
2017-08-01 18:31:07.489653 [pool] initializing SLOT 1
2017-08-01 18:31:07.490576 [render] [140711856961016] viewport size is set to 1024x768
2017-08-01 18:31:07.490692 [pool] [140711856961016] SLOT 1 is starting
2017-08-01 18:31:07.490829 [render] [140711856961016] function main(splash, args)\r\n --assert(splash:autoload("https://code.jquery.com/jquery-3.1.1.min.js"))\r\n splash.resource_timeout = 30.0\r\n splash.images_enabled = false\r\n assert(splash:go(args.url))\r\n assert(splash:wait(0.5))\r\n return {\r\n html = splash:html(),\r\n --png = splash:png(),\r\n --har = splash:har(),\r\n }\r\nend
2017-08-01 18:31:07.493641 [render] [140711856961016] [lua_runner] dispatch cmd_id=__START__
2017-08-01 18:31:07.493782 [render] [140711856961016] [lua_runner] arguments are for command __START__, waiting for result of __START__
2017-08-01 18:31:07.493865 [render] [140711856961016] [lua_runner] entering dispatch/loop body, args=()
2017-08-01 18:31:07.493937 [render] [140711856961016] [lua_runner] send None
2017-08-01 18:31:07.494010 [render] [140711856961016] [lua_runner] send (lua) None
2017-08-01 18:31:07.494270 [render] [140711856961016] [lua_runner] got AsyncBrowserCommand(id=None, name='go', kwargs={'baseurl': None, 'http_method': 'GET', 'headers': None, 'body': None, 'url': 'http://boingboing.net/blog', 'errback': '<an errback>', 'callback': '<a callback>'})
2017-08-01 18:31:07.494416 [render] [140711856961016] [lua_runner] instructions used: 166
2017-08-01 18:31:07.494502 [render] [140711856961016] [lua_runner] executing AsyncBrowserCommand(id=0, name='go', kwargs={'baseurl': None, 'http_method': 'GET', 'headers': None, 'body': None, 'url': 'http://boingboing.net/blog', 'errback': '<an errback>', 'callback': '<a callback>'})
2017-08-01 18:31:07.494576 [render] [140711856961016] HAR event: _onStarted
2017-08-01 18:31:07.494697 [render] [140711856961016] callback 0 is connected to loadFinished
2017-08-01 18:31:07.495031 [network] [140711856961016] GET http://boingboing.net/blog
2017-08-01 18:31:07.495617 [pool] [140711856961016] SLOT 1 is working
2017-08-01 18:31:07.495741 [pool] [140711856961016] queued
2017-08-01 18:31:37.789845 [network-manager] timed out, aborting: http://boingboing.net/blog
2017-08-01 18:31:37.790154 [network-manager] Finished downloading http://boingboing.net/blog
2017-08-01 18:31:37.791064 [render] [140711856961016] mainFrame().urlChanged http://boingboing.net/blog
2017-08-01 18:31:37.796078 [render] [140711856961016] mainFrame().initialLayoutCompleted
2017-08-01 18:31:37.796343 [render] [140711856961016] loadFinished: RenderErrorInfo(type='Network', code=5, text='Operation canceled', url='http://boingboing.net/blog')
2017-08-01 18:31:37.796420 [render] [140711856961016] loadFinished: disconnecting callback 0
2017-08-01 18:31:37.796518 [render] [140711856961016] [lua_runner] dispatch cmd_id=0
2017-08-01 18:31:37.796576 [render] [140711856961016] [lua_runner] arguments are for command 0, waiting for result of 0
2017-08-01 18:31:37.796640 [render] [140711856961016] [lua_runner] entering dispatch/loop body, args=(PyResult('return', None, 'network5'),)
2017-08-01 18:31:37.796699 [render] [140711856961016] [lua_runner] send PyResult('return', None, 'network5')
2017-08-01 18:31:37.796765 [render] [140711856961016] [lua_runner] send (lua) (b'return', None, b'network5')
2017-08-01 18:31:37.796883 [render] [140711856961016] [lua_runner] instructions used: 175
2017-08-01 18:31:37.796943 [render] [140711856961016] [lua_runner] caught LuaError LuaError('[string "function main(splash, args)\\r..."]:5: network5',)
2017-08-01 18:31:37.797093 [pool] [140711856961016] SLOT 1 finished with an error <splash.qtrender_lua.LuaRender object at 0x7ffa083ff828>: [Failure instance: Traceback: <class 'splash.exceptions.ScriptError'>: {'error': 'network5', 'type': 'LUA_ERROR', 'source': '[string "function main(splash, args)\r..."]', 'message': 'Lua error: [string "function main(splash, args)\r..."]:5: network5', 'line_number': 5}
/app/splash/browser_tab.py:533:_on_content_ready
/app/splash/qtrender_lua.py:702:error
/app/splash/lua_runner.py:27:return_result
/app/splash/qtrender.py:17:stop_on_error_wrapper
--- <exception caught here> ---
/app/splash/qtrender.py:15:stop_on_error_wrapper
/app/splash/qtrender_lua.py:2257:dispatch
/app/splash/lua_runner.py:195:dispatch
]
2017-08-01 18:31:37.797158 [pool] [140711856961016] SLOT 1 is closing <splash.qtrender_lua.LuaRender object at 0x7ffa083ff828>
2017-08-01 18:31:37.797217 [render] [140711856961016] [splash] clearing 0 objects
2017-08-01 18:31:37.797310 [render] [140711856961016] close is requested by a script
2017-08-01 18:31:37.797430 [render] [140711856961016] cancelling 0 remaining timers
2017-08-01 18:31:37.797491 [pool] [140711856961016] SLOT 1 done with <splash.qtrender_lua.LuaRender object at 0x7ffa083ff828>
2017-08-01 18:31:37.798067 [events] {"user-agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0", "error": {"error": 400, "info": {"error": "network5", "type": "LUA_ERROR", "source": "[string \"function main(splash, args)\r...\"]", "message": "Lua error: [string \"function main(splash, args)\r...\"]:5: network5", "line_number": 5}, "type": "ScriptError", "description": "Error happened while executing Lua script"}, "active": 0, "status_code": 400, "maxrss": 113372, "qsize": 0, "path": "/execute", "timestamp": 1501612297, "fds": 21, "args": {"render_all": false, "http_method": "GET", "png": 1, "url": "http://boingboing.net/blog", "wait": 0.5, "html": 1, "response_body": false, "har": 1, "load_args": {}, "lua_source": "function main(splash, args)\r\n --assert(splash:autoload(\"https://code.jquery.com/jquery-3.1.1.min.js\"))\r\n splash.resource_timeout = 30.0\r\n splash.images_enabled = false\r\n assert(splash:go(args.url))\r\n assert(splash:wait(0.5))\r\n return {\r\n html = splash:html(),\r\n --png = splash:png(),\r\n --har = splash:har(),\r\n }\r\nend", "resource_timeout": 0, "uid": 140711856961016, "save_args": [], "viewport": "1024x768", "timeout": 3600, "images": 1}, "client_ip": "172.17.0.1", "rendertime": 30.308406591415405, "method": "POST", "_id": 140711856961016, "load": [0.39, 0.42, 0.49]}
2017-08-01 18:31:37.798190 [-] "172.17.0.1" - - [01/Aug/2017:18:31:37 +0000] "POST /execute HTTP/1.1" 400 309 "http://localhost:8050/info?wait=0.5&images=1&expand=1&timeout=3600.0&url=http%3A%2F%2Fboingboing.net%2Fblog&lua_source=function+main%28splash%2C+args%29%0D%0A++--assert%28splash%3Aautoload%28%22https%3A%2F%2Fcode.jquery.com%2Fjquery-3.1.1.min.js%22%29%29%0D%0A++splash.resource_timeout+%3D+30.0%0D%0A++splash.images_enabled+%3D+false%0D%0A++assert%28splash%3Ago%28args.url%29%29%0D%0A++assert%28splash%3Await%280.5%29%29%0D%0A++return+%7B%0D%0A++++html+%3D+splash%3Ahtml%28%29%2C%0D%0A++++--png+%3D+splash%3Apng%28%29%2C%0D%0A++++--har+%3D+splash%3Ahar%28%29%2C%0D%0A++%7D%0D%0Aend" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
2017-08-01 18:31:37.798294 [pool] SLOT 1 is available
If I additionally comment out the resource_timeout line, I get a network3 LUA error (again, invalid hostname, but this time presenting differently).
Any idea what I'm doing wrong?
As it turns out, it wasn't a Scrapy/Splash issue at all -- it was a Docker / IP route / network admin issue. The network admins set it up so that I can only make HTTP requests through a particular destination; adding "--net=host" to my docker start-up seems to have fixed this.
This webpage was very helpful.
Try changing
function main(splash, args)
...
assert(splash:go(args.url))
...
to
function main(splash)
...
assert(splash:go(splash.args.url))
...
At least that's how it reads when I open Splash on port 8050 in the default script. With that change, your script works for me.
I'm having issues with HLS streaming and the HTML5 mwembed video player. I'm using mwembed v2.29.1.2, Kaltura CE 6.0, and Wowza Streaming Engine 4.1.2. Playing video through the Kaltura flash video player works fine, (streaming through Wowza with RTMP) but when trying to access the video on an iOS device, I get the message "Error: No source video was found."
I've tested the Wowza streaming directly and have not issues playing any of the videos directly: http://wowza.medquestreviews.com/kaltura/_definst_/mp4:content/entry/data/4/268/0_uxutn4hk_0_v5nyr0u2_1/playlist.m3u8
Here is a video with the issue: http://medquestreviews.com/step-3-high-yield/hematology/chapter-3-transfusion-reactions
I've looked through the Kaltura log files, apache_access, apache_errors, Kaltura_api_v3, etc. and the only error related to this that I can find is "error-10" in the apache log and [thumbnailAction->execute] NOTICE: Error - no FileSync for entry in the api log.
How can I go about finding the URL that Kaltura is using to look for the HLS stream? I've double checked the remote storage settings and everything is how it should be.
Accessing the m3u8 playlist/manifest itself works too: http://kmc.medquestreviews.com/p/101/sp/10100/playManifest/entryId/0_uxutn4hk/format/applehttp/a.m3u8
JS debug output from the HTML5 player is:
kWidget: Kaltura HTML5 Version: 2.29.1.2
11170242:1458kWidget: appendScriptUrls
load.php:1565getMarker> No <meta name="ResourceLoaderDynamicStyles"> found, inserting dynamically.
load.php:1565jQuery.fn.embedPlayer :: #kaltura-player
load.php:1565processEmbedPlayers:: playerSelector: #kaltura-player
load.php:1565EmbedPlayer:: addElement:: kaltura-player
load.php:1565processEmbedPlayers::runPlayerSwap::kaltura-player
load.php:1565EmbedPlayer::init:kaltura-player duration is: NaN, size: 355x200
load.php:1565EmbedPlayer::mediaElement:init:kaltura-player
load.php:1565processEmbedPlayers::swapEmbedPlayerElement: kaltura-player
load.php:1565processEmbedPlayers::trigger:: EmbedPlayerNewPlayer kaltura-player
11170242:1458kWidget: jsCallbackReady for kaltura-player
load.php:1565EmbedPlayer::addPlayerElement :trigger startPlayerBuildOut:kaltura-player
load.php:1565EmbedPlayer::checkPlayerSources: kaltura-player
load.php:1565KWidgetSupport::loadAndUpdatePlayerData
load.php:1565KWidgetSupport:: trigger Kaltura_CheckConfig
load.php:1565Error: function enableNativeControls should be implemented by embed player interface
load.php:1565Warning: mediawiki.UtilitiesTime, trying to get npt time on NaN:NaN
load.php:1565Warning: mediawiki.UtilitiesTime, trying to get npt time on NaN:NaN
load.php:1565!!EmbedPlayer:updatePosterHTML:kaltura-player poster:http://kmc.medquestreviews.com/p/101/sp/10100/thumbnail/entry_id/0_mip1dmg3/version/100000/width/355/height/200
load.php:1565EmbedPlayer:updatePosterHTML:kaltura-player poster:http://kmc.medquestreviews.com/p/101/sp/10100/thumbnail/entry_id/0_mip1dmg3/version/100000/width/355/height/200
load.php:1565Error: EmbedPlayer, Must override 'addPlayScreenWithNativeOffScreen' with player inteface
load.php:1565KAnalytics :: doSendAnalyticsEvent > MEDIA_LOADED
load.php:1565KWidgetSupport:: trigger KalturaSupport_DoneWithUiConf
load.php:1565EmbedPlayer::setupSourcePlayer: kaltura-player sources: 0
load.php:1565EmbedPlayer::mediaElement::autoSelectSource
load.php:1565MediaElement::GetPlayableSources mimeFilter:undefined 0 sources playable out of 0
load.php:1565EmbedPlayer::showPlayerError
load.php:1565PlayerLayoutBuilder::displayAlert:: Error
load.php:1565PlayerLayoutBuilder:: displayMenuOverlay
load.php:1565EmbedPlayer::pause()
load.php:1565EmbedPlayer::pauseInterfaceUpdate
load.php:1565EmbedPlayer:: setupSourcePlayer > player ready ( but with errors )
load.php:1565LiveStream :: removeLiveStreamStatusMonitor
load.php:1565EmbedPlayer:: Trigger: widgetLoaded
load.php:1565KAnalytics :: doSendAnalyticsEvent > WIDGET_LOADED
I keep running into these lines in the kaltura_api_v3 log:
2015-04-21 16:56:59 [0.000160] [71.234.194.254] [1637142224] [PS2] [kFileSyncUtils::getReadyFileSyncForKey] NOTICE: kFileSyncUtils::getReadyFileSyncForKey - FileSync was found but doesn't exists locally
2015-04-21 16:56:59 [0.000116] [71.234.194.254] [1637142224] [PS2] [flvclipperAction->execute] NOTICE: Error - no FileSync for flavor [0_v5nyr0u2]
2015-04-21 16:56:59 [0.000139] [71.234.194.254] [1637142224] [PS2] [KExternalErrors::dieError] ERR: exiting on error 5 - required file was not found
Files are being stored locally as well as transferred via FTP to the Wowza server which is setup as a remote storage profile.
kaltura:
[ec2-user#ip-xxxx 268]$ ll *uxutn4hk*
-rwxrwxrwx 1 apache apache 150237 Oct 27 19:17 0_uxutn4hk_0_d6g74099_2.jpg
-rwxrwxrwx 1 apache apache 142816 Oct 27 19:31 0_uxutn4hk_0_jnagm88n_0.conv.log
-rwxrwxrwx 1 apache apache 139861815 Oct 27 19:31 0_uxutn4hk_0_jnagm88n_1.mp4
-rw-r--r-- 1 apache apache 326215877 Oct 27 19:17 0_uxutn4hk_0_v3a7ee0g_1.mp4
-rwxrwxrwx 1 apache apache 39239 Oct 27 19:20 0_uxutn4hk_0_v5nyr0u2_0.conv.log
-rwxrwxrwx 1 apache apache 35568566 Oct 27 19:20 0_uxutn4hk_0_v5nyr0u2_1.mp4
-rwxrwxrwx 1 apache apache 75212 Oct 27 19:24 0_uxutn4hk_0_ye4swbad_0.conv.log
-rwxrwxrwx 1 apache apache 86644533 Oct 27 19:24 0_uxutn4hk_0_ye4swbad_1.mp4
[ec2-user#ip-xxxx 268]$ pwd
/home/ec2-user/kaltura-app-dir/web/content/entry/data/4/268
wowza:
[ec2-user#ip-xxxx 268]$ ll *uxutn4hk*
-rwxrwxrwx 1 kaltura_ftp wowza 139861815 Oct 27 19:32 0_uxutn4hk_0_jnagm88n_1.mp4
-rwxrwxrwx 1 kaltura_ftp wowza 326215877 Oct 27 19:32 0_uxutn4hk_0_v3a7ee0g_1.mp4
-rwxrwxrwx 1 kaltura_ftp wowza 35568566 Oct 27 19:20 0_uxutn4hk_0_v5nyr0u2_1.mp4
-rwxrwxrwx 1 kaltura_ftp wowza 86644533 Oct 27 19:24 0_uxutn4hk_0_ye4swbad_1.mp4
[ec2-user#ip-xxxx 268]$ pwd
/media/video_content/content/entry/data/4/268
the Kaltura database file_sync table also has entries for the flavor asset:
id partner_id object_type object_id version object_sub_type dc original created_at updated_at ready_at sync_time status file_type linked_id link_count file_root file_path file_size custom_data
6350 101 4 0_v5nyr0u2 1 1 3 0 2014-10-27 15:20:20 2014-10-27 15:20:35 2014-10-27 15:20:35 NULL 2 3 NULL NULL /opt/kaltura/web/ /content/entry/data/4/268/0_uxutn4hk_0_v5nyr0u2_1.mp4 -1 NULL
6348 101 4 0_v5nyr0u2 1 1 0 1 2014-10-27 15:20:13 2014-10-27 15:20:35 2014-10-27 15:20:13 NULL 3 1 NULL NULL /opt/kaltura/web/ /content/entry/data/4/268/0_uxutn4hk_0_v5nyr0u2_1.mp4 35568566 NULL
6349 101 4 0_v5nyr0u2 0 2 0 1 2014-10-27 15:20:13 2014-10-27 15:20:13 2014-10-27 15:20:13 NULL 2 1 NULL NULL /opt/kaltura/web/ /content/entry/data/4/268/0_uxutn4hk_0_v5nyr0u2_0.conv.log 39239 NULL
The errors in the log also differ slightly depending on if I load the whole page with the embedded video or I try to load the dataUrl from the iframe player config JS.
logs from access whole page:
2015-04-21 17:14:13 [0.000241] [71.234.194.254] [753153844] [PS2] [kFileSyncUtils::getReadyLocalFilePathForKey] NOTICE: kFileSyncUtils::getReadyLocalFilePathForKey - key [object_type:[4], object_id:[0_v3a7ee0g], version:[1], object_sub_type[1], partner_id[101]], strict []
2015-04-21 17:14:13 [0.000158] [71.234.194.254] [753153844] [PS2] [kFileSyncUtils::getReadyFileSyncForKey] NOTICE: kFileSyncUtils::getReadyFileSyncForKey - key [object_type:[4], object_id:[0_v3a7ee0g], version:[1], object_sub_type[1], partner_id[101]], fetch_from_remote_if_no_local [], strict []
2015-04-21 17:14:13 [0.000932] [71.234.194.254] [753153844] [PS2] [KalturaStatement->execute] DEBUG: /* ip-10-239-172-244[753153844][propel2] */ SELECT file_sync.ID, file_sync.PARTNER_ID, file_sync.OBJECT_TYPE, file_sync.OBJECT_ID, file_sync.VERSION, file_sync.OBJECT_SUB_TYPE, file_sync.DC, file_sync.ORIGINAL, file_sync.CREATED_AT, file_sync.UPDATED_AT, file_sync.READY_AT, file_sync.SYNC_TIME, file_sync.STATUS, file_sync.FILE_TYPE, file_sync.LINKED_ID, file_sync.LINK_COUNT, file_sync.FILE_ROOT, file_sync.FILE_PATH, file_sync.FILE_SIZE FROM `file_sync` WHERE file_sync.OBJECT_ID='0_v3a7ee0g' AND file_sync.OBJECT_TYPE='4' AND file_sync.OBJECT_SUB_TYPE='1' AND file_sync.VERSION='1' AND file_sync.DC='0' AND (file_sync.STATUS='2' AND file_sync.STATUS NOT IN ('3','4')) ORDER BY file_sync.DC ASC
2015-04-21 17:14:13 [0.000438] [71.234.194.254] [753153844] [PS2] [KalturaStatement->execute] DEBUG: Sql took - 0.00026082992553711 seconds
2015-04-21 17:14:13 [0.000187] [71.234.194.254] [753153844] [PS2] [kFileSyncUtils::getReadyFileSyncForKey] NOTICE: kFileSyncUtils::getReadyFileSyncForKey - FileSync was not found
2015-04-21 17:14:13 [0.000749] [71.234.194.254] [753153844] [PS2] [kCoreException->__construct] ERR: Code: [1] Message: [no ready filesync on current DC]
2015-04-21 17:14:13 [0.000670] [71.234.194.254] [753153844] [PS2] [KalturaStatement->execute] DEBUG: /* ip-10-239-172-244[753153844][propel2] */ SELECT flavor_asset.ID, flavor_asset.INT_ID, flavor_asset.PARTNER_ID, flavor_asset.TAGS, flavor_asset.CREATED_AT, flavor_asset.UPDATED_AT, flavor_asset.DELETED_AT, flavor_asset.ENTRY_ID, flavor_asset.FLAVOR_PARAMS_ID, flavor_asset.STATUS, flavor_asset.VERSION, flavor_asset.DESCRIPTION, flavor_asset.WIDTH, flavor_asset.HEIGHT, flavor_asset.BITRATE, flavor_asset.FRAME_RATE, flavor_asset.SIZE, flavor_asset.IS_ORIGINAL, flavor_asset.FILE_EXT, flavor_asset.CONTAINER_FORMAT, flavor_asset.VIDEO_CODEC_ID, flavor_asset.TYPE, flavor_asset.CUSTOM_DATA FROM `flavor_asset` WHERE flavor_asset.ENTRY_ID='0_uxutn4hk' AND flavor_asset.IS_ORIGINAL='1' AND flavor_asset.STATUS<>'3' LIMIT 1
2015-04-21 17:14:13 [0.000400] [71.234.194.254] [753153844] [PS2] [KalturaStatement->execute] DEBUG: Sql took - 0.00022983551025391 seconds
2015-04-21 17:14:13 [0.000927] [71.234.194.254] [753153844] [PS2] [KalturaStatement->execute] DEBUG: /* ip-10-239-172-244[753153844][propel2] */ SELECT file_sync.ID, file_sync.PARTNER_ID, file_sync.OBJECT_TYPE, file_sync.OBJECT_ID, file_sync.VERSION, file_sync.OBJECT_SUB_TYPE, file_sync.DC, file_sync.ORIGINAL, file_sync.CREATED_AT, file_sync.UPDATED_AT, file_sync.READY_AT, file_sync.SYNC_TIME, file_sync.STATUS, file_sync.FILE_TYPE, file_sync.LINKED_ID, file_sync.LINK_COUNT, file_sync.FILE_ROOT, file_sync.FILE_PATH, file_sync.FILE_SIZE FROM `file_sync` WHERE file_sync.OBJECT_ID='0_v3a7ee0g' AND file_sync.OBJECT_TYPE='4' AND file_sync.OBJECT_SUB_TYPE='1' AND file_sync.VERSION='1' AND file_sync.ORIGINAL='1' AND file_sync.STATUS NOT IN ('3','4')
2015-04-21 17:14:13 [0.000468] [71.234.194.254] [753153844] [PS2] [KalturaStatement->execute] DEBUG: Sql took - 0.00028085708618164 seconds
2015-04-21 17:14:13 [0.000202] [71.234.194.254] [753153844] [PS2] [thumbnailAction->execute] NOTICE: Error - no FileSync for entry [0_uxutn4hk]
2015-04-21 17:14:13 [0.000207] [71.234.194.254] [753153844] [PS2] [KExternalErrors::dieError] ERR: exiting on error 10 - missing thumbnail fileSync for entry
accessing dataUrl directly: http://kmc.medquestreviews.com/p/101/sp/10100/flvclipper/entry_id/0_uxutn4hk/version/0
2015-04-21 17:17:26 [0.000106] [71.234.194.254] [1347402480] [PS2] [kFileSyncUtils::getReadyFileSyncForKey] NOTICE: kFileSyncUtils::getReadyFileSyncForKey - key [object_type:[4], object_id:[0_v5nyr0u2], version:[1], object_sub_type[1], partner_id[101]], fetch_from_remote_if_no_local [1], strict []
2015-04-21 17:17:26 [0.000733] [71.234.194.254] [1347402480] [PS2] [KalturaStatement->execute] DEBUG: /* ip-10-239-172-244[1347402480][propel] */ SELECT file_sync.ID, file_sync.PARTNER_ID, file_sync.OBJECT_TYPE, file_sync.OBJECT_ID, file_sync.VERSION, file_sync.OBJECT_SUB_TYPE, file_sync.DC, file_sync.ORIGINAL, file_sync.CREATED_AT, file_sync.UPDATED_AT, file_sync.READY_AT, file_sync.SYNC_TIME, file_sync.STATUS, file_sync.FILE_TYPE, file_sync.LINKED_ID, file_sync.LINK_COUNT, file_sync.FILE_ROOT, file_sync.FILE_PATH, file_sync.FILE_SIZE FROM `file_sync` WHERE file_sync.OBJECT_ID='0_v5nyr0u2' AND file_sync.OBJECT_TYPE='4' AND file_sync.OBJECT_SUB_TYPE='1' AND file_sync.VERSION='1' AND (file_sync.STATUS='2' AND file_sync.STATUS NOT IN ('3','4')) ORDER BY file_sync.DC ASC
2015-04-21 17:17:26 [0.000419] [71.234.194.254] [1347402480] [PS2] [KalturaStatement->execute] DEBUG: Sql took - 0.00025010108947754 seconds
2015-04-21 17:17:26 [0.000241] [71.234.194.254] [1347402480] [PS2] [kFileSyncUtils::getReadyFileSyncForKey] NOTICE: kFileSyncUtils::getReadyFileSyncForKey - FileSync was found but doesn't exists locally
2015-04-21 17:17:26 [0.000169] [71.234.194.254] [1347402480] [PS2] [flvclipperAction->execute] NOTICE: Error - no FileSync for flavor [0_v5nyr0u2]
2015-04-21 17:17:26 [0.000212] [71.234.194.254] [1347402480] [PS2] [KExternalErrors::dieError] ERR: exiting on error 5 - required file was not found
Thanks in advance for any advice!
This is resolved.
We fixed this by using a newer version of the Kaltura CE and using the native HLS integration it has in it.
We also upgraded the player and the HLS plugin.
I am getting this strange situation in my project using windows 64 bit platform.
The project involves JNI calls.
The project requires an environment variable "My_Env" to be set.
This variable is internally used by java classes to load some Dlls(64-bit).
This "Junit" target is called from Ant build along with some other targets in a particular sequence. Initially, the environment variable "My_Env" was not set on 64 bit. When I was running the all ant targets in sequence JVM was crashing on Junit target. Even on running Junit target alone the JVM was crashing.
Then I realized that "My_ENv"wasnot set and set it. Now when I run the Junit target alone it is working fine without a crash. However on running the ant build default target(which involves all targets in a sequence) the JVM is still crashing on Junit target.
The Junit target is having "fork=on".
The error report is as following:
># A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000180001051, pid=4172, tid=4608
#
# JRE version: 6.0_27-b01
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.2-b01 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C [MyDLL.dll+0x1051]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
--------------- T H R E A D ---------------
Current thread (0x000000000051b800): JavaThread "main" [_thread_in_native, id=4608, stack(0x0000000000de0000,0x0000000000ee0000)]
siginfo: ExceptionCode=0xc0000005, reading address 0x0000000000000000
Registers:
RAX=0x0000000000000000, RBX=0x0000000000000000, RCX=0xfffffffffffffff5, RDX=0x0000000000000000
RSP=0x0000000000edea00, RBP=0x0000000000edeaa8, RSI=0x00000000faf53270, RDI=0x000000018000362a
R8 =0x7efefefefefefeff, R9 =0x7efefefefefeff6e, R10=0x0000000005497f21, R11=0x8101010101010100
R12=0x0000000000000000, R13=0x00000000fb4104d8, R14=0x0000000000edeac0, R15=0x000000000051b800
RIP=0x0000000180001051, EFLAGS=0x0000000000010246
Top of Stack: (sp=0x0000000000edea00)
0x0000000000edea00: 000000000051c5a0 0000000000ef11d2
0x0000000000edea10: 00000000fb4104d8 0000000000edeaa8
0x0000000000edea20: fffffffffffffffe 0000000000ee5b22
0x0000000000edea30: 0000000000ee5b22 0000000000ef1346
0x0000000000edea40: 00000000fb4104d8 000000000051c598
0x0000000000edea50: 000000000051c500 000000000051b800
0x0000000000edea60: 00000000fffffffe 0000000000edea68
0x0000000000edea70: 00000000fb4104d8 0000000000edeac0
0x0000000000edea80: 00000000fb410830 0000000000000000
0x0000000000edea90: 00000000fb4104d8 0000000000000000
0x0000000000edeaa0: 0000000000edeac8 0000000000edeb08
0x0000000000edeab0: 0000000000ee5b22 00000000fb4107c8
0x0000000000edeac0: 0000000000eee338 0000000000edeac8
0x0000000000edead0: 00000000fb40ff20 0000000000edeb10
0x0000000000edeae0: 00000000fb410270 0000000000000000
0x0000000000edeaf0: 00000000fb40ff28 0000000000edeac8
Instructions: (pc=0x0000000180001051)
0x0000000180001031: 00 0f b7 0d 67 11 00 00 66 89 0d e8 25 00 00 48
0x0000000180001041: 8d 3d d9 25 00 00 33 c0 48 83 c9 ff f2 ae 33 d2
0x0000000180001051: 0f b6 04 13 88 44 17 ff 48 ff c2 84 c0 75 f1 48
0x0000000180001061: 8d 15 b9 25 00 00 48 8d 0d 3a 11 00 00 ff 15 1c
Register to memory mapping:
RAX=0x0000000000000000 is an unknown value
RBX=0x0000000000000000 is an unknown value
RCX=0xfffffffffffffff5 is an unknown value
RDX=0x0000000000000000 is an unknown value
RSP=0x0000000000edea00 is pointing into the stack for thread: 0x000000000051b800
RBP=0x0000000000edeaa8 is pointing into the stack for thread: 0x000000000051b800
RSI=0x00000000faf53270 is an oop
{instance class}
- klass: {other class}
RDI=0x000000018000362a is an unknown value
R8 =0x7efefefefefefeff is an unknown value
R9 =0x7efefefefefeff6e is an unknown value
R10=0x0000000005497f21 is an unknown value
R11=0x8101010101010100 is an unknown value
R12=0x0000000000000000 is an unknown value
R13=0x00000000fb4104d8 is an oop
{method}
- klass: {other class}
R14=0x0000000000edeac0 is pointing into the stack for thread: 0x000000000051b800
R15=0x000000000051b800 is a thread
Stack: [0x0000000000de0000,0x0000000000ee0000], sp=0x0000000000edea00, free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [MyDLL.dll+0x1051] init+0x51
j src.com.MyDLL.init()V+0
j src.com.MyClass.initMyDLL()V+108
j src.com.MyClass.<init>(Lsrc/com/Configuration;)V+21
j src.com.MyUtil.testLoadFiles()V+113
v ~StubRoutines::call_stub
V [jvm.dll+0x1e9af7]
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j src.com.MyDLLJNI.init()V+0
j src.com.MyDLL.init()V+0
j src.com.MyClass.initMyDLL()V+108
j src.com.MyClass.<init>(Lsrc/com/Configuration;)V+21
j src.com.MyUtil.testLoadFiles()V+113
v ~StubRoutines::call_stub
j sun.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+0
j sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+87
j sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+6
j java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+161
j junit.framework.TestCase.runTest()V+107
j junit.framework.TestCase.runBare()V+7
j junit.framework.TestResult$1.protect()V+4
j junit.framework.TestResult.runProtected(Ljunit/framework/Test;Ljunit/framework/Protectable;)V+1
j junit.framework.TestResult.run(Ljunit/framework/TestCase;)V+18
j junit.framework.TestCase.run(Ljunit/framework/TestResult;)V+2
j junit.framework.TestSuite.runTest(Ljunit/framework/Test;Ljunit/framework/TestResult;)V+2
j junit.framework.TestSuite.run(Ljunit/framework/TestResult;)V+40
j junit.framework.TestSuite.runTest(Ljunit/framework/Test;Ljunit/framework/TestResult;)V+2
j junit.framework.TestSuite.run(Ljunit/framework/TestResult;)V+40
j org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run()V+431
j org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(Lorg/apache/tools/ant/taskdefs/optional/junit/JUnitTest;ZZZZZZ)I+39
j org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main([Ljava/lang/String;)V+741
v ~StubRoutines::call_stub
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x0000000000610000 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=4884, stack(0x0000000004b70000,0x0000000004c70000)]
0x000000000060e800 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=3216, stack(0x0000000004a70000,0x0000000004b70000)]
0x00000000005fd000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=4436, stack(0x0000000004970000,0x0000000004a70000)]
0x00000000005fb800 JavaThread "Attach Listener" daemon [_thread_blocked, id=4748, stack(0x0000000004870000,0x0000000004970000)]
0x00000000005f7800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=2688, stack(0x0000000004770000,0x0000000004870000)]
0x00000000005e8000 JavaThread "Finalizer" daemon [_thread_blocked, id=4824, stack(0x0000000004470000,0x0000000004570000)]
0x00000000005e0000 JavaThread "Reference Handler" daemon [_thread_blocked, id=4676, stack(0x0000000004370000,0x0000000004470000)]
=>0x000000000051b800 JavaThread "main" [_thread_in_native, id=4608, stack(0x0000000000de0000,0x0000000000ee0000)]
Other Threads:
0x00000000005da000 VMThread [stack: 0x0000000004270000,0x0000000004370000] [id=4480]
0x000000000061f000 WatcherThread [stack: 0x0000000004c70000,0x0000000004d70000] [id=4768]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap
def new generation total 19648K, used 2455K [0x00000000bae00000, 0x00000000bc350000, 0x00000000d0350000)
eden space 17472K, 14% used [0x00000000bae00000, 0x00000000bb065d08, 0x00000000bbf10000)
from space 2176K, 0% used [0x00000000bbf10000, 0x00000000bbf10000, 0x00000000bc130000)
to space 2176K, 0% used [0x00000000bc130000, 0x00000000bc130000, 0x00000000bc350000)
tenured generation total 43712K, used 0K [0x00000000d0350000, 0x00000000d2e00000, 0x00000000fae00000)
the space 43712K, 0% used [0x00000000d0350000, 0x00000000d0350000, 0x00000000d0350200, 0x00000000d2e00000)
compacting perm gen total 21248K, used 6210K [0x00000000fae00000, 0x00000000fc2c0000, 0x0000000100000000)
the space 21248K, 29% used [0x00000000fae00000, 0x00000000fb410878, 0x00000000fb410a00, 0x00000000fc2c0000)
No shared spaces configured.
Code Cache [0x0000000000ee0000, 0x0000000001150000, 0x0000000003ee0000)
total_blobs=233 nmethods=13 adapters=174 free_code_cache=49891776 largest_free_block=12160
Dynamic libraries:
0x0000000000400000 - 0x000000000042e000 C:\Java\jdk1.6.0_27\jre\bin\java.exe
0x0000000077b90000 - 0x0000000077d39000 C:\Windows\SYSTEM32\ntdll.dll
0x0000000077970000 - 0x0000000077a8f000 C:\Windows\system32\kernel32.dll
0x000007fefdb90000 - 0x000007fefdbfc000 C:\Windows\system32\KERNELBASE.dll
0x000007fefe260000 - 0x000007fefe33b000 C:\Windows\system32\ADVAPI32.dll
0x000007feffe00000 - 0x000007feffe9f000 C:\Windows\system32\msvcrt.dll
0x000007fefe540000 - 0x000007fefe55f000 C:\Windows\SYSTEM32\sechost.dll
0x000007fefe5d0000 - 0x000007fefe6fd000 C:\Windows\system32\RPCRT4.dll
0x000000006d890000 - 0x000000006e048000 C:\Java\jdk1.6.0_27\jre\bin\server\jvm.dll
0x0000000077a90000 - 0x0000000077b8a000 C:\Windows\system32\USER32.dll
0x000007fefe560000 - 0x000007fefe5c7000 C:\Windows\system32\GDI32.dll
0x000007feffdf0000 - 0x000007feffdfe000 C:\Windows\system32\LPK.dll
0x000007fefe190000 - 0x000007fefe259000 C:\Windows\system32\USP10.dll
0x000007fefa260000 - 0x000007fefa29b000 C:\Windows\system32\WINMM.dll
0x000007feff4e0000 - 0x000007feff50e000 C:\Windows\system32\IMM32.DLL
0x000007fefdec0000 - 0x000007fefdfc9000 C:\Windows\system32\MSCTF.dll
0x000000006d800000 - 0x000000006d80e000 C:\Java\jdk1.6.0_27\jre\bin\verify.dll
0x000000006d450000 - 0x000000006d477000 C:\Java\jdk1.6.0_27\jre\bin\java.dll
0x0000000077d60000 - 0x0000000077d67000 C:\Windows\system32\PSAPI.DLL
0x000000006d850000 - 0x000000006d862000 C:\Java\jdk1.6.0_27\jre\bin\zip.dll
0x000000006d6a0000 - 0x000000006d6b7000 C:\Java\jdk1.6.0_27\jre\bin\net.dll
0x000007feff490000 - 0x000007feff4dd000 C:\Windows\system32\WS2_32.dll
0x000007fefdeb0000 - 0x000007fefdeb8000 C:\Windows\system32\NSI.dll
0x000007fefd2d0000 - 0x000007fefd325000 C:\Windows\system32\mswsock.dll
0x000007fefd540000 - 0x000007fefd547000 C:\Windows\System32\wship6.dll
0x000007fefc670000 - 0x000007fefc685000 C:\Windows\system32\NLAapi.dll
0x000007fefa630000 - 0x000007fefa645000 C:\Windows\system32\napinsp.dll
0x000007fefd160000 - 0x000007fefd1bb000 C:\Windows\system32\DNSAPI.dll
0x000007fefa660000 - 0x000007fefa66b000 C:\Windows\System32\winrnr.dll
0x000007fefccf0000 - 0x000007fefccf7000 C:\Windows\System32\wshtcpip.dll
0x000007fefb650000 - 0x000007fefb677000 C:\Windows\system32\IPHLPAPI.DLL
0x000007fefb640000 - 0x000007fefb64b000 C:\Windows\system32\WINNSI.DLL
0x000007fefa880000 - 0x000007fefa888000 C:\Windows\system32\rasadhlp.dll
0x000007fefac40000 - 0x000007fefac93000 C:\Windows\System32\fwpuclnt.dll
0x0000000180000000 - 0x0000000180007000 C:project\output\bin\MyDLL.dll
0x0000000072bb0000 - 0x0000000072c82000 C:\Windows\system32\MSVCR100.dll
VM Arguments:
jvm_args: -Djava.library.path=C:\project\output\bin
java_command: org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner src.com.AllTests filtertrace=true haltOnError=false haltOnFailure=false showoutput=false outputtoformatters=true logtestlistenerevents=true formatter=org.apache.tools.ant.taskdefs.optional.junit.BriefJUnitResultFormatter formatter=org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter,C:\project\unittests\junit-results\TEST-AllTests.xml crashfile=C:\project\junitvmwatcher7335361868739082409.properties propsfile=C:\project\junit3775055844850964019.properties
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=C:\Java\jdk1.6.0_27
PATH=C:\project\output\bin
USERNAME=relmgr
OS=Windows_NT
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 26 Stepping 5, GenuineIntel
--------------- S Y S T E M ---------------
OS: Windows NT 6.1 Build 7601 Service Pack 1
CPU:total 1 (1 cores per cpu, 1 threads per core) family 6 model 26 stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt
Memory: 4k page, physical 4193848k(2877556k free), swap 8385848k(6803796k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (20.2-b01) for windows-amd64 JRE (1.6.0_27-ea-b01), built on May 18 2011 08:17:50 by "java_re" with MS VC++ 8.0 (VS2005)
time: Fri Jan 27 07:03:19 2012
elapsed time: 4 seconds
And here's the piece of ant , the Junit target if this helps:
<target name="Junit" depends="JunitCompile" >
<delete dir="${Junit.Results}"/>
<mkdir dir="${Junit.Results}"/>
<property name="fs" value="${file.separator}"/>
<property name="path" value="${basedir}${fs}${project.dir}${fs}bin"/>
<!-- junit errorProperty="test.failed" failureProperty="test.failed" fork="on" -->
<junit errorProperty="test.failed" fork="on">
<env key="Path" value="${path}"/>
<jvmarg value="-Djava.library.path=${basedir}${fs}${project.dir}${fs}bin"/>
<test name="${test.class}" todir="${Junit.Results}"/>
<formatter type="brief" usefile="false"/>
<formatter type="xml"/>
<classpath refid="classpath"/>
</junit>
</target>