Where do I see the HTML logs? - docker

If I run sitespeed within a docker and obtain the following output:
Google Chrome 63.0.3239.84
Mozilla Firefox 54.0.1
[2017-12-27 18:10:01] INFO: Versions OS: linux 4.9.49-moby nodejs: v8.9.1 sitespeed.io: 6.2.2 browsertime: 2.1.2 coach: 1.1.1
[2017-12-27 18:10:02] INFO: Starting chrome for analysing https://www.google.com/ 3 time(s)
[2017-12-27 18:10:02] INFO: Testing url https://www.google.com/ run 1
[2017-12-27 18:10:18] INFO: Testing url https://www.google.com/ run 2
[2017-12-27 18:10:29] INFO: Testing url https://www.google.com/ run 3
[2017-12-27 18:10:40] INFO: 18 requests, 584.40 kb, backEndTime: 158ms (±6.42ms), firstPaint: 321ms (±3.32ms), firstVisualChange: 389ms (±7.78ms), DOMContentLoaded: 376ms (±3.63ms), Load: 529ms (±91.22ms), speedIndex: 477 (±9.23), visualComplete85: 422ms (±7.90ms), lastVisualChange: 2.65s (±137.82ms), rumSpeedIndex: 321 (±3.32) (3 runs)
[2017-12-27 18:10:43] INFO: HTML stored in /sitespeed.io/reports
[2017-12-27 18:10:43] INFO: Finished analysing https://www.google.com/
Where are the HTML logs stored? '/sitespeed.io/reports', I'm not sure where to go to access this.

The example from the Docker hub page says:
docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io http://www.sitespeed.io/ -b chrome
the --rm part of the command means that the container is removed after it finishes. So you will not be able to get "inside" the container and access the results, but...
the -v "$(pwd)":/sitespeed.io part means that
-v: a volume is created
"$(pwd)": at your working directory
pointing to the /sitespeed.io folder of the container
Practically this means that if you run the above command at a directory named /my-docker-tests, (despite of the fact that the container is removed) you will be able to see the result files on your host's file system at /my-docker-tests/sitespeed-result/

Related

jib gradle plugin + static docker client: cannot build image due to permission error: layer.tar: A required privilege is not held by the client

I want to build Docker image with jib Gradle plugin in Windows, and use a Windows docker client to load it into my WSL 2 container running dockerd, and use WSL 2 as server. Resource-wise I think this is the lightest solution. .
On WSL 2 side, I run dockerd service in Ubuntu 20 on WSL 2, and it's listening on [::]:2375. TLS disabled(--tls=false), only http.
On Windows side, I only downloaded the Docker client(static client, from https://download.docker.com/win/static/stable/x86_64/), and added the dynamic WSL 2 container IP into the insecure-registry in daemon.json. This file is put in the same dir of docker.exe client.
On Intellij IDEA side, I use gradle 5.2.1 wrapper, and jib plugin 3.2.1. I configure jib as follows:
jib {
dockerClient.executable = 'E:\\coding\\environment\\docker\\docker.exe'
dockerClient.environment = [ DOCKER_HOST: '172.21.169.180:2375',
DOCKER_INSECURE_REGISTRIES: "172.21.169.180:5000"]
from.image = 'docker://mini/java#sha256:d3ded1fd0df592c33185d930d976304994bbc539c7bf70a6091cb3da0f7e11fa'
to.image = 'spring-plugins-demo'
container.mainClass = 'dev.westerngun.oldway.ApplicationV1'
}
I know it can connect to dockerd in my WSL 2, because before I add the dynamic IP of Ubuntu the error was not able to connect to daemon. Now it can load the base image and start building.
Then, when I run jibDockerBuild --stacktrace, I see this error:
Execution failed for task ':jibDockerBuild'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: C:\Users\WESTER~1\AppData\Local\Temp\16164656866093264693\33e3f3775358985441c3bea658f06f5307326c83f9c0bcbf8aa4acb327abffde\layer.tar: �ͻ���û���������Ȩ��
* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':jibDockerBuild'.
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$2.accept(ExecuteActionsTaskExecuter.java:121)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$2.accept(ExecuteActionsTaskExecuter.java:117)
at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:184)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:110)
at org.gradle.api.internal.tasks.execution.ResolveIncrementalChangesTaskExecuter.execute(ResolveIncrementalChangesTaskExecuter.java:84)
at org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:91)
at org.gradle.api.internal.tasks.execution.ResolveBeforeExecutionStateTaskExecuter.execute(ResolveBeforeExecutionStateTaskExecuter.java:74)
at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:58)
at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:109)
at org.gradle.api.internal.tasks.execution.ResolveBeforeExecutionOutputsTaskExecuter.execute(ResolveBeforeExecutionOutputsTaskExecuter.java:67)
at org.gradle.api.internal.tasks.execution.ResolveAfterPreviousExecutionStateTaskExecuter.execute(ResolveAfterPreviousExecutionStateTaskExecuter.java:46)
at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:93)
at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:45)
at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:94)
at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:63)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:49)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:46)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:416)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:406)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:102)
at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:43)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:355)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:343)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:336)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:322)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker$1.execute(DefaultPlanExecutor.java:134)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker$1.execute(DefaultPlanExecutor.java:129)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:202)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:193)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:129)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
Caused by: org.gradle.internal.UncheckedException: com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: C:\Users\WESTER~1\AppData\Local\Temp\16164656866093264693\33e3f3775358985441c3bea658f06f5307326c83f9c0bcbf8aa4acb327abffde\layer.tar: �ͻ���û���������Ȩ��
at org.gradle.internal.UncheckedException.throwAsUncheckedException(UncheckedException.java:67)
at org.gradle.internal.UncheckedException.throwAsUncheckedException(UncheckedException.java:41)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:106)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:48)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:41)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:28)
at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:705)
at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:672)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$4.run(ExecuteActionsTaskExecuter.java:338)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:402)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:394)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:92)
at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:327)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:312)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.access$200(ExecuteActionsTaskExecuter.java:75)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.execute(ExecuteActionsTaskExecuter.java:158)
at org.gradle.internal.execution.impl.steps.ExecuteStep.execute(ExecuteStep.java:46)
at org.gradle.internal.execution.impl.steps.CancelExecutionStep.execute(CancelExecutionStep.java:34)
at org.gradle.internal.execution.impl.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:69)
at org.gradle.internal.execution.impl.steps.TimeoutStep.execute(TimeoutStep.java:49)
at org.gradle.internal.execution.impl.steps.CatchExceptionStep.execute(CatchExceptionStep.java:34)
at org.gradle.internal.execution.impl.steps.CreateOutputsStep.execute(CreateOutputsStep.java:49)
at org.gradle.internal.execution.impl.steps.SnapshotOutputStep.execute(SnapshotOutputStep.java:42)
at org.gradle.internal.execution.impl.steps.SnapshotOutputStep.execute(SnapshotOutputStep.java:28)
at org.gradle.internal.execution.impl.steps.CacheStep.executeWithoutCache(CacheStep.java:133)
at org.gradle.internal.execution.impl.steps.CacheStep.lambda$execute$5(CacheStep.java:83)
at org.gradle.internal.execution.impl.steps.CacheStep.execute(CacheStep.java:82)
at org.gradle.internal.execution.impl.steps.CacheStep.execute(CacheStep.java:37)
at org.gradle.internal.execution.impl.steps.PrepareCachingStep.execute(PrepareCachingStep.java:33)
at org.gradle.internal.execution.impl.steps.StoreSnapshotsStep.execute(StoreSnapshotsStep.java:38)
at org.gradle.internal.execution.impl.steps.StoreSnapshotsStep.execute(StoreSnapshotsStep.java:23)
at org.gradle.internal.execution.impl.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:95)
at org.gradle.internal.execution.impl.steps.SkipUpToDateStep.lambda$execute$0(SkipUpToDateStep.java:88)
at org.gradle.internal.execution.impl.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:52)
at org.gradle.internal.execution.impl.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:36)
at org.gradle.internal.execution.impl.DefaultWorkExecutor.execute(DefaultWorkExecutor.java:34)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:109)
... 37 more
Caused by: com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: C:\Users\WESTER~1\AppData\Local\Temp\16164656866093264693\33e3f3775358985441c3bea658f06f5307326c83f9c0bcbf8aa4acb327abffde\layer.tar: �ͻ���û���������Ȩ��
at com.google.cloud.tools.jib.plugins.common.JibBuildRunner.runBuild(JibBuildRunner.java:285)
at com.google.cloud.tools.jib.gradle.BuildDockerTask.buildDocker(BuildDockerTask.java:126)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:103)
... 75 more
Caused by: java.nio.file.FileSystemException: C:\Users\WESTER~1\AppData\Local\Temp\16164656866093264693\33e3f3775358985441c3bea658f06f5307326c83f9c0bcbf8aa4acb327abffde\layer.tar: �ͻ���û���������Ȩ��
at com.google.cloud.tools.jib.tar.TarExtractor.extract(TarExtractor.java:93)
at com.google.cloud.tools.jib.tar.TarExtractor.extract(TarExtractor.java:49)
at com.google.cloud.tools.jib.builder.steps.LocalBaseImageSteps.cacheDockerImageTar(LocalBaseImageSteps.java:217)
at com.google.cloud.tools.jib.builder.steps.LocalBaseImageSteps.lambda$retrieveDockerDaemonLayersStep$0(LocalBaseImageSteps.java:133)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
The error message in Chinese is
C:\Users\WESTER~1\AppData\Local\Temp\16164656866093264693\33e3f3775358985441c3bea658f06f5307326c83f9c0bcbf8aa4acb327abffde\layer.tar: 客户端没有所需的特权。
And I think it can be translated into "A required permission is not held by the client".
I suspect this is because my user is not added to docker-user group, as stated here. But, I uninstalled Docker toolbox and I don't see this group anymore, as it sets DOCKER_HOST and interferes with my setup. Secondly, I don't have Local Users and Group available, seems Windows 10 Home edition does not have it.
Should I try to install gpedit in my Home Edition, add the group and try? But without Docker toolbox, I doubt it would work. Docker documentation explains here that it creates the group and configure it to ensure separation of permissions between root/admin and non-root/non-admin users; I think only creating that group will not work. https://docs.docker.com/desktop/windows/permission-requirements/
But, when I use docker.exe to connect to WSL 2 and save a tar file to C:\Users\WESTER~1\AppData\Local\Temp, it works. The tar file is created and not corrupted. So I think it's not a permission error; anyone can access that dir.
Windows bundled bsd-tar.exe has nothing to do with it; renaming the tar.exe in System32 and build, the error is the same.
It is solved when I run cmd as admin and cd to project dir and do gradlew jibDockerBuild. Image built and loaded into WSL 2 daemon successfully. It is indeed file system permission error.
Although still very strange(as I allowed the permission to everyone on that folder), but at least this is one workaround.
Another workaround, even better:
As per https://github.com/microsoft/WSL/issues/4983, I changed the jib config to set docker host to be http://[::1]:2375, and suddenly it works. Seems only ipv6 is bind.
Now not only the host is reachable, even permission error disappears; no insecure_registries settings needed, neither.

tdb2.tdbcompact command line tool returns Failed to get a lock: file

I'm running apache-jena-fuseki-3.13-1 and just found tdb2.tdbcompact from its bin-directory. I should run tdb2.tdbcompact nightly to prevent my jena-fuseki from running out of disk space, but now I get error message( Failed to get a lock: file) when running it:
miettinj#ramen:~/jena> ./apache-jena-3.13.1/bin/tdb2.tdbcompact --loc=./apache-jena-fuseki- 3.13.1/run/databases/test_TDB2
org.apache.jena.dboe.DBOpEnvException: Failed to get a lock: file='/srv/work/miettinj/jena/apache-jena-fuseki-3.13.1/run/databases/test_TDB2/tdb.lock': held by process 6136
ps -x|grep 6136
6136 ? Sl 30:48 /usr/lib64/jvm/java/bin/java -Xmx1200M -cp /srv/work/miettinj/jena/apache-jena-fuseki-3.13.1/fuseki-server.jar
"held by process 6136"
Another process is using the database. Compaction has to happen from the process using the database.
Apache Jena Fuseki Jena 3.17.0 added a function endpoint so that the administrator can ask for compaction on a running Fuseki server.

Custom Runtime Won't Use Dockerfile

I have an App Engine service I deploy a custom runtime in a flexible environment. Deployments functioned normally on 11/20. On 11/21 gcloud app deploy stopped using the Dockerfile and began treating it as a non-custom runtime. Neither the app.yaml nor the Dockerfile have changed.
Below is a sample log from 11/20 and 11/21 respectively. You will note Using Dockerfile found in... of the first log is not present in the second log.
First log, 11/20:
2020-11-20 11:12:02,202 DEBUG root Loaded Command Group: ['gcloud', 'app']
2020-11-20 11:12:02,547 DEBUG root Loaded Command Group: ['gcloud', 'app', 'deploy']
2020-11-20 11:12:02,551 DEBUG root Running [gcloud.app.deploy] with arguments: [--project: "distributed-computing-qa", --version: "9-2-0rc9"]
2020-11-20 11:12:02,621 INFO oauth2client.client Refreshing access_token
2020-11-20 11:12:03,043 DEBUG root Loading runtimes experiment config from [gs://runtime-builders/experiments.yaml]
2020-11-20 11:12:03,076 INFO root Reading [<googlecloudsdk.api_lib.storage.storage_util.ObjectReference object at 0x0000021920ECA548>]
2020-11-20 11:12:03,526 DEBUG root API endpoint: [https://appengine.googleapis.com/], API version: [v1]
2020-11-20 11:12:04,419 INFO ___FILE_ONLY___ Services to deploy:
2020-11-20 11:12:04,420 INFO ___FILE_ONLY___ descriptor: [C:\Users\Benjamin Filkins\Documents\Projects\Deployment\QA\dci\app.yaml]
source: [C:\Users\Benjamin Filkins\Documents\Projects\Deployment\QA\dci]
target project: [distributed-computing-qa]
target service: [default]
target version: [9-2-0rc9]
target url: [https://distributed-computing-qa.uc.r.appspot.com]
2020-11-20 11:12:05,272 DEBUG root No bucket specified, retrieving default bucket.
2020-11-20 11:12:05,274 DEBUG root Using bucket [gs://staging.distributed-computing-qa.appspot.com].
2020-11-20 11:12:05,941 DEBUG root Service [appengineflex.googleapis.com] is already enabled for project [distributed-computing-qa]
2020-11-20 11:12:06,109 INFO ___FILE_ONLY___ Beginning deployment of service [default]...
2020-11-20 11:12:06,123 INFO root Ignoring directory [node_modules]: Directory matches ignore regex.
2020-11-20 11:12:09,085 INFO root Ignoring directory [server\node_modules]: Directory matches ignore regex.
2020-11-20 11:12:09,679 INFO root Using Dockerfile found in C:\Users\Benjamin Filkins\Documents\Projects\Deployment\QA\dci
2020-11-20 11:12:09,679 INFO ___FILE_ONLY___ Building and pushing image for service [default]
2020-11-20 11:12:10,305 DEBUG root Could not call git with args ('config', '--get-regexp', 'remote\\.(.*)\\.url'): Command '['git', 'config', '--get-regexp', 'remote\\.(.*)\\.url']' returned non-zero exit status 1.
2020-11-20 11:12:10,305 INFO root Could not generate [source-context.json]: Could not list remote URLs from source directory: C:\Users\Benjamin Filkins\Documents\Projects\Deployment\QA\dci
2020-11-20 11:12:37,592 INFO root Uploading [C:\Users\BENJAM~1\AppData\Local\Temp\tmpwbdhi28f\src.tgz] to [staging.distributed-computing-qa.appspot.com/us.gcr.io/distributed-computing-qa/appengine/default.9-2-0rc9:latest]
2020-11-20 11:13:03,413 DEBUG root Using builder image: [gcr.io/cloud-builders/docker]
Second log, 11/21:
2020-11-21 05:10:39,041 DEBUG root Loaded Command Group: ['gcloud', 'app']
2020-11-21 05:10:39,177 DEBUG root Loaded Command Group: ['gcloud', 'app', 'deploy']
2020-11-21 05:10:39,181 DEBUG root Running [gcloud.app.deploy] with arguments: [--project: "distributed-computing-qa", --version: "9-2-0rc10"]
2020-11-21 05:10:39,203 DEBUG root Loading runtimes experiment config from [gs://runtime-builders/experiments.yaml]
2020-11-21 05:10:39,231 INFO root Reading [<googlecloudsdk.api_lib.storage.storage_util.ObjectReference object at 0x000001E60B3ED208>]
2020-11-21 05:10:39,522 DEBUG root API endpoint: [https://appengine.googleapis.com/], API version: [v1]
2020-11-21 05:10:40,196 INFO ___FILE_ONLY___ Services to deploy:
2020-11-21 05:10:40,198 INFO ___FILE_ONLY___ descriptor: [C:\Users\Benjamin
Filkins\Documents\Projects\Deployment\QA\dci\app.yaml]
source: [C:\Users\Benjamin Filkins\Documents\Projects\Deployment\QA\dci]
target project: [distributed-computing-qa]
target service: [default]
target version: [9-2-0rc10]
target url: [https://distributed-computing-qa.uc.r.appspot.com]
2020-11-21 05:10:44,749 DEBUG root No bucket specified, retrieving default bucket.
2020-11-21 05:10:44,758 DEBUG root Using bucket [gs://staging.distributed-computing-qa.appspot.com].
2020-11-21 05:10:45,460 DEBUG root Service [appengineflex.googleapis.com] is already enabled for project [distributed-computing-qa]
2020-11-21 05:10:45,645 INFO ___FILE_ONLY___ Beginning deployment of service [default]...
2020-11-21 05:10:45,658 INFO root Ignoring directory [node_modules]: Directory matches ignore regex.
2020-11-21 05:10:48,255 INFO root Ignoring directory [server\node_modules]: Directory matches ignore regex.
2020-11-21 05:10:57,261 DEBUG root Could not call git with args ('config', '--get-regexp', 'remote\\.(.*)\\.url'): Command '['git', 'config', '--get-regexp', 'remote\\.(.*)\\.url']' returned non-zero exit status 1.
2020-11-21 05:10:57,261 INFO root Could not find any remote repositories associated with [C:\Users\Benjamin Filkins\Documents\Projects\Deployment\QA\dci]. Cloud diagnostic tools may not be able to display the correct source code for this deployment.
2020-11-21 05:11:19,099 DEBUG root Skipping upload of [.env]
2020-11-21 05:11:19,099 INFO root Incremental upload skipped 100.0% of data
There are four separate projects this is now occurring on. A co-worker can also confirm the same behavior. What I have tried and can confirm:
Updated Google Cloud SDK to latest version (319.0.0)
Confirmed Cloud Build API is active
Confirmed the Cloud Build service account has the App Engine Admin, Cloud Build Service Account and Service Account User roles
App.yaml and Dockerfile present in root and unchanged between attempts
App.yaml contains runtime: custom and env: flex
What I cannot confirm with certainty or prove did not have an impact:
Changes in OS (Windows 10), though no update had occurred during this time period
Changes in my GCP service account roles/permissions, though given the spread across four distinct projects and impacting multiple users seems incredibly unlikely
Any additional insight into this issue or additional items I may have missed would be greatly appreciated.
I have solved the issue by downgrading to SDK version 271.0.0. My machine has both Python 2.7 and 3 and I noted 274 and above began support for using Python 3.
Upgrading to 274 or above results in the reported issue. 273 and below (I only went as far as 267) does not have the reported issues. While I am currently unable to provide concrete evidence, my suspicion would be down to the SDK's ability to determine which version of Python to prefer. As noted here support of Python 2 was deprecated on 09/30/2020.

How can I build and run Druid locally

My environment are below.
MacBook Pro (13-inch, 2019, Four Thunderbolt 3 ports)
2.8 GHz Quad CoreIntel Core i7
16 GB 2133 MHz LPDDR3
Intel Iris Plus Graphics 655 1536 MB
Docker: 19.03.12
Druid: 0.19.0
Although I followed official instructions, I failed to build or run Druid locally.
About this: https://github.com/apache/druid/tree/master/distribution/docker
I typed the following commands.
git clone https://github.com/apache/druid.git
docker build -t apache/druid:tag -f distribution/docker/Dockerfile .
However, the program never proceed.
Sending build context to Docker daemon 78.19MB
Step 1/18 : FROM maven:3-jdk-8-slim as builder
---> addee4586ff4
Step 2/18 : RUN export DEBIAN_FRONTEND=noninteractive && apt-get -qq update && apt-get -qq -y install --no-install-recommends python3 python3-yaml
---> Using cache
---> cdb74d0f6b3d
Step 3/18 : COPY . /src
---> 60d35cb6c0ce
Step 4/18 : WORKDIR /src
---> Running in 73dfa666a186
Removing intermediate container 73dfa666a186
---> 4839bf923b21
Step 5/18 : RUN mvn -B -ff -q dependency:go-offline install -Pdist,bundle-contrib-exts -Pskip-static-checks,skip-tests -Dmaven.javadoc.skip=true
---> Running in 1c9d4aa3d4e8
PLUS
Moreover, I followed this instruction and run docker-compose -f distribution/docker/docker-compose.yml up but I failed and get the error below.
coordinator | 2020-08-06T08:41:24,295 WARN [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorRuleRunner - Uh... I have no servers. Not assigning anything...
PLUS END
About this: https://hub.docker.com/r/apache/druid/tags
I typed the following commands.
docker pull apache/druid:0.19.0
docker run apache/druid:0.19.0
This program seems to work like this.
2020-08-06T07:50:22+0000 startup service
Setting 172.17.0.2= in /runtime.properties
cat: can't open '/jvm.config': No such file or directory
2020-08-06T07:50:24,024 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.2.5.Final
2020-08-06T07:50:24,988 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-hdfs-storage], jars: jackson-annotations-2.10.2.jar, hadoop-mapreduce-client-common-2.8.5.jar, httpclient-4.5.10.jar, htrace-core4-4.0.1-incubating.jar, apacheds-kerberos-codec-2.0.0-M15.jar, jackson-mapper-asl-1.9.13.jar, commons-digester-1.8.jar, jetty-sslengine-6.1.26.jar, jackson-databind-2.10.2.jar, api-asn1-api-1.0.0-M20.jar, ion-java-1.0.2.jar, hadoop-mapreduce-client-shuffle-2.8.5.jar, asm-7.1.jar, jsp-api-2.1.jar, druid-hdfs-storage-0.19.0.jar, api-util-1.0.3.jar, json-smart-2.3.jar, jackson-core-2.10.2.jar, hadoop-client-2.8.5.jar, httpcore-4.4.11.jar, commons-collections-3.2.2.jar, hadoop-hdfs-client-2.8.5.jar, hadoop-annotations-2.8.5.jar, hadoop-auth-2.8.5.jar, xmlenc-0.52.jar, aws-java-sdk-s3-1.11.199.jar, commons-net-3.6.jar, nimbus-jose-jwt-4.41.1.jar, hadoop-common-2.8.5.jar, jackson-dataformat-cbor-2.10.2.jar, hadoop-yarn-server-common-2.8.5.jar, accessors-smart-1.2.jar, gson-2.2.4.jar, commons-configuration-1.6.jar, joda-time-2.10.5.jar, hadoop-aws-2.8.5.jar, aws-java-sdk-core-1.11.199.jar, commons-codec-1.13.jar, hadoop-mapreduce-client-app-2.8.5.jar, hadoop-yarn-api-2.8.5.jar, aws-java-sdk-kms-1.11.199.jar, jackson-core-asl-1.9.13.jar, curator-recipes-4.3.0.jar, hadoop-mapreduce-client-jobclient-2.8.5.jar, jcip-annotations-1.0-1.jar, jmespath-java-1.11.199.jar, hadoop-mapreduce-client-core-2.8.5.jar, commons-logging-1.1.1.jar, leveldbjni-all-1.8.jar, curator-framework-4.3.0.jar, hadoop-yarn-client-2.8.5.jar, apacheds-i18n-2.0.0-M15.jar
2020-08-06T07:50:25,004 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-kafka-indexing-service], jars: lz4-java-1.7.1.jar, kafka-clients-2.5.0.jar, druid-kafka-indexing-service-0.19.0.jar, zstd-jni-1.3.3-1.jar, snappy-java-1.1.7.3.jar
2020-08-06T07:50:25,006 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-datasketches], jars: druid-datasketches-0.19.0.jar, commons-math3-3.6.1.jar
usage: druid <command> [<args>]
The most commonly used druid commands are:
help Display help information
index Run indexing for druid
internal Processes that Druid runs "internally", you should rarely use these directly
server Run one of the Druid server types.
tools Various tools for working with Druid
version Returns Druid version information
See 'druid help <command>' for more information on a specific command.
However, even if I add an argument like version, it does not work like this.
❯ docker run apache/druid:0.19.0 version
2020-08-06T07:51:30+0000 startup service version
Setting druid.host=172.17.0.2 in /runtime.properties
cat: can't open '/jvm.config': No such file or directory
2020-08-06T07:51:32,517 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.2.5.Final
2020-08-06T07:51:33,503 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-hdfs-storage], jars: jackson-annotations-2.10.2.jar, hadoop-mapreduce-client-common-2.8.5.jar, httpclient-4.5.10.jar, htrace-core4-4.0.1-incubating.jar, apacheds-kerberos-codec-2.0.0-M15.jar, jackson-mapper-asl-1.9.13.jar, commons-digester-1.8.jar, jetty-sslengine-6.1.26.jar, jackson-databind-2.10.2.jar, api-asn1-api-1.0.0-M20.jar, ion-java-1.0.2.jar, hadoop-mapreduce-client-shuffle-2.8.5.jar, asm-7.1.jar, jsp-api-2.1.jar, druid-hdfs-storage-0.19.0.jar, api-util-1.0.3.jar, json-smart-2.3.jar, jackson-core-2.10.2.jar, hadoop-client-2.8.5.jar, httpcore-4.4.11.jar, commons-collections-3.2.2.jar, hadoop-hdfs-client-2.8.5.jar, hadoop-annotations-2.8.5.jar, hadoop-auth-2.8.5.jar, xmlenc-0.52.jar, aws-java-sdk-s3-1.11.199.jar, commons-net-3.6.jar, nimbus-jose-jwt-4.41.1.jar, hadoop-common-2.8.5.jar, jackson-dataformat-cbor-2.10.2.jar, hadoop-yarn-server-common-2.8.5.jar, accessors-smart-1.2.jar, gson-2.2.4.jar, commons-configuration-1.6.jar, joda-time-2.10.5.jar, hadoop-aws-2.8.5.jar, aws-java-sdk-core-1.11.199.jar, commons-codec-1.13.jar, hadoop-mapreduce-client-app-2.8.5.jar, hadoop-yarn-api-2.8.5.jar, aws-java-sdk-kms-1.11.199.jar, jackson-core-asl-1.9.13.jar, curator-recipes-4.3.0.jar, hadoop-mapreduce-client-jobclient-2.8.5.jar, jcip-annotations-1.0-1.jar, jmespath-java-1.11.199.jar, hadoop-mapreduce-client-core-2.8.5.jar, commons-logging-1.1.1.jar, leveldbjni-all-1.8.jar, curator-framework-4.3.0.jar, hadoop-yarn-client-2.8.5.jar, apacheds-i18n-2.0.0-M15.jar
2020-08-06T07:51:33,524 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-kafka-indexing-service], jars: lz4-java-1.7.1.jar, kafka-clients-2.5.0.jar, druid-kafka-indexing-service-0.19.0.jar, zstd-jni-1.3.3-1.jar, snappy-java-1.1.7.3.jar
2020-08-06T07:51:33,526 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-datasketches], jars: druid-datasketches-0.19.0.jar, commons-math3-3.6.1.jar
ERROR!!!!
Found unexpected parameters: [version]
===
usage: druid <command> [<args>]
The most commonly used druid commands are:
help Display help information
index Run indexing for druid
internal Processes that Druid runs "internally", you should rarely use these directly
server Run one of the Druid server types.
tools Various tools for working with Druid
version Returns Druid version information
See 'druid help <command>' for more information on a specific command
So I see a few things here:
docker run apache/druid:0.19.0 means "fire and forget", if you don't have an endless running service here, your docker container will be shut down shortly after start.
To have an interaction within the docker container start it with "-it" command.
To let it run without interaction run it with "-d" command for detached.
YOu can find information about this here: https://docs.docker.com/engine/reference/run/
You have to check the start command.
The thing you wrote after the run command is the start command (in your case "version") - this is runned like you would type it into the running shell after words (just "version").
Additional to that, if you DONT add a startup command, there could be a startup command in the default druid dockerfile.
You can see the dockerfile of your selected image at docker.hub, like here:
https://hub.docker.com/layers/apache/druid/0.19.0/images/sha256-eb2a4852b4ad1d3ca86cbf4c9dc7ed9b73c767815f187eb238d2b80ca26dfd9a?context=explore
There you see, the start command, wihtin a dockerfile this is called ENTRYPOINT, is a shellscript:
ENTRYPOINT ["/druid.sh"]
So writing "version" after your run commands stops the shell command from running - we should not do that :)

Jenkins Job Builder failing authentication with Jenkins due to encoding issue?

I'm trying to run jenkins-jobs update for the first time on my system, but it fails on authentication.
Command:
jenkins-jobs --conf ./jjb.ini update jobs/
Where jobs contains a test.yml - a miniature build project just for testing. jjb.ini is:
[jenkins]
user=admin
password={{ admin_api_token }} # Inserted API token here.
url=http://127.0.0.1:8080
query_plugins_info=False
Expected result:
Success, and import of the example build project into Jenkins.
Actual result:
INFO:jenkins_jobs.cli.subcommand.update:Updating jobs in ['jobs/'] ([])
INFO:jenkins_jobs.builder:Number of jobs generated: 1
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/jenkins/__init__.py", line 557, in jenkins_request
self._request(req))
File "/usr/local/lib/python3.5/dist-packages/jenkins/__init__.py", line 508, in _response_handler
response.raise_for_status()
File "/usr/lib/python3/dist-packages/requests/models.py", line 840, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Invalid password/token for user: b'admin' for url: http://127.0.0.1:8080/crumbIssuer/api/json
What catches my eye here is that the authentication fails for b'admin, not for admin. This is also reflected in the "People" page on Jenkins web-interface, which before the attempted login only showed admin, but after the attempted login shows:
From what I've been able to figure out, there may be a problem with encoding in the login request from JJB, but I'm looking for help when it comes to how to go about trying to fix this.
Current setup:
Ubuntu 16.04.4 LTS
Jenkins 2.125 (working as expected, at :8080)
jenkins-job-builder 2.0.9
Python 3.5.2
pip 10.0.1 from /usr/local/lib/python3.5/dist-packages/pip-10.0.1-py3.5.egg/pip (python 3.5)
java -version: openjdk version "1.8.0_171"
It works with requests==2.19.1
sudo pip uninstall requests
sudo pip install requests
$ pip freeze | grep requests
requests==2.19.1
The following steps resolved the issue in my case:
pip3.7 freeze | grep requests
pip3.7 uninstall requests
pip3.7 install requests
pip3.7 freeze | grep requests

Resources