I'm trying to build a service (Google Cloud Composer, which is basically just Airflow) locally via Docker. However, I'm getting the following error message:
│ http+docker://localhost/v1.41/containers/create?name=composer-local-dev-gcp_composer_locally: Bad Request ("invalid mount │
│ config for type "bind": bind source path does not exist: │
│ /opt/homebrew/lib/python3.10/site-packages/composer_local_dev/docker_files/entrypoint.sh")
However, such a file does exist and is executable:
16777233 2895095 -rwxr-xr-x 1 <my_unix_name> admin 0 1281 "Feb 13 14:26:24 2023" "Feb 13 14:26:24 2023" "Feb 13 14:26:24 2023" "Feb 10 17:04:20 2023" 4096 8 0 /opt/homebrew/lib/python3.10/site-packages/composer_local_dev/docker_files/entrypoint.sh
I've confirmed my user is a member of the "admin" group, so I don't think this is a file permissions error.
How can I get Docker to recognize this entrypoint and run the relevant script?
Google Cloud Skills Boost
(Quest) Secure Workloads in Google Kubernetes Engine
(Lab) Securing Applications on Kubernetes Engine - Three Examples
On the section "Provisioning the Kubernetes Engine cluster", run the command
make create
I got into an error:
enter code here
│ Error: Error resolving image name 'debian-cloud/debian-9': Could not find image or family debian-cloud/debian-9
│
│ with module.bastion.google_compute_instance.instance,
│ on modules/instance/main.tf line 54, in resource "google_compute_instance" "instance":
│ 54: resource "google_compute_instance" "instance" {
Step 1: Run the command to see the available images
gcloud compute images list | grep debian
Step 2: At the project root, run the below command
vi ./terraform/modules/instance/main.tf
Step 3: At VIM normal mode, search for the debian-9 by type /debian-9, hit enter
// Specify the Operating System Family and version.
boot_disk {
initialize_params {
image = "debian-cloud/debian-9"
}
}
Step 4: As my case, the debian-10 is available. So I change the above block code into
// Specify the Operating System Family and version.
boot_disk {
initialize_params {
image = "debian-cloud/debian-10"
}
}
Step 5: Change directory to the project root, then re-run the command make create.
Note: this is not (directly) a question about how to print PDF in chrome, instead this is a question about how to get more information when printing fails.
In short: I cannot solve a printing PDF problem, which occurs only for certain (presumably large) pages and could use some assistance in debugging the actual issue.
Background: I am using the chromedriver (v83) and chromium-browser (v83) to print PDF files from webpages by utilizing python selenium. I am building a docker image to contain the required dependencies for this. I have tried to use Debian (buster and stretch) as well as Alpine base images, but all eventually result in the same error, when trying to print some pages. The odd thing is that for other (smaller) pages the printing works, but when many assets and pages are to be printed, the printing fails. I might add that this docker images is eventually being run inside of a Kubernetes cluster, where I assigned up to 4GB of RAM.
What code am I running?
This project was written for python3, so here are some relevant code fragments. Please note that I removed all error handling and waiting for the page loads to complete here.
from selenium import webdriver
appState = {
"recentDestinations": [
{
"id": "Save as PDF",
"origin": "local"
}
],
"selectedDestinationId": "Save as PDF",
"version": 2
}
def get_chrome_options(headless: bool, enable_logging: bool) -> Options:
chrome_options = webdriver.ChromeOptions()
profile = {'printing.print_preview_sticky_settings.appState': json.dumps(appState)}
chrome_options.add_experimental_option('prefs', profile)
if headless:
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--window-size=1920,1080')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--disable-web-security')
chrome_options.add_argument('-–allow-file-access-from-files')
chrome_options.add_argument('--run-all-compositor-stages-before-draw')
chrome_options.add_argument('--kiosk-printing')
if enable_logging:
chrome_options.add_argument('--enable-logging')
return chrome_options
def print_the_page(url):
driver = webdriver.Chrome(chrome_options=get_chrome_options(headless, enable_logging))
driver.execute(driver_command=Command.GET, params={'url': url})
command_url = f"{driver.command_executor._url}/session/{driver.session_id}/chromium/send_command_and_get_result"
response = driver.command_executor._request('POST', command_url, json.dumps({'cmd': 'Page.printToPDF', 'params': {}}))
Then what happens?
For some pages this fails - meaning - there is this message in the response:
{'status': 500, 'value': '{"value":{"error":"unknown error","message":"unknown error: unhandled inspector error: {\\"code\\":-32000,\\"message\\":\\"Printing failed\\"}\\n (Session info: headless chrome=83.0.4103.116)","stacktrace":""}}'}
[UPDATE]
I have managed to produce some more error output when using the --print-to-pdf option directly, which seems to hint at an "out-of-memory" issue here:
[0923/135406.102857:WARNING:discardable_shared_memory_manager.cc(194)] Less than 64MB of free space in temporary directory for shared memory files: 23
[0923/135406.110108:WARNING:dns_config_service_posix.cc(341)] Failed to read DnsConfig.
[0923/135406.180892:WARNING:dns_config_service_posix.cc(341)] Failed to read DnsConfig.
[0923/135406.613221:FATAL:memory.cc(38)] Out of memory. size=796176
Received signal 6
r8: 00007fa6f39dadc4 r9: 0000000000000000 r10: 0000000000000008 r11: 0000000000000246
r12: 0000557efd1b0660 r13: 0000000000000000 r14: 00007fa6f39db240 r15: 0000000000000043
di: 0000000000000002 si: 00007fa6f39dac90 bp: 00007fa6f39dac90 bx: 0000000000000000
dx: 0000000000000000 ax: 0000000000000000 cx: 00007fa6fd347a71 sp: 00007fa6f39dac88
ip: 00007fa6fd347a71 efl: 0000000000000246 cgf: 002b000000000033 erf: 0000000000000000
trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000
[end of stack trace]
Calling _exit(1). Core file will not be generated.
[0923/135406.626313:ERROR:headless_shell.cc(399)] Abnormal renderer termination.
I will note here that I have been running this docker container locally on my machine (which has more than enough RAM) as well as on a Kubernetes cluster where this image has requested 4GB RAM. I also monitored the RAM usage and it didn't seem to be an issue - although - that could be illusive if the RAM usage is so radically high that chrome just fails and you never really see that in the overall RAM usage.
[UPDATE 2]
I have tried to use the --print-to-pdf option again, but I am seeing issues with that as well. The resources are loading, but the printing still fails.
│ [0923/144355.169080:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
...
│ [0923/141758.393923:WARNING:dns_config_service_posix.cc(341)] Failed to read DnsConfig. │
│ [0923/141758.401925:ERROR:zygote_host_impl_linux.cc(262)] Failed to adjust OOM score of renderer with pid 32: Permission denied (13) │
│ [0923/141758.413475:ERROR:zygote_host_impl_linux.cc(262)] Failed to adjust OOM score of renderer with pid 36: Permission denied (13)
... loading all the resources ...
│ [0923/141824.611661:ERROR:print_render_frame_helper.cc(1889)] Printing failed. │
│ [0923/141824.612439:ERROR:headless_shell.cc(562)] Print to PDF failed
What's the question(s)?
How can I get more information about why the "Printing failed" - unfortunately the "unknown error: unhandled inspector error" hasn't given me any ideas about how to proceed.
Are there potentially any additional flags to get more debug output from chrome or is there a log somewhere that I should be able to find?
What else have I tried?
I have initially been running this under Debian buster with the latest google-chrome and chromium binaries (v85). I have switched to an Alpine base image and chromium - hoping that this might change something, but it didn't.
I have experimented with setting up Xvfb ${DISPLAY} -screen ${SCREEN} ${RESOLUTION} & in Docker, but it didn't seem to have any effect either.
I have tried to switch to using the direct cli google-chrome --print-to-pdf= option, but since it's a page that requires to pass through login authentication, I could only get the login page to print and it also seems to have some not so nice formatting issues.
I have been running this on my machine, outside of Docker, and was able to print as expected, but as soon as I put the same code inside a Docker container, it fails.
Unfortunately, I cannot share the page where this fails with you.
The relevant warning from your logs seems to be this:
[0923/135406.102857:WARNING:discardable_shared_memory_manager.cc(194)] Less than 64MB of free space in temporary directory for shared memory files: 23
The problem appears to stem from Docker's mounted /dev/shm being too small for Chromium to do things like you're trying to do.
I found a closed bug report against chromium referencing this issue in certain limited environments such as AWS Lambda and Docker, it was fixed in chromium v65 behind a command line flag --disable-dev-shm-usage.
The last few comments reference another bug report (now closed) about this issue in chromium v83 where the command line flag was not working properly. It has been fixed in version 84 - per comment 28:
You can find the fix in current stable release of Chrome (version 84.0.4147.89 and above).
You've indicated you're using chromium v83, so you'll need to update at least version 84.0.4147.89, then use the command line flag --disable-dev-shm-usage.
I am unable to update an already created google-cloud-composer environment. This happens if I am working with an already created environment but not when I am creating a new one. Seems like I am missing some default settings here. Has anyone else faced a similar issue?
gcloud composer environments list --locations us-east1
┌───────────────┬──────────┬─────────┬──────────────────────────┐
│ NAME │ LOCATION │ STATE │ CREATE_TIME │
├───────────────┼──────────┼─────────┼──────────────────────────┤
│ dummy-airflow │ us-east1 │ RUNNING │ 2018-11-21T09:50:19.793Z │
└───────────────┴──────────┴─────────┴──────────────────────────┘
cloud composer environments update dummy-airflow
--location us-east1 --update-env-variables gcp_project=data-rubrics
Waiting for [projects/data-rubrics/locations/us-east1/environments/dummy-airflow] to be updated with [projects/data-rubrics/locations/us-
east1/operations/b6746709-1529-4d67-a08c-453de1a0063a]...failed.
ERROR: (gcloud.composer.environments.update) Error updating [projects/data-rubrics/locations/us-east1/environments/dummy-airflow]: Operation [projects/data-rubrics/locations/us-east1/operations/b6746709-1529-4d67-a08c-453de1a0063a] failed: Composer Backend timed out. Currently running tasks are [stage: CP_COMPOSER_AGENT_RUNNING
description: "Composer Agent Running. Latest Agent Stage: stage: PATCH_CREATED\n ."
response_timestamp {
seconds: 1543236373
nanos: 570000000
}
].
The issue was resolved by performing the two steps below:
Disable and then Enable the Cloud Composer API; and
Set the AIRFLOW_GPL_UNIDECODE to yes.
I am seeing this error for quite sometime now.
I am running ant build on CYGWIN which inturn runs on WindowsXP.
The resolution(bad one) I found was to delete my gcct/first directory and run ant build again (which runs from another directory). It runs successfully but if I modify some code under gcct/first, I do not want to delete it because of this error.
I did see this link. The resolution here does not apply to me since I do not have .cvspass defined anywhere in the build.xml.
C:\svn\CEL_v3681\buildCore.xml:1883: cvs exited with error code 1
Command line was [Executing 'cvs' with arguments:
'checkout'
'-A'
'-rfirst_v2_126'
'gcct/first'
The ' characters around the executable and arguments are
not part of the command.
environment:
ALLUSERSPROFILE=C:\Documents and Settings\All Users
ANT_HOME=C:/Apps/Apache/apache-ant-1.7.0
APPDATA=C:\Documents and Settings\shankarc\Application Data
CLASSPATH=./;C:/Program Files/Java/jre1.5.0_07/lib/ext/QTJava.zip
COMMONPROGRAMFILES=C:\Program Files\Common Files
COMPUTERNAME=NYKPWM2035798
COMSPEC=C:\WINNT\system32\cmd.exe
CUSTPROF=Roaming700Live
CVSROOT=:pserver:shankarc#amcvs2.lehman.com:/home/eqcvs/cmte
CVS_RSH=/bin/ssh
FP_NO_HOST_CHECK=NO
HOME=C:\Apps\CYGWIN\home\shankarc
HOMEDRIVE=F:
HOMEPATH=\
HOSTNAME=nykpwm2035798
IDEA_PROPERTIES=C:\Documents and Settings\shankarc\idea.properties
INFOPATH=/usr/local/info:/usr/share/info:/usr/info:
JAVA_HOME=C:/Program Files/Java/jdk1.6.0_21/
JDK_HOME=C:\Program Files\Java\jdk1.6.0_21\
LOGONSERVER=\\NYKPSM00069
MANPATH=/usr/local/man:/usr/share/man:/usr/man::/usr/ssl/man
NUMBER_OF_PROCESSORS=2
OS=Windows_NT
PATH=C:\Apps\CYGWIN\usr\local\bin;C:\Apps\CYGWIN\bin;C:\Apps\CYGWIN\bin;C:\Apps\CYGWIN\usr\X11R6\bin;C:\Apps\Apache\apache-ant-1.7.0\bin;C:\Program Files\Java\jdk1.6.0_21\bin\;C:\Apps\CYGWIN\bin;C:\Program Files\VisualSVN Server\bin;C:\Program Files\Sudowin\Clients\Console;C:\Program Files\Fortify Software\Fortify 360 v2.5.0\bin
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.PSC1
PRINTER=\\NYKPSM04020\NYKLPR1301-03-03C05
PROCESSOR_ARCHITECTURE=x86
PROCESSOR_IDENTIFIER=x86 Family 6 Model 15 Stepping 6, GenuineIntel
PROCESSOR_LEVEL=6
PROCESSOR_REVISION=0f06
PROFGROUP=FONP
PROGRAMFILES=C:\Program Files
PROMPT=$P$G
PWD=/cygdrive/c/svn/CEL_v3681/gcct/cel
QHOME=c:\q
QTJAVA=C:\Program Files\Java\jre1.5.0_07\lib\ext\QTJava.zip
SESSIONNAME=Console
SHLVL=1
SITECODE=NYK
SITEIDENT=NYK
SVN_ASP_DOT_NET_HACK=1
SYSTEMDRIVE=C:
SYSTEMROOT=C:\WINNT
TEMP=C:\TEMP
TERM=cygwin
TMP=C:\TEMP
UATDATA=C:\WINNT\system32\CCM\UATData\D9F8C395-CAB8-491d-B8AC-179A1FE1BE77
USER=shankarc
USERDNSDOMAIN=INTRANET.BARCAPINT.COM
USERDOMAIN=INTRANET
USERNAME=shankarc
USERPROFILE=C:\Documents and Settings\shankarc
WINDIR=C:\WINNT
CVS_PASSFILE=C:\Apps\CYGWIN\home\shankarc\.cvspass]
Total time: 58 seconds
How I resolve this?
I had the same issue and found that even though I was not using .cvspass I did have a build property of cvs.pass set which needed to be reset to OVERRIDE to function depending on how you set up your cvs access (though it looked similar from your post). This needed to be changed in build.properties and .build.properties. Hope this helps!