I am running some services with Google Cloud Run. While performance has been satisfactory, there's a recurrent issue with extremely slow startup time, which leads to occasionally dropped requests when new containers can't spin up in time.
Currently, with first gen execution environments and startup CPU boost enabled, Google's dashboard reports around 18 to 50 seconds of startup time. Image is based on ruby:3.0.2, and it runs a Ruby on Rails 6 application. In a development environment, startup (timed from run to container accepting requests) never seems to take more than 5 seconds.
I want to know what tools are available to diagnose this issue, and if there are any obvious pitfalls with my specific case that I might be missing.
I've tried playing around with the service's configuration options, to no avail. The biggest suspect is a startup bash script that handles migrations on the first boot, and asset compiling on development. However, I've tried building with an empty script, and the problem persists. I also think the container images might be too large (around 700Mb), but I haven't gotten around to slimming then down nor found evidence that this is the problem.
Hello and thank you for looking at this. I have been using docker to create various multi-arch builds. I have noticed some interesting behavior. When trying to run buildx for image creation, scikit_learn 0.21.3 takes a very long time to download/install (about 2.5 hours). However, when using the regular build command through docker on a single arch, it only takes about 10 minutes or so. The reason I am having to use this specific version of scikit_learn is due to an error I receive from my application where it is unable to find the sklearn.utils.linear_assignment module.
I receive this \nModuleNotFoundError: No module named 'sklearn.utils.linear_assignment_'
The only version I have been able to run is 0.21.3 up to this point. Having said that, I have found more recent versions do install much faster, but again they do not have the linear assigmment module, which is a dependency for my application. Any help or guidance would be greatly appreciated.
I am trying to debug a huge project using VS Code, the project is supposed to run on dapr using docker. I have installed dapr, docker, both seem to work fine. I also have VS Code, and I managed to get dapr extension for VS Code.
I build the project using dotnet build, no errors, so I am assuming that works fine, but when I try to debug it (run it) I get that error. I have read that it can be caused by having to set up environmental variables, but I don't know which environmental variables, as they seem to be present in the path (c:\users\myself.dapr\bin) is present.
So I have a few questions: what should I do now? and what is the reason behind it? Basically, I want to fix this, but knowing and understanding how, thanks.
As simple as restarting the laptop fix it, I guess that I did all of the installations without restarting and it needed a restart.
It was nothing to do with the environment variables in the end.
This is my first foyer into .Net Core and App Engines, so please forgive me if I sound uninformed.
We have a .Net Core Application that we're trying to get published to a GCP App engine (obviously). when I run dotnet publish -c Release it builds just fine without any errors. When I test the program locally it runs just fine and I'm able to access it. However whenever I try to get it on GCP I get the following error:
Updating service [default] (this may take several minutes)...
.................................................................................................................................................failed.
ERROR: (gcloud.app.deploy) Error Response: [9]
Application startup error:
Error:
An assembly specified in the application dependencies manifest (ApplicationName.deps.json) was not found:
package: 'Microsoft.AspNetCore.Mvc.Abstractions', version: '2.0.2'
path: 'lib/netstandard2.0/Microsoft.AspNetCore.Mvc.Abstractions.dll'
This assembly was expected to be in the local runtime store as the application was published using the following target manifest files:
aspnetcore-store-2.0.5.xml
Failed to deploy project WebApiDotNetCore to App Engine Flex.
We tried removing it from the dependencies JSON, and that just ended up breaking everything, so it is indeed required. It is installed in the project via nuget, so it should be included with dotnet restore. I've looked around and some sources seem to think that it's the installation of the dotnet core sdk, but I've tried it on three computers and always get the same thing.
Lastly, I should say this happens when I try to deploy through command line as well as directly through Visual Studio with the GCP SDK.
Has anyone experienced this error, or something similar? Any advice or guidance is very much appreciated.
Thanks!
-BT
OP REVISION
As an update I was able to get this resolved aside from the fact that I get a 502 error when I try to load the application. Here are the steps I took for anyone else that is looking what to do:
Pre-reqs: Docker for Windows and Google Cloud SDK installed and running. Running turned out to be a pain with Docker for Windows. Many many restarts and reinstallations.
Open the solution and ensure that the startup project is set correctly.
Right click the startup Project, and select Add > Docker Support.
Select Linux in the popup window and allow the files to be created.
When complete, the Dockerfile should appear in the preview window. Do the following:
For me the first line read: FROM microsoft/aspnetcore:2.0 AS base. Change this to FROM microsoft/aspnetcore-build:2.0 AS base.
Additionally, check to make sure that the last line has the correct .dll name. Docker for Windows will put whatever the project name is rather than the class name, so for me my final .dll names were different than the project name.
Lastly, if your project has any dependencies that are required to run but not to build, then you'll need to manually add them. For me we have a couple of XML files that needed to be put in the app folder, so I had to add COPY *.xml /app/ and put those files in the same folder as the solution file is in.
If there's anything else you need to do to the Dockerfile I highly recommend this page. It's a how-to on all Dockerfile commands written in ENGLISH! (that was my biggest problem with all of this - I have little experience with Linux and even less with Docker and everything was written in Greek for me).
Create an app.yaml file. I just used the standard:
runtime: custom
env: flex
Copy the Dockerfile found in the startup project's folder into the folder with the solution.
Initialize gcloud to the right project, then navigate to the solution folder. The type gcloud app deploy app.yaml, and follow the onscreen guide.
for me it takes about 15 minutes to deploy the GCP, so depending on the complexity of your project it may take longer, though this one is rather complex.
Now I'm trying to figure out my 502 error... I've tried what seems like everything - changing the listening port in the application, exposing the listening port on the dockerfile, trying to get GCP to open that port, and trying half a dozen different ports. It's slow-going since it's such a chore to deploy each time.
Hope this helps anyone that was like me a couple weeks ago and had never even heard of Docker!
Which version of .NET Core is this? Also, have you tried to run in Cloud Shell? Maybe that will provide more clues on what might be wrong.
It looks that you don't have the Microsoft.AspNetCore.Mvc.Abstractions library installed in your system. Using the .NET CLI, type the following command:
dotnet add package Microsoft.AspNetCore.Mvc.Abstractions --version 2.0.2
After that, to ensure the library is included, run the following:
dotnet restore
dotnet build
Try running it locally (it should work), and then use the dotnet publish -c Release command again.
I updated some plugins and restarted the jenkins but now it says:
Please wait while Jenkins is restarting
Your browser will reload automatically when Jenkins is ready.
It is taking too much time (waiting from last 40 minutes). I have only 1 project with around 20 builds. I have restarted jenkins many times and worked fine but now it stucks.
Is there any way out to kill/suspend jenkins to avoid this wait?
I had a very similar issue when using jenkins build-in restart function. To fix it I killed the service (with crossed fingers), but somehow it kept serving the "Please wait" page. I guess it is served by a separate thread, but since i could not see any running java or jenkins processes i restarted the server to stop it.
After reboot jenkins worked but it was not updated. To make it work it I ran the update again and restarted the jenkins service manually - it took less than a minute and worked just fine...
Jenkins seems to have a number of bugs related to restarting, and at least one unresolved: jenkins issue
Windows ONLY....
All the solutions here didn't work and restarting the server was not an option. If you are in the same situation.
I had to kill java.exe and restart the jenkins service. After I did this Jenkins reloaded several times and then went back to normal.
I was stuck on the jenkins restarting page for 10-ish minutes untill I did this.
Hope this helps.
Running this in the command line helped me:
service jenkins restart
I had a similar issue when updating plugins from the pluging update page and I marked the restart jenkins options. jenkins only showed the waiting message for a long time.
I solved the issue restoring .bak to .jpi files of the the plugins that I tried to update.
I did the follow in my jenkins
cd $JENKINS_HOME/plugins/
>sudo mv git.bak git.jpi
.
. (more plugins files)
.
>sudo mv ldap.bak ldap.jpi
>sudo /sbin/service jenkins restart
Check Event Viewer.
I found that my Java died.
Faulting application java.exe, version 7.0.250.17, time stamp 0x51c4b3fd, faulting module ntdll.dll, version 6.0.6002.18541, time stamp 0x4ec3e39f, exception code 0xc0000374, fault offset 0x000abc4f, process id 0x1188, application start time 0x01cee4f42968bc81.
Finally I found that it's Jenkins 1.540 problem. Don't use it.
https://issues.jenkins-ci.org/browse/JENKINS-20630
I faced the same issue after upgrading some plugins on Windows. Looking on jenkins.err.log it displayed this error
Exception in thread "main" java.io.IOException: Jenkins has failed to create a temporary file in C:\Users\builder\AppData\Local\Temp\
at Main.extractFromJar(Main.java:350)
at Main._main(Main.java:194)
at Main.main(Main.java:91)
Caused by: java.io.IOException: There is not enough space on the disk
at java.io.WinNTFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(Unknown Source)
at Main.extractFromJar(Main.java:347)
... 2 more
The problem was that the TEMP folder of the jenkins user had lots of temporary files. After cleaning that folder jenkins restarted correctly.
just performed a restart on the server. That fixed the issue !
In Command prompt execute this
C:\>service jenkins restart
Or
You can go for Service currently running in your machine( Win + R ) seach for Jenkins and Click on restart
For me, the cause seemed to be having lots of old job build logs hanging around. To clean them up, I ran:
cd $JENKINS_HOME/jobs
find -name 'builds' | xargs -n 1 bash -c 'rm -rf $0/[1-9]*'
Then I stopped and started Jenkins again, and it came up within a minute.
Credit to: https://stackoverflow.com/a/39230597/2255242
This is an old thread.. but my personal recommendation is to WAIT before attempting to do anything (such as restarting service, etc).
I wasted hours once trying to fix something that turned out to be not an issue in the first place. In the end, I messed things up and wasted a lot of time.
Just because you see errors in the logs doesn't necessarily mean that you need to take action.
The upgrade took about 45 minutes in the end for me. All i did at one point was refreshing my browser window. It can take a while.
Just my opinion
On Win 10: Stopping with the service command from the command line reported failure to stop the service, but I was able to stop it from services.msc (running as administrator). The updates were applied. Sorry, no definitive answer from me. YMMV.
I used TCPView and killed process that was using port 8080. BAsically it was all Java.exe from Jenkins. Killed all processes and restarted Jenkins Service
try to restart that inside windows services console, it will work
I have observed the same issue after installing a plugin and opting to restart the jenkins when no jobs are running.
When I looked at the jenkins server process, it was running fine and no issues.
On restarting the jenkins service using the below command and reloading the browser, Jenkins was up.
sudo service jenkins restart
If Jenkins is taking an unusually long time to restart the best recourse is to check the generated logs to see what may be wrong. However, even that may be of little help because many plugins try to be "quiet" by default, even if they are furiously working to load content. So if all else fails, you may have to resort to manually disabling plugins.
However here is a free tip: Some plugins are known to be messy. For example the Job Config History plugin we observed to write hundreds of thousands of records for both job configuration changes AND agent changes. Removing this plugin, and deleting the configHistory folder fixed one problem where our startup literally took > 4 hours.
In our case, the problem was we were launching ephemeral agents (via docker and/or kubernetes). Each new "agent" was treated as a configuration change. With thousands of agents per day, it didn't take long to fill up a substantial part of the disk with history that never was effectively cleared.
There are other plugins that leak data in this way. And you can also create self-inflicted wounds, e.g. by using a standalone process to remove "obsolete" files. An example where we were "bitten" is a process that tried to discard old build records, but did an incomplete job - and was "warring" with the running Jenkins process. Jenkins will try breaking its neck to load a build.xml record that is empty or incomplete.
Three more tips:
You can install the monitoring plugin. Often when the jenkins UI proper didn't start, we were able to see the /monitoring in action.
Likewise, /userContent can often be loaded even when the rest of the UI is not fully up.
Don't rule out bad actors. It just takes one aggressive script that tries, e.g. to load the entire build history and ship it back via a REST call to effectively deny service to all other UI users.
I try to fix a file named hudson.model.UpdateCenter.xml located /var/lib/jenkins
I change the URL to https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json
Finally, restart Jenkins. it solves my problem