I'm trying to upgrade my jenkins instance RAM at GCP. It's all normal until I start again the instance. It starts gracefully but cannot run the jenkins service.
Jun 24 12:44:57 jenkins-1-vm bitnami[6657]: no listening sockets available, shutting down
Jun 24 12:44:57 jenkins-1-vm bitnami[6657]: AH00015: Unable to open logs
Jun 24 12:44:57 jenkins-1-vm bitnami[6657]: ## 2021-06-24 12:44:57+00:00 ## INFO ## Running /opt/bitnami/var/init/post-start/010_bitnami_agent_extra…
Jun 24 12:44:57 jenkins-1-vm bitnami[6657]: ## 2021-06-24 12:44:57+00:00 ## INFO ## Running /opt/bitnami/var/init/post-start/020_bitnami_agent…
Jun 24 12:44:57 jenkins-1-vm bitnami[6657]: ## 2021-06-24 12:44:57+00:00 ## INFO ## Running /opt/bitnami/var/init/post-start/030_update_welcome_file…
Jun 24 12:44:57 jenkins-1-vm bitnami[6657]: ## 2021-06-24 12:44:57+00:00 ## INFO ## Running /opt/bitnami/var/init/post-start/040_bitnami_credentials_file…
Jun 24 12:44:57 jenkins-1-vm bitnami[6657]: ## 2021-06-24 12:44:57+00:00 ## INFO ## Running /opt/bitnami/var/init/post-start/050_clean_metadata…
Jun 24 12:44:57 jenkins-1-vm systemd[1]: bitnami.service: Control process exited, code=exited, status=1/FAILURE
Jun 24 12:44:57 jenkins-1-vm systemd[1]: bitnami.service: Failed with result ‘exit-code’.
Jun 24 12:44:57 jenkins-1-vm systemd[1]: Failed to start LSB: bitnami init script.
I spent hours for walk around try to fix this issue but still cannot find any solution. Any suggestion how can I fix it?
Many thanks.
After walk around and asking to Bitnami support for Jenkins, finally I got the point of the issues. The log doesn't provide me the real issue when i checked it. So, Its just because bitnami automaticly running apache on the instance when I restart again. Just stop and disabled the apache service then try to start the bitnami service. bitnami works well again. you can see here, the full step how to fixing the problem with support tools from bitnami.
If going back to previous RAM amount does not solve the issue, then the issue lies somewhere else.
I suspect bitnami config files have been corrupted when the VM has been stopped.
Try re-deploying and make sure you shutdown the system gracefully (e.g. shutdown -h) before changing specs.
Related
since last week, I am unable to set up a TLJH Development Environment as described here:
https://tljh.jupyter.org/en/latest/contributing/dev-setup.html.
Up until January 23 2023 the process went smoothly, but after rebuilding the image last Friday (27 January), after running bootstrap.py in the container, the server doesn't spawn. Browsing to my localhost:12000 I just see the message "404 page not found".
I am running TLJH in a container in an Ubuntu 22.04 host, running Docker version 20.10.21.
On the same host, when using the image I build before January 23 with the same method I can spawn the server ok. There were no new commits on the TLJH repo since, so the issue should not be there.
Here is the output of journalctl -u jupyterhub -f ran inside the container:
-- Logs begin at Mon 2023-01-30 12:44:34 UTC. --
Jan 30 12:44:41 87f857896c91 python3[64]: raise exc.ArgumentError(msg, code=code) from err
Jan 30 12:44:41 87f857896c91 python3[64]: sqlalchemy.exc.ArgumentError: Column expression, FROM clause, or other columns clause element expected, got [1]. Did you mean to say select(1)?
Jan 30 12:44:41 87f857896c91 python3[64]:
Jan 30 12:44:41 87f857896c91 systemd[1]: jupyterhub.service: Main process exited, code=exited, status=1/FAILURE
Jan 30 12:44:41 87f857896c91 systemd[1]: jupyterhub.service: Failed with result 'exit-code'.
Jan 30 12:44:41 87f857896c91 systemd[1]: jupyterhub.service: Scheduled restart job, restart counter is at 5.
Jan 30 12:44:41 87f857896c91 systemd[1]: Stopped jupyterhub.service.
Jan 30 12:44:41 87f857896c91 systemd[1]: jupyterhub.service: Start request repeated too quickly.
Jan 30 12:44:41 87f857896c91 systemd[1]: jupyterhub.service: Failed with result 'exit-code'.
Jan 30 12:44:41 87f857896c91 systemd[1]: Failed to start jupyterhub.service.
Since the logs show an error in sqlalchemy, I tried dowongrading sqlalchemy from 2.0.0 to 1.4.46, and then reloaded the hub, but that didn't solve the problem
Is anyone else experiancing the same issue? Any ideas for further troubleshooting?
Thanks!
EDIT: I applied the solution described here and it worked.
Commands to run:
pip3 install --upgrade 'SQLAlchemy<2.0.0'
tljh-config reload hub
I applied the solution described here and it worked.
The reason seems to be an issue introduced in SQLAlchemy 2.0.0, is resolved by downgrading SQLAlchemy.
Commands to run:
pip3 install --upgrade 'SQLAlchemy<2.0.0'
tljh-config reload hub
I've google this, but so far no way to fix it. My syslog under /var/log is being flooded every second with messages like this;
Aug 27 20:58:27 mail-server systemd[1]: run-docker-runtime\x2drunc-moby-e4bfb13118b141bf232cf981fe9b535706243c47ae0659466b8e6667bd4feceb-runc.YHoxmJ.mount: Succeeded.
Aug 27 20:58:27 mail-server systemd[1083]: run-docker-runtime\x2drunc-moby-e4bfb13118b141bf232cf981fe9b535706243c47ae0659466b8e6667bd4feceb-runc.YHoxmJ.mount: Succeeded.
Aug 27 20:58:27 mail-server systemd[8395]: run-docker-runtime\x2drunc-moby-e4bfb13118b141bf232cf981fe9b535706243c47ae0659466b8e6667bd4feceb-runc.YHoxmJ.mount: Succeeded.
Aug 27 20:58:28 mail-server systemd[1]: run-docker-runtime\x2drunc-moby-5dc4f4e0b3cbd5e5bfbcc88b8d22f92575706b7c3603847ccb2fd4e56f188f99-runc.gt51Ek.mount: Succeeded.
Aug 27 20:58:28 mail-server systemd[1083]: run-docker-runtime\x2drunc-moby-5dc4f4e0b3cbd5e5bfbcc88b8d22f92575706b7c3603847ccb2fd4e56f188f99-runc.gt51Ek.mount: Succeeded.
Aug 27 20:58:28 mail-server systemd[8395]: run-docker-runtime\x2drunc-moby-5dc4f4e0b3cbd5e5bfbcc88b8d22f92575706b7c3603847ccb2fd4e56f188f99-runc.gt51Ek.mount: Succeeded.
I am running Ubuntu 20.04 and dockerd is run by systemd.
Could anyone help me to find the cause if this? It seems that every single container is generating this.
Best,
Francis
Those messages are from systemd itself about the mount. This is addressed in systemd v249; see https://github.com/systemd/systemd/issues/6432 for more information.
In a nutshell, that version of systemd allows controlling of that mount via its unit file using the following:
[Mount]
LogLevelMax=0
The LogLevelMax setting applies not just to the unit but also to systemd's log messages itself about the unit. That is the change introduced in v249.
Trying to upgrade local Neo4j instances and getting errors on start.
Ubuntu 16.04: Trying to upgrade local instances of Neo4j databases which are currently at 3.3.1.
Installed Neo4j Desktop thinking I could do it with that and found the lowest version it upgrades from is 3.4. I now can't remove that. Ubuntu software fails (I just click remove, restart and Neo4j Desktop is still there). dpkg --list doesn't list neo4j desktop, so I can't use "apt-get remove" to remove it.
Uninstalled 3.3.1 and installed 3.3.9 (latest version of 3.3.x). Started Neo4j and ran fine, updating the database stores to 3.3.9.
Uninstalled 3.3.9 and installed 3.5.7.
Expecting Neo4j to start normally with "sudo service neo4j start", but now getting the following:
neo4j.service - Neo4j Graph Database
Loaded: loaded (/lib/systemd/system/neo4j.service; disabled; vendor preset: enabled)
Active: failed (Result: start-limit-hit) since Tue 2019-07-09 14:00:22 BST; 58s ago
Process: 1417 ExecStart=/usr/share/neo4j/bin/neo4j console (code=exited, status=1/FAILURE)
Main PID: 1417 (code=exited, status=1/FAILURE)
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Main process exited, code=exited, status=1/FAILURE
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Unit entered failed state.
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Failed with result 'exit-code'.
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Service hold-off time over, scheduling restart.
Jul 09 14:00:22 doug-ubuntu systemd[1]: Stopped Neo4j Graph Database.
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Start request repeated too quickly.
Jul 09 14:00:22 doug-ubuntu systemd[1]: Failed to start Neo4j Graph Database.
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Unit entered failed state.
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Failed with result 'start-limit-hit'.
Checked that I am running JVE 1.8:
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (build 1.8.0_212-8u212-b03-0ubuntu1.16.04.1-b03)
OpenJDK 64-Bit Server VM (build 25.212-b03, mixed mode)
Unsure how to proceed. Any ideas welcome please. Thank you!
Found the answer by running "journalctl -e -u neo4j" to view the full error log. Turns out that it was an APOC jar that was installed for 3.3.x but not compatible with 3.5.x. Removing the jar file solved it.
I am running the gitlab with docker, but it always exits after a period of time
==> /var/log/gitlab/redis/current <==
2019-06-21_18:00:08.72435 459:signal-handler (1561140008) Received SIGTERM scheduling shutdown...
2019-06-21_18:00:08.81864 459:M 21 Jun 18:00:08.817 # User requested shutdown...
2019-06-21_18:00:08.81866 459:M 21 Jun 18:00:08.817 * Saving the final RDB snapshot before exiting.
2019-06-21_18:00:08.83736 459:M 21 Jun 18:00:08.837 * DB saved on disk
2019-06-21_18:00:08.83741 459:M 21 Jun 18:00:08.837 * Removing the pid file.
2019-06-21_18:00:08.83817 459:M 21 Jun 18:00:08.838 * Removing the unix socket file.
2019-06-21_18:00:08.83935 459:M 21 Jun 18:00:08.839 # Redis is now ready to exit, bye bye...
ok: down: redis-exporter: 0s, normally up
==> /var/log/gitlab/gitlab-rails/sidekiq.log <==
2019-06-21_18:00:09.57615 2019-06-21T18:00:09.576Z 807 TID-oviw2sgmf INFO: Shutting down
2019-06-21_18:00:09.57625 2019-06-21T18:00:09.576Z 807 TID-ovivo05i7 INFO: Scheduler exiting...
2019-06-21_18:00:09.57655 2019-06-21T18:00:09.576Z 807 TID-oviw2sgmf INFO: Terminating quiet workers
This was reported in gitlab-org/omnibus-gitlab issue 4137: "runsv send SIGTERM to redis in docker version"
runsv sends SIGTERM to redis every 60 secs
gitlab-org/omnibus-gitlab issue 1611 suggests a docker restart first.
But the general issue is not conclusively resolved yet.
I have a mongrel server running behind Apache. It works fine; however, every now and then the Apache server shuts downs seemingly by itself. I'm not sure if there is configuration issue or if it's an attack. Here is Apache error log:
[Thu Apr 30 02:15:07 2009] [notice] SIGHUP received. Attempting to restart
[Thu Apr 30 02:15:07 2009] [warn] NameVirtualHost *:0 has no VirtualHosts
[Thu Apr 30 02:15:07 2009] [notice] Apache/2.2.3 (Debian) PHP/5.2.0-8+etch13 configured -- resuming normal operations
[Thu Apr 30 02:17:13 2009] [error] [client 61.139.105.163] File does not exist: /var/www/fastenv
[Thu Apr 30 02:24:06 2009] [error] [client 61.139.105.163] File does not exist: /var/www/fastenv
[Thu Apr 30 10:49:18 2009] [warn] pid file /var/run/apache2.pid overwritten -- Unclean shutdown of previous Apache run?
[Thu Apr 30 10:49:18 2009] [notice] Apache/2.2.3 (Debian) PHP/5.2.0-8+etch13 configured -- resuming normal operations
[Thu Apr 30 12:53:08 2009] [notice] SIGHUP received. Attempting to restart
[Thu Apr 30 12:53:08 2009] [warn] NameVirtualHost *:0 has no VirtualHosts
[Thu Apr 30 12:53:08 2009] [notice] Apache/2.2.3 (Debian) PHP/5.2.0-8+etch13 configured -- resuming normal operations
[Thu Apr 30 12:59:15 2009] [notice] SIGHUP received. Attempting to restart
[Thu Apr 30 12:59:15 2009] [warn] NameVirtualHost *:0 has no VirtualHosts
[Thu Apr 30 12:59:15 2009] [notice] Apache/2.2.3 (Debian) PHP/5.2.0-8+etch13 configured -- resuming normal operations
[Thu Apr 30 13:58:49 2009] [notice] SIGHUP received. Attempting to restart
[Thu Apr 30 13:58:49 2009] [warn] NameVirtualHost *:0 has no VirtualHosts
[Thu Apr 30 13:58:49 2009] [notice] Apache/2.2.3 (Debian) PHP/5.2.0-8+etch13 configured -- resuming normal operations
[Fri May 01 10:59:07 2009] [warn] pid file /var/run/apache2.pid overwritten -- Unclean shutdown of previous Apache run?
[Fri May 01 10:59:07 2009] [notice] Apache/2.2.3 (Debian) PHP/5.2.0-8+etch13 configured -- resuming normal operations
[Fri May 01 17:51:15 2009] [warn] pid file /var/run/apache2.pid overwritten -- Unclean shutdown of previous Apache run?
[Fri May 01 17:51:15 2009] [notice] Apache/2.2.3 (Debian) PHP/5.2.0-8+etch13 configured -- resuming normal operations
Not quite sure what is /var/www/fastenv but I don't think there is anything in my application that calls that. Also, website is still in Beta mode with few users and I don't think any have 61.139.105.163 IP address but it's possible that they might have it.
Any ideas? It would be good if you can give me hints where to look or how to go about anaysing this problem
I have the exact same log from the same IP. Looking it up shows it to belong to the Chinese government. It appears to be a scan using server side includes to find out as much as they can about your server. I banned the IP.
Not sure this is entirely programming-related, but anyway... none of those look like serious errors to me. The accesses to /var/www/fastenv just mean that the computer at IP address 61.139.105.163 sent a request for http://www.example.com/fastenv or something like that (it depends on exactly how you've configured your virtual hosts); I'd look at the access log for more information, to see what other requests have been coming from that IP address. It's probably not anything to worry about.
The line about NameVirtualHost *:0 means that somewhere in your configuration file you have an incorrect NameVirtualHost directive, maybe with no arguments. You should probably look for that and remove it, but if the server is running fine anyway, it's not a big deal.
The reason your server is terminating (restarting, actually) appears to be a SIGHUP - that is, something on the system is sending Apache a signal telling it to restart. It's basically the same thing that happens if you run apache2 restart, I think. Without knowing what's sending that signal, there's not more I can say.
61.139.105.163 is known for doing all kinds of hacking type things, just google the IP address. You should definitly ban this IP address.
Click on Apache Config --> Apache(httpd.conf)
Search for #Listen 12.34.56.78:80 and replace it with #Listen 12.34.56.78:8081.
Search for Listen 80 and replace it with Listen 8081.
Now you can start Apache now, and can run it with this URL: localhost:8081/xampp/