I am running Rstudio-server inside a centos container. But I am not able to get the logs from the running container. I have already tried RStudio Server - Application Logs as well as on stackoverflow answer, but they were not any help to me. I have tried various versions of Rstudio-servers.
In my CentOS 7 Docker container, and with my RStudio Server version 1.0.136, I've managed to access to the logs using:
journalctl -u rstudio-server
Some more logs can be found using
journalctl -f
In a more general point of view, there are many logs in the following folder: /var/log/anaconda
According to RStudio Server support, another solution would be to launch rstudio-server with --server-daemonize=0 in order for the output to be directly available on standard output & error.
i performed on my machine and i got all the logs
[root#ccanalytics-dev-dev01 rksharma]# journalctl -u rstudio-server
-- Logs begin at Wed 2017-05-17 15:43:44 EDT, end at Fri 2018-01-12 03:49:17 EST. --
May 18 08:14:00 ccanalytics-dev-dev01.netlink.com systemd[1]: Starting RStudio Server...
May 18 08:14:00 ccanalytics-dev-dev01.netlink.com systemd[1]: Started RStudio Server.
May 18 08:16:02 ccanalytics-dev-dev01.netlink.com rserver[40106]: ERROR system error 13 (Permission denied) [user-value=admin]; OCCURRED AT:
May 18 08:16:02 ccanalytics-dev-dev01.netlink.com rserver-pam[40159]: ERROR pam_authenticate failed: User not known to the underlying authent
May 18 08:44:16 ccanalytics-dev-dev01.netlink.com rserver[40106]: ERROR system error 13 (Permission denied) [user-value=rishabs]; OCCURRED AT
May 18 08:44:16 ccanalytics-dev-dev01.netlink.com rserver-pam[40193]: ERROR pam_authenticate failed: User not known to the underlying authent
May 23 05:17:06 ccanalytics-dev-dev01.netlink.com killall[20898]: rserver: no process found
May 23 05:17:06 ccanalytics-dev-dev01.netlink.com systemd[1]: rstudio-server.service: control process exited, code=exited status=1
May 23 05:17:06 ccanalytics-dev-dev01.netlink.com systemd[1]: Unit rstudio-server.service entered failed state.
May 23 05:17:06 ccanalytics-dev-dev01.netlink.com systemd[1]: rstudio-server.service failed.
May 23 05:17:07 ccanalytics-dev-dev01.netlink.com systemd[1]: rstudio-server.service holdoff time over, scheduling restart.
May 23 05:17:07 ccanalytics-dev-dev01.netlink.com systemd[1]: Starting RStudio Server...
May 23 05:17:07 ccanalytics-dev-dev01.netlink.com systemd[1]: Started RStudio Server.
May 23 05:19:00 ccanalytics-dev-dev01.netlink.com systemd[1]: rstudio-server.service: main process exited, code=killed, status=9/KILL
May 23 05:19:00 ccanalytics-dev-dev01.netlink.com killall[20956]: rserver: no process found
May 23 05:19:00 ccanalytics-dev-dev01.netlink.com systemd[1]: rstudio-server.service: control process exited, code=exited status=1
May 23 05:19:00 ccanalytics-dev-dev01.netlink.com systemd[1]: Unit rstudio-server.service entered failed state.
May 23 05:19:00 ccanalytics-dev-dev01.netlink.com systemd[1]: rstudio-server.service failed.
May 23 05:19:00 ccanalytics-dev-dev01.netlink.com systemd[1]: rstudio-server.service holdoff time over, scheduling restart.
May 23 05:19:00 ccanalytics-dev-dev01.netlink.com systemd[1]: Starting RStudio Server...
May 23 05:19:00 ccanalytics-dev-dev01.netlink.com systemd[1]: Started RStudio Server.
May 23 05:22:25 ccanalytics-dev-dev01.netlink.com killall[20993]: rserver: no process found
May 23 05:22:25 ccanalytics-dev-dev01.netlink.com systemd[1]: rstudio-server.service: control process exited, code=exited status=1
May 23 05:22:25 ccanalytics-dev-dev01.netlink.com systemd[1]: Unit rstudio-server.service entered failed state.
May 23 05:22:25 ccanalytics-dev-dev01.netlink.com systemd[1]: rstudio-server.service failed.
May 23 05:22:25 ccanalytics-dev-dev01.netlink.com systemd[1]: rstudio-server.service holdoff time over, scheduling restart.
May 23 05:22:25 ccanalytics-dev-dev01.netlink.com systemd[1]: Starting RStudio Server...
May 23 05:22:25 ccanalytics-dev-dev01.netlink.com systemd[1]: Started RStudio Server.
May 23 05:22:51 ccanalytics-dev-dev01.netlink.com systemd[1]: rstudio-server.service: main process exited, code=killed, status=9/KILL
May 23 05:22:51 ccanalytics-dev-dev01.netlink.com killall[21022]: rserver: no process found
May 23 05:22:51 ccanalytics-dev-dev01.netlink.com systemd[1]: rstudio-server.service: control process exited, code=exited status=1
May 23 05:22:51 ccanalytics-dev-dev01.netlink.com systemd[1]: Unit rstudio-server.service entered failed state.
May 23 05:22:51 ccanalytics-dev-dev01.netlink.com systemd[1]: rstudio-server.service failed.
May 23 05:22:51 ccanalytics-dev-dev01.netlink.com systemd[1]: rstudio-server.service holdoff time over, scheduling restart.
May 23 05:22:51 ccanalytics-dev-dev01.netlink.com systemd[1]: Starting RStudio Server...
May 23 05:22:51 ccanalytics-dev-dev01.netlink.com systemd[1]: Started RStudio Server.
May 23 05:23:10 ccanalytics-dev-dev01.netlink.com rsession-rksharma[21050]: ERROR session hadabend; LOGGED FROM: rstudio::core::Error
Related
I was trying to install jenkins in ec2 ubuntu and amazon-linux but I am getting this error
jenkins.service - Jenkins Continuous Integration Server
Loaded: loaded (/lib/systemd/system/jenkins.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2022-11-27 16:29:54 UTC; 1h 8min ago
Process: 18706 ExecStart=/usr/bin/jenkins (code=exited, status=1/FAILURE)
Main PID: 18706 (code=exited, status=1/FAILURE)
Nov 27 16:29:54 ip-172-31-19-156 systemd[1]: jenkins.service: Main process exited, code=exited, status=1/FAILURE
Nov 27 16:29:54 ip-172-31-19-156 systemd[1]: jenkins.service: Failed with result 'exit-code'.
Nov 27 16:29:54 ip-172-31-19-156 systemd[1]: Failed to start Jenkins Continuous Integration Server.
Nov 27 16:29:54 ip-172-31-19-156 systemd[1]: jenkins.service: Scheduled restart job, restart counter is at 5.
Nov 27 16:29:54 ip-172-31-19-156 systemd[1]: Stopped Jenkins Continuous Integration Server.
Nov 27 16:29:54 ip-172-31-19-156 systemd[1]: jenkins.service: Start request repeated too quickly.
Nov 27 16:29:54 ip-172-31-19-156 systemd[1]: jenkins.service: Failed with result 'exit-code'.
Nov 27 16:29:54 ip-172-31-19-156 systemd[1]: Failed to start Jenkins Continuous Integration Server.
root#ip-172-31-19-156:~# cd /car/usr
First I installed jenkins and was interrupted in between that I changed the ec2 volume for reset my ec2, after this I'm unable to install jenkins in ec2
After installing Java and Jenkins on my CentOS 7 server. I tried to start the Jenkins, and I am getting the below error message.
Job for jenkins.service failed. See "systemctl status jenkins.service"
and "journalctl -xe" for details.
When I run "systemctl status jenkins.service" to see what the issue is, I get the below output
● jenkins.service - Jenkins Continuous Integration Server
Loaded: loaded (/usr/lib/systemd/system/jenkins.service; disabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Thu 2022-08-18 14:23:02 UTC; 20s ago
Process: 8847 ExecStart=/usr/bin/jenkins (code=exited, status=0/SUCCESS)
Main PID: 8847 (code=exited, status=0/SUCCESS)
Aug 18 14:23:02 localhost.localdomain systemd[1]: Failed to start Jenkins Continuous Integration Server.
Aug 18 14:23:02 localhost.localdomain systemd[1]: Unit jenkins.service entered failed state.
Aug 18 14:23:02 localhost.localdomain systemd[1]: jenkins.service failed.
Aug 18 14:23:02 localhost.localdomain systemd[1]: jenkins.service holdoff time over, scheduling restart.
Aug 18 14:23:02 localhost.localdomain systemd[1]: Stopped Jenkins Continuous Integration Server.
Aug 18 14:23:02 localhost.localdomain systemd[1]: start request repeated too quickly for jenkins.service
Aug 18 14:23:02 localhost.localdomain systemd[1]: Failed to start Jenkins Continuous Integration Server.
Aug 18 14:23:02 localhost.localdomain systemd[1]: Unit jenkins.service entered failed state.
Aug 18 14:23:02 localhost.localdomain systemd[1]: jenkins.service failed.
Not quite sure how to fix this. Anybody with a solution? Thanks
can you please use journalctl -xe for more detailed logs.
also can you run Jenkins in interactive mode to see why its failing to start like -
java -jar jenkins.war
you can get command details in /usr/bin/jenkins file.
I noticed that when launching some Jenkins builds sometimes the node hosting Jenkins get stuck forever. It means the whole node is not reachable, and all its pods are down (not ready in the dashboard).
To make things up again I need to remove it from the cluster and add it again (I'm on GCE so I need to remove it from the instance group to be able to delete it).
Note: during hours I'm not able to connect through SSH to the node, it is clearly out of service ^^
From my understanding, reaching memory top crashes a node, but reaching top CPU usage should just slow down the server and not make a big deal like what I'm experiencing. In the worst case Kubelet should be unavailable until CPU gets better.
Does someone is able to help me determining the origin of this issue? What could cause such a problem?
Node metrics 1
Node metrics 2
Jenkins slave metrics
Node metrics from GCE
On the other side, after waiting hours, I've been able to access the node through SSH and I run sudo journalctl -u kubelet to see what's going on. I don't see anything specific at 7pm o'clock but I'm able to see recurrent error like:
Apr 04 19:00:58 nodes-s2-2g5v systemd[43508]: kubelet.service: Failed at step EXEC spawning /home/kubernetes/bin/kubelet: Permission denied
Apr 04 19:00:58 nodes-s2-2g5v systemd[1]: kubelet.service: Main process exited, code=exited, status=203/EXEC
Apr 04 19:00:58 nodes-s2-2g5v systemd[1]: kubelet.service: Unit entered failed state.
Apr 04 19:00:58 nodes-s2-2g5v systemd[1]: kubelet.service: Failed with result 'exit-code'.
Apr 04 19:01:00 nodes-s2-2g5v systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Apr 04 19:01:00 nodes-s2-2g5v systemd[1]: Stopped Kubernetes Kubelet Server.
Apr 04 19:01:00 nodes-s2-2g5v systemd[1]: Started Kubernetes Kubelet Server.
Apr 04 19:01:00 nodes-s2-2g5v systemd[43511]: kubelet.service: Failed at step EXEC spawning /home/kubernetes/bin/kubelet: Permission denied
Apr 04 19:01:00 nodes-s2-2g5v systemd[1]: kubelet.service: Main process exited, code=exited, status=203/EXEC
Apr 04 19:01:00 nodes-s2-2g5v systemd[1]: kubelet.service: Unit entered failed state.
Apr 04 19:01:00 nodes-s2-2g5v systemd[1]: kubelet.service: Failed with result 'exit-code'.
Apr 04 19:01:02 nodes-s2-2g5v systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Apr 04 19:01:02 nodes-s2-2g5v systemd[1]: Stopped Kubernetes Kubelet Server.
Apr 04 19:01:02 nodes-s2-2g5v systemd[1]: Started Kubernetes Kubelet Server.
I go to older logs and I found at 5:30pm the start of this kind of messages:
Apr 04 17:26:50 nodes-s2-2g5v kubelet[1841]: I0404 17:25:05.168402 1841 prober.go:111] Readiness probe for "...
Apr 04 17:26:50 nodes-s2-2g5v kubelet[1841]: I0404 17:25:04.021125 1841 prober.go:111] Readiness probe for "...
-- Reboot --
Apr 04 17:31:31 nodes-s2-2g5v systemd[1]: Started Kubernetes Kubelet Server.
Apr 04 17:31:31 nodes-s2-2g5v systemd[1699]: kubelet.service: Failed at step EXEC spawning /home/kubernetes/bin/kubelet: Permission denied
Apr 04 17:31:31 nodes-s2-2g5v systemd[1]: kubelet.service: Main process exited, code=exited, status=203/EXEC
Apr 04 17:31:31 nodes-s2-2g5v systemd[1]: kubelet.service: Unit entered failed state.
Apr 04 17:31:31 nodes-s2-2g5v systemd[1]: kubelet.service: Failed with result 'exit-code'.
Apr 04 17:31:33 nodes-s2-2g5v systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Apr 04 17:31:33 nodes-s2-2g5v systemd[1]: Stopped Kubernetes Kubelet Server.
Apr 04 17:31:33 nodes-s2-2g5v systemd[1]: Started Kubernetes Kubelet Server.
At this time node kubelet reboots and it corresponds to a Jenkins build. There is the same pattern with high CPU usage. I don't know why earlier it just rebooted and around 7pm the node just get stuck :/
I'm really sorry, it's a lot of information but I'm totally lost, that's not the first time it happens to me ^^
Thank you,
As mentioned by #Brandon, it was related to resource limits applied to my Jenkins slaves.
In my case even if precised in my Helm chart YAML file, the values were not set. I had to go deeper in the UI to set them manually.
From this modification, everything is now stable! :)
We have installed docker 17.12 in our Centos 7.x and after the installation is complete, am facing an error while trying to start the docker service. Initially, I tried for systemctl docker start then for more output on this when I tried journalctl it says docker.service entered failed state.
More details below:
Docker :
17.12.1-ce , build 7390fc6
Command tried:
sudo systemctl start docker
journalctl -u docker.service
Expected Output:
Docker service should be started successfully
Actual output:
Mar 26 23:51:19 docker[16420]: See 'docker --help'
Mar 26 23:51:19 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Mar 26 23:51:19 systemd[1]: Failed to start Docker Application Container Engine.
Mar 26 23:51:19 systemd[1]: Unit docker.service entered failed state.
Mar 26 23:51:19 docker.service failed.
Mar 26 23:51:21 systemd[1]: docker.service holdoff time over, scheduling restart.
Mar 26 23:51:21 systemd[1]: start request repeated too quickly for docker.service
Mar 26 23:51:21 systemd[1]: Failed to start Docker Application Container Engine.
Mar 26 23:51:21 systemd[1]: Unit docker.service entered failed state.
Mar 26 23:51:21 systemd[1]: docker.service failed.
Mar 26 23:52:22 systemd[1]: Starting Docker Application Container Engine...
Mar 26 23:52:22 docker[16582]: docker: 'daemon' is not a docker command.
Mar 26 23:52:22 docker[16582]: See 'docker --help'
Mar 26 23:52:22 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Mar 26 23:52:22 systemd[1]: Failed to start Docker Application Container Engine.
Mar 26 23:52:22 systemd[1]: Unit docker.service entered failed state.
Mar 26 23:52:22 systemd[1]: docker.service failed.
Mar 26 23:52:24 systemd[1]: docker.service holdoff time over, scheduling restart.
Mar 26 23:52:24 systemd[1]: Starting Docker Application Container Engine...
Mar 26 23:52:25 docker[16601]: docker: 'daemon' is not a docker command.
Mar 26 23:52:25 docker[16601]: See 'docker --help'
Mar 26 23:52:25 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Mar 26 23:52:25 systemd[1]: Failed to start Docker Application Container Engine.
Mar 26 23:52:25 systemd[1]: Unit docker.service entered failed state.
Mar 26 23:52:25 systemd[1]: docker.service failed.
Mar 26 23:52:27 systemd[1]: docker.service holdoff time over, scheduling restart.
Mar 26 23:52:27 systemd[1]: Starting Docker Application Container Engine...
Mar 26 23:52:27 docker[16619]: docker: 'daemon' is not a docker command.
Mar 26 23:52:27 docker[16619]: See 'docker --help'
Mar 26 23:52:27 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Mar 26 23:52:27 systemd[1]: Failed to start Docker Application Container Engine.
Mar 26 23:52:27 systemd[1]: Unit docker.service entered failed state.
Mar 26 23:52:27 systemd[1]: docker.service failed.
Mar 26 23:52:29 systemd[1]: docker.service holdoff time over, scheduling restart.
Mar 26 23:52:29 systemd[1]: start request repeated too quickly for docker.service
Mar 26 23:52:29 systemd[1]: Failed to start Docker Application Container Engine.
Mar 26 23:52:29 systemd[1]: Unit docker.service entered failed state.
Mar 26 23:52:29 systemd[1]: docker.service failed.
Please check on this issue and help us resolve the docker start issue.
no evidence in your log.
Would you just reinstall with the official way ?
$ curl -fsSL https://get.docker.com -o get-docker.sh
$ sh get-docker.sh
Check if there's another issue with:
sudo dockerd --debug
In my situation I had invalid config in the daemon.json.
24 15:28:57 ivum01-HP-Pro-3330-SFF systemd[1]: marathon.service: Main process exited, code=exited, status=1/FAILURE
Jan 24 15:28:57 ivum01-HP-Pro-3330-SFF systemd[1]: marathon.service: Unit entered failed state.
Jan 24 15:28:57 ivum01-HP-Pro-3330-SFF systemd[1]: marathon.service: Failed with result 'exit-code'.
Jan 24 15:29:57 ivum01-HP-Pro-3330-SFF systemd[1]: marathon.service: Service hold-off time over, scheduling restart.
Jan 24 15:29:57 ivum01-HP-Pro-3330-SFF systemd[1]: Stopped Scheduler for Apache Mesos.
Jan 24 15:29:57 ivum01-HP-Pro-3330-SFF systemd[1]: Starting Scheduler for Apache Mesos...
Jan 24 15:29:57 ivum01-HP-Pro-3330-SFF systemd[1]: Started Scheduler for Apache Mesos.
Jan 24 15:29:57 ivum01-HP-Pro-3330-SFF marathon[1838]: No start hook file found ($HOOK_MARATHON_START). Proceeding with the start script.
Jan 24 15:29:59 ivum01-HP-Pro-3330-SFF marathon[1838]: [scallop] Error: Required option 'master' not found
Jan 24 15:29:59 ivum01-HP-Pro-3330-SFF systemd[1]: marathon.service: Main process exited, code=exited, status=1/FAILURE
Jan 24 15:29:59 ivum01-HP-Pro-3330-SFF systemd[1]: marathon.service: Unit entered failed state.
These are the commands I am using for Marathon:
sudo mkdir -p /etc/marathon/conf
sudo cp /etc/mesos-master/hostname /etc/marathon/conf
sudo cp /etc/mesos/zk /etc/marathon/conf/master
sudo cp /etc/marathon/conf/master /etc/marathon/conf/zk
sudo nano /etc/marathon/conf/zk
The only portion I need to modify in this file is the endpoint. Change it from /mesos to /marathon.
That’s an out of memory error. Are you sure your node has enough memory to run both Mesos Master and Marathon?