We have a large number of linux machines as a compute farm. We
launch jobs on the farm using LSF.
Sporadically, and randomly, some job is creating and delete thousands of 'tmp' files in
my home directly:
ls /home/cpp_home/tmp*
---------- 1 cpp_home dev 0 Dec 10 14:25 tmpxJL9In
-rw------- 1 cpp_home dev 0 Dec 10 14:25 tmpnvAtiS
-rw------- 1 cpp_home dev 0 Dec 10 14:25 tmphSrnk7
-rw------- 1 cpp_home dev 0 Dec 10 14:25 tmpJFO5Cr
---------- 1 cpp_home dev 0 Dec 10 14:25 tmpRIzn7A
-rw------- 1 cpp_home dev 0 Dec 10 14:25 tmpvulwsT
---------- 1 cpp_home dev 0 Dec 10 14:25 tmpeSz_gN
---------- 1 cpp_home dev 0 Dec 10 14:25 tmpEcatTM
-rw------- 1 cpp_home dev 0 Dec 10 14:25 tmpOy1jdi
---------- 1 cpp_home dev 0 Dec 10 14:26 tmp4oB8ua
How the hell can I found out what process is doing this?
They look suspiciously like std 'C' library tempfile, or standard
python tempfiles.... but since they don't stick around for
long I can't find out what job (of the thousands that are running via LSF)
are creating them.
I don't have source code for all the jobs... There is a great deal of
third party CAD/EDA tools in use, so it could be one of them. Or it could
be perl, or python scripts, or...
If the variety of jobs is high, this could be difficult to find; however, if the job types are not too diverse, you should be able to correlate the times that you find these being created with jobs owned by the user owning the files. By running similar jobs in isolation, you could then test to verify the behavior.
There is perhaps though another possibility. Depending on LSF settings some output/logging files may be created as temp files and copied into their final location. I could imagine this may explain the phenomenon, but normally these would fall under a $HOME/.lsbatch directory. Settings might be able to adjust this location. See this text from the BSUB command reference:
If the parameter LSB_STDOUT_DIRECT in lsf.conf is set to Y or y, and you use
the -o or -oo option, the standard output of a job is written to the file you
specify as the job runs. If LSB_STDOUT_DIRECT is not set, and you use -o or -oo,
the standard output of a job is written to a temporary file and copied to the
specified file after the job finishes.
Related
Jenkins version 2.114 is running builds but they seem to disappear from the Dashboard after a restart. I have Jenkins setup on Linux and all the log is present under
${JENKINS_HOME}/jobs/${ITEM_FULL_NAME}/builds
I have tried Reload Configuration from Disk with following set in /etc/sysconfig/jenkins
JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true"
Below is build and workspace directory settings:
Workspace Root Directory :${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}
Build Record Root Directory: ${ITEM_ROOTDIR}/builds
Below you can see the build number 117 and 118, but still, the lastFailedBuild, lastStableBuild , lastSuccessfulBuild are set to -1, which doesn't exist
drwxr-xr-x 4 jenkins jenkins 4096 Apr 2 20:17 117
drwxr-xr-x 4 jenkins jenkins 4096 Apr 3 20:17 118
lrwxrwxrwx 1 jenkins jenkins 2 Apr 3 11:09 lastFailedBuild -> -1
lrwxrwxrwx 1 jenkins jenkins 2 Apr 4 10:44 lastStableBuild -> -1
lrwxrwxrwx 1 jenkins jenkins 2 Apr 4 03:21 lastSuccessfulBuild -> -1
lrwxrwxrwx 1 jenkins jenkins 2 Apr 3 10:58 lastUnstableBuild -> -1
lrwxrwxrwx 1 jenkins jenkins 2 Apr 3 10:58 lastUnsuccessfulBuild -> -1
I have also set the
Discard Old Builds Slicer - Days to keep artifacts as 30
Discard Old Builds Slicer - Max # of builds to keep as 40
I ran the jenkins-job-checker[ https://github.com/docwhat/jenkins-job-checker ] on one of the jobs and found the following :
Problem: NOTLINK: The number link builds/1 is not a symlink!
Problem: NOTLINK: The number link builds/2 is not a symlink!
Problem: NOTLINK: The number link builds/3 is not a symlink!
Proposal: Archive non-link builds/1
Proposal: Archive non-link builds/2
Proposal: Archive non-link builds/3
Really appreaciate some help
The links have been replaced with the permalinks file:
rwxr-xr-x 2 jenkins jenkins 4096 Aug 6 2018 98
drwxr-xr-x 2 jenkins jenkins 4096 Aug 7 2018 99
lrwxrwxrwx 1 jenkins jenkins 3 Jan 19 13:47 lastFailedBuild -> 649
lrwxrwxrwx 1 jenkins jenkins 3 Jan 19 13:48 lastSuccessfulBuild -> 622
lrwxrwxrwx 1 jenkins jenkins 2 Jun 14 2018 lastUnstableBuild -> -1
lrwxrwxrwx 1 jenkins jenkins 3 Jan 19 13:49 lastUnsuccessfulBuild -> 649
-rw-r--r-- 1 jenkins jenkins 0 Jun 14 2018 legacyIds
-rw-r--r-- 1 jenkins jenkins 134 Jan 19 14:42 permalinks
jenkins#ip-1-9-2-253_jenkins:~/jobs/cron_dockers/jobs/ios/builds$ cat permalinks
lastCompletedBuild 651
lastFailedBuild 650
lastStableBuild 651
lastSuccessfulBuild 651
lastUnstableBuild -1
lastUnsuccessfulBuild 650
Make sure the values in the permalinks file are correct, these are the values that are used to generate the values in the Jenkins GUI.
I installed the plugin operations-center-context and that solved my problem
I found the solution in below link:
https://support.cloudbees.com/hc/en-us/articles/216227047-After-moving-a-job-symlinks-for-folders-became-actual-folders
I have a jenkins running a job, after which I want it to "archive artifact", which is basically just get some files.
The Job clones a github and when ran, it generates some log files which I need.
The patter is:
logfiles/*
I added a command to list the directory to make sure the files are there, and indeed they are.
+ ls -la logfiles
total 24
drwxr-xr-x 2 root root 4096 Apr 22 23:23 .
drwxr-xr-x 8 root root 4096 Apr 22 23:23 ..
-rw-r--r-- 1 root root 0 Apr 22 23:23 1461367410777_testuuid_61746144-3A3A-5555-4944-3D5343414C41.log
-rw-r--r-- 1 root root 1248 Apr 22 23:23 1461367410777_testuuid_61746144-3A3A-5555-4944-3D5343414C41_shortmsg.csv
-rw-r--r-- 1 root root 2521 Apr 22 23:23 1461367410777_testuuid_61746144-3A3A-5555-4944-3D5343414C41_stats.log
-rw-r--r-- 1 root root 8035 Apr 22 23:23 1461367410777_testuuid_61746144-3A3A-5555-4944-3D5343414C41_trace_msg.log
Archiving artifacts
ERROR: No artifacts found that match the file pattern "logfiles/*". Configuration error?
ERROR: ‘logfiles/*’ doesn’t match anything, but ‘*’ does. Perhaps that’s what you mean?
Build step 'Archive the artifacts' changed build result to FAILURE
I don't really understand why they are not found.
Hope someone can help!
Thanks!
I finally found my problem. I had configured the job to do the git clone. But for some reason I also added the cloning on the build steps... once i removed that jenkins started getting the files properly.
I would still like to understand why this would happen...
Thanks to all
I am really happy to see that a Valgrind plugin exists for Jenkins. I use it for C/C++ code at work.
I have set it up in Jenkins (Linux Ubuntu 14.04 - the valgrind plugin version 0.22) to dump xml files.
My config is like this
I can see that I do get my memcheck files out in the xml directory
-rw------- 1 jenkins jenkins 1379 Oct 25 18:21 main.18996.memcheck
-rw------- 1 jenkins jenkins 1379 Oct 25 18:22 main.19100.memcheck
-rw------- 1 jenkins jenkins 2452 Oct 25 18:27 main.19489.memcheck
-rw------- 1 jenkins jenkins 2453 Oct 25 18:28 main.19605.memcheck
-rw------- 1 jenkins jenkins 2453 Oct 25 18:28 main.19692.memcheck
-rw------- 1 jenkins jenkins 2453 Oct 25 18:28 main.19774.memcheck
-rw------- 1 jenkins jenkins 1379 Oct 25 18:29 main.19963.memcheck
I can see that the memcheck files look fine with some "dirty underwear" such as
<error>
<unique>0xb</unique>
<tid>1</tid>
<kind>InvalidWrite</kind>
<what>Invalid write of size 4</what>
<stack>
<frame>
<ip>0x80483EB</ip>
<obj>/home/jenkins/workspace/DemoValgrind/main</obj>
<fn>main</fn>
<dir>/home/jenkins/workspace/DemoValgrind</dir>
<file>main.c</file>
<line>12</line>
</frame>
</stack>
<auxwhat>Address 0x41ae21c is not stack'd, malloc'd or (recently) free'd</auxwhat>
</error>
My problem is that the Valgrind plugin counts the sum of all errors in all of the main.*.memcheck files.
I expected a view more like this one:
https://wiki.jenkins-ci.org/download/attachments/60918012/valgrind-trend-graph.jpg?version=1&modificationDate=1336573302000
where the number of errors go up and down.
I must be configuring the Valgrind "Publish Valgrind Results" wrongly.
Is there a syntax where I can get the overview diagram (the URL just above) to match the number of errors according to the build, i.e.
not accumulate?
I am trying to generate GDCA files using gcov in XCode, and have been using BubbleFoundry and this post as guides.
I have a separate test target created, which prints a bunch of lines when done like:
profiling: /Users/me/Library/Developer/Xcode/DerivedData/
MyProject-HASH/Build/Intermediates/MyProject.build/Debug-iphoneos/
MyProject Tests.build/Objects-normal/
armv7/mysource.gcda: cannot open: No such file or directory
After trying the instructions in both posts I tried setting up CoverStory, which resulted in the same error message.
My permissions in that directory:
8 -rwxrwxr-x 1 me staff 474 Nov 5 21:54 mysource.d
8 -rwxrwxr-x 1 me staff 220 Nov 5 21:54 mysource.dia
144 -rwxrwxr-x 1 me staff 71136 Nov 5 21:54 mysource.gcno
296 -rwxrwxr-x 1 me staff 148752 Nov 5 21:54 mysource.o
I previously tried running the permissions script from BubbleFoundy, but I still get the cannot open error (a.k.a. gdca files not written).
Apparently, I was generating GCDA files to the phone, which weren't being found. I fixed this by manual deleting all the directories in /Users/me/Library/Developer/Xcode/DerivedData/MyProject-HASH/Build/Intermediates/MyProject.build/ and then rerunning tests in the simulator.
Once an iOS device is jailbroken, we can build jailbreak apps (with theos) and it gets installed in the /Applications directory where the preloaded apps run with root privileges. If an app is built with Xcode, once it is installed, it gets into the /private/var/mobile/Applications/ folder, which is supposed to have Apple sandbox enforced (before jailbreak).
So, the questions I have are:
For a jailbroken device, will the apps in /private/var/mobile/Applications/ execute with root privileges or with mobile user privileges?
In case of Android, once rooted, the apps will have to gain root privileges by executing the su command. Is it the case when it comes to iOS as well?
I would like to understand the difference between these two development options (Theos / Xcode) and how it affects what operations my app can perform.
Not disagreeing with anything H2CO3 said, but to add some further clarification ...
Apps installed in /private/var/mobile/Applications/(†) with Xcode will run with user mobile privileges, even on jailbroken phones.
Even on a jailbroken phone, apps installed to /private/var/mobile/Applications/(†) will be sandboxed almost (‡) like apps on a jailed phone. So, no reading other (normal) apps' data, even if those files are owned by user mobile.
For a good description of the process that apps like Cydia use to run as root, see this answer. Or, just ssh into your phone, and take a look inside /Applications/Cydia.app/ yourself.
If you simply copy/install an app (without doing what H2CO3 suggested) to /Applications/, it won't be sandboxed, but it will still run with mobile (UID=501) privileges:
iPhone5:~ root# cd /Applications
iPhone5:/Applications root# ls -altr ./HelloJB.app/
total 220
-rw-r--r-- 1 root wheel 711 Apr 3 20:36 entitlements.xml
-rw-r--r-- 1 root wheel 297 Apr 3 20:36 entitlements-daemon.xml
-rw-r--r-- 1 root wheel 7972 Apr 3 20:36 embedded.mobileprovision
-rw-r--r-- 1 root wheel 58755 Apr 3 20:36 date.zip
-rw-r--r-- 1 root wheel 485 Apr 3 20:36 ResourceRules.plist
-rw-r--r-- 1 root wheel 8 Apr 3 20:36 PkgInfo
-rw-r--r-- 1 root wheel 1226 Apr 3 20:36 Info.plist
-rw-r--r-- 1 root wheel 10960 Apr 3 20:36 Icon\#2x.png
-rw-r--r-- 1 root wheel 8328 Apr 3 20:36 Icon.png
-rw-r--r-- 1 root wheel 451 Apr 3 20:36 HelloJB.plist
-rwxr-xr-x 1 root wheel 61088 Apr 3 20:36 HelloJB*
-rwxr-xr-x 1 root wheel 42688 Apr 3 20:36 HelloDaemon*
drwxr-xr-x 2 root wheel 136 Apr 3 20:36 en.lproj/
drwxr-xr-x 2 root wheel 102 Apr 3 20:36 _CodeSignature/
drwxr-xr-x 4 root wheel 544 Apr 3 20:36 ./
drwxrwxr-x 54 root admin 1904 Apr 5 02:14 ../
iPhone5:/Applications root# ps -Aef | grep HelloJB
501 9412 1 0 0:00.00 ?? 0:00.33 /Applications/HelloJB.app/HelloJB
iPhone5:/Applications root# grep mobile /etc/passwd
mobile:*:501:501:Mobile User:/var/mobile:/bin/sh
(‡) Here's a good discussion, with input from Saurik, about how different jailbreaks may affect the sandbox. Long story short: it depends.
(†) Update: in recent versions of iOS, the location of 3rd-party apps has been moved to /var/mobile/Containers, and later to /var/containers/, but the same basic sandbox issues remain.
Long story short: no.
Jailbreaking is a necessary but not sufficient condition for gaining root. Apps will still be sandboxed by default.
What you can do for making your app run with root privileges is creating a startup shell script that has root:wheel ownership and 755 permissions, then create your actual executable with the same ownership, 7555 as permissions (i. e. set its "setuid" bit), then call setuid(0); from within main(), before calling UIApplicationMain().