We have a number of broken docker image uploads in Artifactory. It's quite difficult to clean these up, since the package search feature does not find these image tags as packages. In the UI, the only way to remove these without search is 1 tag at a time. I'm curious as to whether anyone else has found a solution for this. Ideally, if there were some AQL or other method to identify and remove any folder in a docker repo that does not contain a manifest file.
You can try creating AQL Query. AQL has capabilities to search for artifacts based on properties which will help you in achieving clean up the way you want. https://www.jfrog.com/confluence/display/RTF/Artifactory+Query+Language
I don't think you can trap this with a single AQL,
but here is an idea that uses 2 AQLs -
Prepare a list of all paths that contain a manifest.json file
Prepare a list of all paths that contain sha256__* files
(will need to make it unique, because the same path will be
listed multiple times, probably)
Sort the two lists and compare them to each-other
Lines (i.e.: paths) that are showing only in the second list are
paths to broken images that are missing their manifest file
Now, after confirming the result-list (from step 4) is correct,
you can construct from it a set of DELETE API-calls (one for each path).
Related
Can you change the location of 'rst' files in sphinx without changing their URIs? I'm working on a documentation where we want to move some files to different folders, without changing the URIs:
For Example: If you create a sphinx project with $ sphinx-quickstart and add some files and folders:
index.rst
/tutorials/howToFoo.rst
/scripts/
With the toctree in in index.rst looking like that:
.. toctree::
:maxdepth: 1
:caption: Processing:
:glob:
scripts/*
tutorials/*
Then after building the project with make html, you have a link in your browser as seen here: tutorials/howToFoo.html
If you want to save the the file in a different folder:
index.rst
/tutorials/
/scripts/howToFoo.rst
Then the URL of your file howToFoo.rst changes depending on where it is saved:
scripts/howToFoo.html.
This is a problem because I don't want links to tutorials or scripts to break.
As the project aims to include many people, it will be very probable that there will be changes in the file structure in the future.
Now my question: Can you create a setup where you can move the files without having to write redirects to their new location, every time you move them?
For Cross Referencing inside of Sphinx this is solved for example with targets, explained here:
https://docs.readthedocs.io/en/stable/guides/cross-referencing-with-sphinx.html#automatically-label-sections
But this doesn't help me because the link in the browser still stays the same.
What I want is a link SomeNeverchangingLinkFor_howToFoo.html regardless of where the file howToFoo.rst is saved.
Assuming I don't know the job status that was supposed to generate some output files (in cloud store), can I assume that if some output files exist they contain all of the job's output?
Or it's possible that partial output is visible?
Thanks,
G
It is possible that only a subset of the files is visible, but the visible files are complete (cannot grow or change).
The filenames contain the total number of files (output-XXXXX-of-NNNNN), so once you have one file, you know how many more to expect.
I am working on product testing automation. I am using Jenkins to create a job that will first browse some file in a directory. But I got a problem with Conditional Step in File Exist condition. It is not working when I search *.job file, it only works with specific file name I put. Sorry I don't have enough reputation to post picture :)
Anyone can help me solve this issue. Thank so much for reading
I don't think File Exists supports more than one file, you could try the "File match condition". Definitions below:
File exists Run if the file exists.
The file is expanded using the Token Macro Plugin. If the file is
relative, then it is relative to the Base directory. If the file is
absolute, then it will be tested on the Node that contains the Base
directory (the master for $JENKINS_HOME and Artifacts dir).
Files match Run if one or more files match the selectors.
Separate multiple includes or excludes patterns with a comma. e.g.
Includes "target/site/**/*.html,target/site/images/" Excludes
"**/*.gif" If no includes pattern is configured, then the pattern
"**" will be used, which matches all files in the directory and all
sub directories. The includes and excludes are Ant Patterns.
I can use:
#+INCLUDE:
to include an org file in another org file, which allows me to assemble, say, a website from various org files. I'm exporting from the C-c C-e exporter in org-mode 7.5.
I could maintain a quite complex publication this way. This modular approach is quite common in, e.g. LaTeX and Texinfo publications.
However, links to images no longer work from the #+INCLUDEd org files. What seems to be happening is that the path to the images is taken as being from the org file that I am exporting from, rather than the actual org file that references the image.
The only ways I can see to resolve this are to:
use a flat file structure; or
make the image path from the referencing file (which I might not know in advance) rather than itself.
Neither of these is really sustainable.
How do I tell org to use the correct image path from its own relevant org file rather than the parent org file?
From what I know of the exporter, INCLUDE files are inserted into the document before export. Therefore the content is part of the document before it starts following paths to reach any links to files (images).
After a bit of testing you likely will need to use absolute file paths. Since you move between Windows and Linux your best bet would be to use a consistent scheme on both starting from your home directory.
Like that you can make the Org link:
[[~/path/to/image.jpg]], which will work on both systems (assuming you have set %HOME% on Windows).
Option 1 is potentially an alternative (although I agree it wouldn't be ideal at all), whereas the second option would have obvious pitfalls if you INCLUDE the file in more than one future document.
I'm familiar with tools like Deadweight for finding CSS not in use in your Rails app, but does anything exist for images? I'm sitting in a project with a massive directory of assets from working with a variety of designers and I'm trying to trim the fat in this project. It's especially a pain when moving assets to our CDN.
Any thoughts?
It depends greatly on the code using the images. It's always possible that a filename is computed (by concatenating two values or string substitution etc) so a simply grepping by filename isn't necessarily enough.
You could try running wget (probably already installed if you've got a linux machine, otherwise http://users.ugent.be/~bpuype/wget/ ) to mirror your whole site. Do this on the same machine or network if you can, it'll crawl your whole site and grab all the images
# mirror mysite.com accepting only jpg, png and gif files
wget -A jpg,png,gif --mirror www.mysite.com
Once you've done that, you're going to have a second copy of your site's hierarchy containing any images that are actively linked to by any page reachable by crawling your site. You can then backup your source image directory, and replace it with wget's copy. Next, monitor your log files for 404's pertaining to gif/jpg/png files. Hope that helps.
Finding unsed images should be easier than CSS.
Just find *.jpg *.png *gif with glob, put those filenames to dictionary or array and find those filenames againt html, css, js files, remove filename if found and you will get unused list, and move those images to another folder with same directory structure (It will be good for restoring for just in case)
Basically like this, and of course for the file names that encrypted/encoded/obcuscated will not work.
require "fileutils"
img=Dir.glob("**/*.jpg")+Dir.glob("**/*.png")+Dir.glob("**/*.gif")
data=Dir.glob("**/*.htm*")+Dir.glob("**/*.css")+Dir.glob("**/*.js")
puts img.length.to_s+" images found & "+data.length.to_s+" files found to search against"
content=""
data.each do |f|
content+=File.open(f, 'r').read
end
img.each do |m|
if not content=~ Regexp.new("\\b"+File.basename(m)+"\\b")
FileUtils.mkdir_p "../unused/"+File.dirname(m)
FileUtils.mv m,"../unused/"+m
puts "Image "+m+" moved to ../unused/"+File.dirname(m)+" folder"
end
end
PS: I used fileutils, because normal makedirs and mv are not works in my windows version of ruby
And I am not good at ruby, so please double check it before you use it.
Here is the sample results I ran in root folder of sample rails folder in my windows
---\ruby>ruby img_coverage.rb
5 images found & 12 files found to search against
Image depot/public/images/test.jpg moved to ../unused/depot/public/images folder
If your image URLs often come from many computed / concatenated strings and other stuff hard to track programmatically within your source code, and your application is in heavy use, you could try a soft "honeypot" approach like this:
Move all the assets to a different directory, e.g. /attic
Set up an empty /images directory (or what your asset directory is called)
Set up a .htaccess file (if you're on Apache of course) that, using the -f flag, redirects all requests to nonexistent image files to a script
The script copies the requested file from the /attic into the /images directory and displays it
The next request to that image will go directly to the image, because it exists now
After some time and sufficient usage, all needed images should have been copied to the assets directory.
It's a "soft" approach of course because a dialog / situation could have not been opened/entered/used by any user during that time (things like error message icons for example). But it will recognize all used files, no matter where they're requested from, and might help sort out much of the unneeded files.
If your file manager supports it, try sorting your images directory by the files' "last accessed" date. Files that haven't been accessed in a long time most likely aren't used any longer.
Along the same lines, you can also filter or grep through your web server's logs and make a list of the image files that it has served up in the last several months. Any images not in this list are likely unused.