Good afternoon,
I am working with Java Saxon 9.8.0.4. I would like to use EXPath File Module function "file:list" with its third "pattern" parameter. But I am in doubt, which style of pattern is supported.
I read both Saxon documentation and EXPath documentation. But I do not know, which patterns are supported in Saxon 9.8.0.4. It would be great to support regular expression, but I understand it is overkill for most users. I tried several blind tests, but just * and ? wildchars works for me as defined in EXPath documentation.
Yes, I can quite easily do regexp postprocessing in for-each, but to know more about list function could help.
Thank You in advance for Your help, Stepan
P.S: My use-case is to get all files without extension ("test" and not "test.txt") recursively from large and deep directory structure and process all of matching files with XSL-T 3.0. Most of such files have identical fileName and thus I can not do "copy to one folder" pre-processing for Saxon's -s:directory -o:directory one time invocation and invocation of Java (Saxon) for each file is of cource terrible time overhead. So I would like to read all matching files into sequence and process each item of such sequence using for-each (files are text ones and I read them using unparsed-text). And no, GAWK is not solution, as I have all transformation infrastructure from XML to SQL already in XSL-T, because 95 % of files are XMLs.
--ADDED code and explanation below:
Example of my test files.
XML file "a.xml":
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="a.xsl"?>
<root/>
XSL-T file "a.xsl":
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:saxon="http://saxon.sf.net/"
xmlns:expathFile="http://expath.org/ns/file"
exclude-result-prefixes="xs saxon"
version="3.0">
<xsl:output method="text" />
<xsl:template match="/root">
<xsl:variable name="list" select="expathFile:list('C:\temp\temp\test\', false(), '^.*$')"/>
<xsl:for-each select="$list">
<xsl:value-of select="."/>
</xsl:for-each>
</xsl:template>
My folder "C:\temp\temp\test\" contains 6 test files: "a.txt", "b.txt", "c.txt", "e", "f", "g".
But after testing of online Java RegExp tester on "http://www.regexplanet.com/advanced/java/index.html" I have found, that the problem is solely on my side, because Java regular expression behaves little different than PCRE (Perl), sed, gawk regular expressions. So it is my fault and I need to learn Java regular expression.
Saxon uses the same code for this pattern as for the filter in select="pattern" in collection URIs, which is described at http://www.saxonica.com/documentation/index.html#!sourcedocs/collections
Extracting the relevant details:
The pattern used in the select parameter can use glob-like syntax, for
example *.xml selects all files with extension "xml". More generally,
the pattern is converted to a regular expression by prepending "^",
appending "$", replacing "." by "\.", "*" by ".*", and "?" by ".?",
and it is then used to match the file names appearing in the directory
using the Java regular expression rules. So, for example, you can
write ?select=*.(xml|xhtml) to match files with either of these two
file extensions. Note however, that special characters used in the URL
(that is, characters such as backslash and curly braces that are not
allowed in the query part of a URI) must be escaped using the %HH
convention. For example, vertical bar needs to be written as %7C. This
escaping can be achieved using the encode-for-uri() function.
Note that Saxon's collection() function now also supports match=pattern in the URI, where the pattern is a standard XPath 3.1 regular expression.
I wrote this grep command:
grep -- "^[0-9a-zA-Z\.-]\+$" file.txt
To get all lines containing only numbers, letters and dashes (legal domains).
This is the result of diff on both files
1,3c1,3
< test.xcom
< hi-th6ere.co.k
< 54
---
> test.xcom
> hi-th6ere.co.k
> 54
I wrote a file with some domains to test and it works great!
But, when I download a file (with the same content!) from the web, and then run this command, grep doesn't return anything.
I've tried to set full permissions on this file, but it still doesn't work.
Any ideas?
Thanks,
What makes you think the file content as the same as the one you've tested?
You can run 'diff filename1 filename2' to see if there are any differences between the two files.
It could be the the file you're downloading is in unicode format, so in a web browser it looks to be have the same content as the file you've tested, but the binary content of the file itself is different.
I'm trying to use the native2ascii ant task but it seems that is not doing anything. Here's my ant task:
<target name="-pre-init">
<native2ascii src="src/com/bluecubs/xinco/messages" dest="src/com/bluecubs/xinco/messages/test"
includes="**/_*.properties"/>
<copy todir="src/com/bluecubs/xinco/messages">
<fileset dir="src/com/bluecubs/xinco/messages/test"/>
</copy>
<delete dir="src/com/bluecubs/xinco/messages/test" />
</target>
I did the copy part to see if it was an overwriting issue but the files come out exactly the same.
This is the output I get when running the task:
Converting 12 files from Z:\Netbeans\Xinco\2.01.xx\Xinco\src\com\bluecubs\xinco\messages to Z:\Netbeans\Xinco\2.01.xx\Xinco\src\com\bluecubs\xinco\messages\test
Copying 12 files to Z:\Netbeans\Xinco\2.01.xx\Xinco\src\com\bluecubs\xinco\messages
Deleting directory Z:\Netbeans\Xinco\2.01.xx\Xinco\src\com\bluecubs\xinco\messages\test
Edit:
Additional information:
OS: Windows 7 (but answer should work on any OD)
File encoding: Western (ISO-8859-1) obtained with this article.
Files location
Any idea?
native2ascii converts native characters like áéí to escaped unicode sequences. It means that á will be \u00e1, é -> \u00e9 and í -> \u00ed. After running native2ascii your files will be standard ASCII files which are more portable.
native2ascii does not touch the characters which are already in the escaped unicode form. Your properties files are already in escaped unicode form so it does not change anything. For example _XincoMessages_cz.properties contains this line:
general.accessrights=opr\u00E1vnen\u00ED k pr\u00EDstupu
It's escaped unicode. The nonescaped unicode form is this:
general.accessrights=oprávnení k prístupu
Wordpad vs. Netbans: When you open the properties files with Wordpad it opens it as a simple text file and shows \u00e1 as \u00e1. It does not convert it back to á. Netbeans does this conversion and you see the 'á' character. Furthermore, it writes it back to the disk as \u00e1 (!) when you save the file. To see the raw files use for example a Total or Double Commander which doesn't do any converting. (Note that Netbeans does this conversion just for properties files.)
If you put for example an á character to your _XincoMessages_cz.properties file it will be changed to \u00e1 if your run your ant task. Of course now don't use Netbeans for the editing, a simple notepad will do.
Loading properties files in java converts the escaped unicode characters to real unicode characters. An example:
final Reader inStream = new FileReader("..../_XincoMessages_cz.properties");
final Properties properties = new Properties();
properties.load(inStream);
System.out.println(properties.getProperty("general.accessrights"));
It prints:
oprávnení k prístupu
The ASCII/escaped unicode form in properites files is usually handled well by java applications. Finally, I think your properties files are good in their current format.
It ended being a view issue. Looking the files in a raw editor (i.e. Wordpad) showed that the files were already converted by the task. Viewing them from NetBeans shows them the same.
My iOS application uses a number of third party components licensed under Apache 2.0 and similar licenses, which requires me to include various bits of text, this kind of thing:
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
There seems to be a reasonable precedent for putting this information under a 'License' subentry in settings bundle (on the ipad facebook, pages, keynote, numbers and wikipanion all seem to do this).
I'm struggling a bit to actually achieve the same though; I seem to need to split the text up line by line and enter into xcode a line at a time (and xcode4 seems to have a crashing problem when editing the plists).
It seems like the kind of thing that there's almost certainly a somewhere script to do, or some simple way to do it that I've missed.
I think I've now managed to solve all the problems I was running into.
It seems to be best to use group element titles to hold the licenses (this is what Apple do in the iWork apps). There is however a limit on the length of these (and I've not yet discovered exactly what the limit is), so you need to break each license file into multiple strings.
You can create a line break within these by include a literal carriage return (ie. otherwise known as ^M, \r or 0x0A)
Make sure not to include any literal "s mid-text. If you do, some or all of the strings in the file will get silently ignored.
I've got a convenience script I use to help generate the .plist and .strings file, shown below.
To use it:
Create a 'licenses' directory under your project
Put script into that directory
Put each license into that directory, one per file, with filenames that end .license
Perform any necessary reformatting on the licenses. (eg. remove extra spaces at the beginning of lines, ensure that there are no line breaks mid-paragraph). There should be a blank line in-between each paragraph
Change to licenses directory & run the script
Edit your settings bundle Root.plist to include a child section called 'Acknowledgements'
Here's the script:
#!/usr/bin/perl -w
use strict;
my $out = "../Settings.bundle/en.lproj/Acknowledgements.strings";
my $plistout = "../Settings.bundle/Acknowledgements.plist";
unlink $out;
open(my $outfh, '>', $out) or die $!;
open(my $plistfh, '>', $plistout) or die $!;
print $plistfh <<'EOD';
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>StringsTable</key>
<string>Acknowledgements</string>
<key>PreferenceSpecifiers</key>
<array>
EOD
for my $i (sort glob("*.license"))
{
my $value=`cat $i`;
$value =~ s/\r//g;
$value =~ s/\n/\r/g;
$value =~ s/[ \t]+\r/\r/g;
$value =~ s/\"/\'/g;
my $key=$i;
$key =~ s/\.license$//;
my $cnt = 1;
my $keynum = $key;
for my $str (split /\r\r/, $value)
{
print $plistfh <<"EOD";
<dict>
<key>Type</key>
<string>PSGroupSpecifier</string>
<key>Title</key>
<string>$keynum</string>
</dict>
EOD
print $outfh "\"$keynum\" = \"$str\";\n";
$keynum = $key.(++$cnt);
}
}
print $plistfh <<'EOD';
</array>
</dict>
</plist>
EOD
close($outfh);
close($plistfh);
Setting up your Settings.bundle
If you haven't created a Settings.bundle, go to File --> New --> New File...
Under the Resource section, find the Settings Bundle. Use the default name and save it to the root of your project.
Expand the Settings.bundle group and select Root.plist. You will need to add a new section where its key will be Preference Items of type Array. Add the following information:
The Filename key points to the plist that was created by this script. You can change the title to what ever you want.
Execute Script At Build Time
Also, if you want this script to run whenever you build your project, you can add a build phase to your target:
Go to your project file
Select the target
Click the Build Phases tab
In the lower right corner of that pane, click on 'Add Build Phase'
Select 'Add Run Script'
Drag and drop your perl script into the section for your script. Modify to look something like this:
cd $SRCROOT/licenses ($SRCROOT points to the root of your project)
./yourScriptName.pl
After you have finished that, you can drag the Run Script build phase sooner in the build process. You'll want to move it up before Compile Sources so that the updates to your Settings Bundle get compiled and copied over.
Update for iOS 7: iOS 7 seems to handle the "Title" key different and is messing up the rendered text. To fix that the generated Acknowledgements.plist needs to use the "FooterText" key instead of "Title". This how to change the script:
for my $str (split /\r\r/, $value)
{
print $plistfh <<"EOD";
<dict>
<key>Type</key>
<string>PSGroupSpecifier</string>
<key>FooterText</key> # <= here is the change
<string>$keynum</string>
</dict>
EOD
print $outfh "\"$keynum\" = \"$str\";\n";
$keynum = $key.(++$cnt);
}
Here's the same solution that #JosephH provided (without translations), but done in Python for anyone who prefers python over perl
import os
import sys
import plistlib
from copy import deepcopy
os.chdir(sys.path[0])
plist = {'PreferenceSpecifiers': [], 'StringsTable': 'Acknowledgements'}
base_group = {'Type': 'PSGroupSpecifier', 'FooterText': '', 'Title': ''}
for filename in os.listdir("."):
if filename.endswith(".license"):
current_file = open(filename, 'r')
group = deepcopy(base_group)
title = filename.split(".license")[0]
group['Title'] = title
group['FooterText'] = current_file.read()
plist['PreferenceSpecifiers'].append(group)
plistlib.writePlist(
plist,
"../Settings.bundle/Acknowledgements.plist"
)
As an alternative, for those using CocoaPods, it will generate an 'Acknowledgements' plist for each target specified in your Podfile which contains the License details for each Pod used in that target (assuming details have been specified in the Pod spec). The property list file that can be added to the iOS settings bundle.
There's also projects under way to allow this data to be converted and displayed within the app instead:
https://github.com/CocoaPods/cocoapods-install-metadata
https://github.com/cocoapods/CPDAcknowledgements
I thought I'd throw my iteration on Sean's awesome python code in the mix. The main difference is that it takes an input directory and then recursively searches it for LICENSE files. It derives the title value from the parent directory of the LICENSE file, so it plays well with cocoapods.
The motivation was to create a build script to automatically keep the legal section of my app up to date as I add or remove pods. It also does some other things like remove forced newlines from licenses so the paragraphs look a bit better on the devices.
https://github.com/carloe/LicenseGenerator-iOS
I made a script in Ruby inspiered by #JosephH script.
This version will, in my own opinion, better represent the individual open source projects.
Wisit iOS-AcknowledgementGenerator to download the script and sample project.
This is what acknowledgements will look like in your App:
This is an addendum to JosephH's answer. (I don't have the rep to comment)
I had to move
<key>StringsTable</key>
<string>Acknowledgements</string>
down to above the last </dict> in the Perl script.
Before this modification, the Acknowledgements Section in the App was empty and XCode couldn't read the resulting Acknowledgements.plist. ( "The data couldn’t be read because it isn’t in the correct format.")
(XCode 6.3.2 iOS 8.3)
The Python script from Sean in this thread works. But there a couple of basic things to know.
in Xcode, right click on the top of the Project Navigator tree, on the name of your project, and add a New Group. This puts a new folder in your project.
Add Sean's script there and make sure to save it as: Acknowledgements.py.
Make sure you have Python installed on your system. I'm using a Mac.
Add a first license file to the folder you created in 1. Make it simple like just having one word in the file, say: Testing. Save it in the folder as Test1.license.
Set up your Settings.bundle as per JosephH above.
Use your Terminal app to CD to the folder you created in 1.
Run the script. Type: python Acknowledgements.py. If you get no errors it will return right back to the Terminal prompt. Do all of this before adding any run script to the Build.
Build and run your app.
Double tap on the iPhone home button and kill Settings. It doesn't often pick up the Settings change for your app until Settings restarts.
After restarting Settings, go to your app and look to see if it worked.
If that all worked, slowly add more license files but run the script each time. You can get errors running the script because of certain characters in the file so the easy way to debug is to add a file, run the script, see if it worked and proceed. Else, edit any special characters out of the .license file.
I did not get the Run Build Script work per the instructions above. But this process works fine if you are not changing the .license files that often.
Ack Ack: Acknowledgement Plist Generator
A while back I've created a Python script that scans for license files and creates a nice Acknowledgements plist that you can use in your Settings.plist. It does a lot of the work for you.
https://github.com/Building42/AckAck
Features
Detects Carthage and CocoaPods folders
Detects existing plists for custom licenses
Cleans up the license texts by removing unnecessary new lines and line breaks
Provides many customization options (see --help for details)
Supports both Python v2 and v3
Install
wget https://raw.githubusercontent.com/Building42/AckAck/master/ackack.py
chmod +x ackack.py
Run
./ackack.py
Screenshot
If you have suggestions for improvements, feel free to post an issue or pull request on GitHub!
Aknowlist is a strong CocoaPod candidate that is actively maintained at the time of this writing. It automates this licensing if you are okay with housing the licenses in your app rather than the settings bundle. It worked great for the project I was working on.
I had to modify sean's script for modern python3:
import os
import sys
import plistlib
from copy import deepcopy
os.chdir(sys.path[0])
plist = {'PreferenceSpecifiers': [], 'StringsTable': 'Acknowledgements'}
base_group = {'Type': 'PSGroupSpecifier', 'FooterText': '', 'Title': ''}
for filename in os.listdir("."):
if filename.endswith(".license"):
with open(filename, 'r') as current_file:
group = deepcopy(base_group)
title = filename.split(".license")[0]
group['Title'] = title
group['FooterText'] = current_file.read()
plist['PreferenceSpecifiers'].append(group)
with open("Acknowledgements.plist", "wb") as f:
plistlib.dump(plist, f)
Wait, license notation is not a setting.
Edit:
I think license notice is not a setting. I think it is irrational to expect users who want to check the license notice to open the settings app. Therefore, I thought we should create a page for the license notice in the appropriate place in the app.
We're generating PartCover reports via the command line tool along with our CruiseControl.Net unit tests. This generates an xml file that displays the results nicely on the cruisecontrol dashboard. The xslt transforms that are included only show you the percentage of coverage in an individual class. We want to know exactly what lines are not being covered. The problem ist when we open the report in the PartCover browser and double click a method it doesn't show us our cs files. I know the PartCover browser is capable of showing you the files because of the following.
Here's a screenshot of PartCover browser with the lines of code showing: http://kjkpub.s3.amazonaws.com/blog/img/partcover-browse.png.
The information looks like it should be available to the browser because the report contains this:
<Method name="get_DeviceType" sig="Cathexis.IDBlue.DeviceType ()" bodysize="19" flags="0" iflags="0">
<pt visit="2" pos="0" len="1" fid="82" sl="35" sc="13" el="35" ec="14" />
<pt visit="2" pos="1" len="4" fid="82" sl="36" sc="17" el="36" ec="39" />
<pt visit="2" pos="5" len="2" fid="82" sl="37" sc="13" el="37" ec="14" />
</Method>
and this:
<File id="66" url="D:\sandbox\idblue\idblue\trunk\software\code\driver\dotnet\Common\AsyncEventQueue.cs" />
All I want to be able to do is view what lines of code are not being covered in my test cases without having to figure out what the xml above is trying to tell me.
Thanks to anyone in advance who replies.
I figured out why the cs files were not displaying. The paths were incorrect in the xml file because our test project was being built on a different machine than the one partcover was on. (partcover must generate the .cs file paths from pdb files maybe?) Once I search and replaced the file switching the base directory of our subversion location to the one on the other machine all was well.