Jenkins Pipeline: How to write UTF-8 files with writeFile? - jenkins

I'm hoping this is not a bug and that I'm just doing something wrong. I have a Jenkins (v2.19.1) Pipeline job and in it's groovy script, I need to search and replace some text in an existing text file, on a Windows node.
I've used fart.exe and powershell to do the search and replace, but I would really like to do this with just the groovy in Jenkins and eliminate dependency on fart/powershell/etc. and make this code more reusable on both linux and windows nodes.
After much googling and trying various approaches, the closest I got was to use readFile and writeFile. However, I've not been able to get writeFile to create a UTF-8 file. It creates an ANSI file even when I specify UTF-8 (assuming I'm doing it correctly).
Here's what I have so far...
def fileContents = readFile file: "test.txt", encoding: "UTF-8"
fileContents = fileContents.replace("hello", "world")
echo fileContents
writeFile file: "test.txt", text: fileContents, encoding: "UTF-8"
I've confirmed with multiple text editors that the test.txt file is UTF-8 when I start, and ANSI after the writeFile line. I've tried all combinations of including/not-including the encoding property and "utf-8" vs "UTF-8". But in all cases, the file is written out as ANSI (as reported by both Notepad++ and VS Code). Also, a question mark (HEX 3F) is added as the very first character of the file.
The echo line does not show the extra 3F character, so it seems the issue is in the writeFile line.

Your code looks correct. See that ANSI and UTF-8 are the same if you are using just non accented chars and numbers. Try to have some accented letters (áéíóúç) in your file that the editor will probably recognize it as an UTF-8 file.
You must use the Jenkins pipeline writeFile function. This is a special Jenkins method to write files inside your workspace. The default Java File objects won't work.
To specify the encoding, you must use named parameters. Here is a an example:
writeFile(file: "filename.txt", text: "áéíóú", encoding: "UTF-8")
This will create at the root of your workspace, a file named filename.txt with "áéíóú" as content and encoded as UTF-8.
BTW, if you have full control of the file you must search and replace, consider using Groovy's builtin SimpleTemplateEngine

Related

Using Umlaut or special characters in ibm-doors from batch

We have a link module that looks something like this:
const string lMod = "/project/_admin/somethingÜ" // Umlaut
We later use the linkMod like this to loop through the outlinks:
for a in obj->lMod do {}
But this only works when executing directly from DOORS and not from a batch script since it for some reason doesn't recognize the Umlaut causing the inside of the loop to never to be run; exchanging lMod with "*" works and also shows the objects linked to by the lMod.
We are already using UTF-8 encoding for the file:
pragma encoding, "UTF-8"
Any solutions are welcome.
Encode the file as UTF-8 in Notepad++ by going to Encoding > Convert to UTF-8. (Make sure it's not already set to UTF-8 before you do it).

Cannot replace line after string match in jenkins pipeline using variables

I need to replace a line in a file. If the line starts with the term "url", I need to replace the value.
file.txt --
...
url : http://www.google.com
..
I need to change this value to
url : http://www.facebook.com
I tried the following code but it did not work -
FACEBOOK_URL = "http://www.facebook.com"
sh("sed -i \\"s?^url.*\\$?url: ${FACEBOOK_URL}?\\" file.txt")
I'm using a Jenkins Pipeline. I need to replace the string using a variable.
Jenkins 2 Pipeline builds use Groovy and it is very easy to read the file using readfile and then we can do the changes
def text = readFile "file.txt"
text.replaceAll("url.*", "url: ${FACEBOOK_URL}")
The above code will help in replacing the text in the file, if you want to write the content to file, you can use writeFile
You can use this for replacing a string in a file in Jenkins 2 Pipeline builds:
def text = readFile file: "file.txt"
text = text.replaceAll("%version%", "${VERSION}")
writeFile file: "file.txt", text: text

How do I send output from the terminal to a text file in Visual Studio Code with the proper encoding?

I'm having issues sending output containing Unicode box-drawing characters to a text file.
The string ┌───top───┐ prints to the terminal fine using the print command. It also renders properly if I open and write directly to a file in my code.
However, if I pipe the terminal output into a text file using
<run command> > out.txt, I get the result
ΓöîΓöÇΓöÇΓöÇtopΓöÇΓöÇΓöÇΓöÉ
Everything else prints fine, but why doesn't it handle certain Unicode characters?
Is there a quick fix for this?

Native2Ascii task not working

I'm trying to use the native2ascii ant task but it seems that is not doing anything. Here's my ant task:
<target name="-pre-init">
<native2ascii src="src/com/bluecubs/xinco/messages" dest="src/com/bluecubs/xinco/messages/test"
includes="**/_*.properties"/>
<copy todir="src/com/bluecubs/xinco/messages">
<fileset dir="src/com/bluecubs/xinco/messages/test"/>
</copy>
<delete dir="src/com/bluecubs/xinco/messages/test" />
</target>
I did the copy part to see if it was an overwriting issue but the files come out exactly the same.
This is the output I get when running the task:
Converting 12 files from Z:\Netbeans\Xinco\2.01.xx\Xinco\src\com\bluecubs\xinco\messages to Z:\Netbeans\Xinco\2.01.xx\Xinco\src\com\bluecubs\xinco\messages\test
Copying 12 files to Z:\Netbeans\Xinco\2.01.xx\Xinco\src\com\bluecubs\xinco\messages
Deleting directory Z:\Netbeans\Xinco\2.01.xx\Xinco\src\com\bluecubs\xinco\messages\test
Edit:
Additional information:
OS: Windows 7 (but answer should work on any OD)
File encoding: Western (ISO-8859-1) obtained with this article.
Files location
Any idea?
native2ascii converts native characters like áéí to escaped unicode sequences. It means that á will be \u00e1, é -> \u00e9 and í -> \u00ed. After running native2ascii your files will be standard ASCII files which are more portable.
native2ascii does not touch the characters which are already in the escaped unicode form. Your properties files are already in escaped unicode form so it does not change anything. For example _XincoMessages_cz.properties contains this line:
general.accessrights=opr\u00E1vnen\u00ED k pr\u00EDstupu
It's escaped unicode. The nonescaped unicode form is this:
general.accessrights=oprávnení k prístupu
Wordpad vs. Netbans: When you open the properties files with Wordpad it opens it as a simple text file and shows \u00e1 as \u00e1. It does not convert it back to á. Netbeans does this conversion and you see the 'á' character. Furthermore, it writes it back to the disk as \u00e1 (!) when you save the file. To see the raw files use for example a Total or Double Commander which doesn't do any converting. (Note that Netbeans does this conversion just for properties files.)
If you put for example an á character to your _XincoMessages_cz.properties file it will be changed to \u00e1 if your run your ant task. Of course now don't use Netbeans for the editing, a simple notepad will do.
Loading properties files in java converts the escaped unicode characters to real unicode characters. An example:
final Reader inStream = new FileReader("..../_XincoMessages_cz.properties");
final Properties properties = new Properties();
properties.load(inStream);
System.out.println(properties.getProperty("general.accessrights"));
It prints:
oprávnení k prístupu
The ASCII/escaped unicode form in properites files is usually handled well by java applications. Finally, I think your properties files are good in their current format.
It ended being a view issue. Looking the files in a raw editor (i.e. Wordpad) showed that the files were already converted by the task. Viewing them from NetBeans shows them the same.

Character Encoding issue in Rails v3/Ruby 1.9.2

I get this error sometimes "invalid byte sequence in UTF-8" when I read contents from a file. Note - this only happens when there are some special characters in the string. I have tried opening the file without "r:UTF-8", but still get the same error.
open(file, "r:UTF-8").each_line { |line| puts line.strip(",") } # line.strip generates the error
Contents of the file:
# encoding: UTF-8
290919,"SE","26","Sk‰l","",59.4500,17.9500,, # this errors out
290956,"CZ","45","HornÌ Bradlo","",49.8000,15.7500,, # this errors out
290958,"NO","02","Svaland","",58.4000,8.0500,, # this works
This is the CSV file I got from outside and I am trying to import it into my DB, it did not come with "# encoding: UTF-8" at the top, but I added this since I read somewhere it will fix this problem, but it did not. :(
Environment:
Rails v3.0.3
ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-darwin10.5.0]
Ruby has a notion of an external encoding and internal encoding for each file. This allows you to work with a file in UTF-8 in your source, even when the file is stored in a more esoteric format. If your default external encoding is UTF-8 (which it is if you're on Mac OS X), all of your file I/O is going to be in UTF-8 as well. You can check this using File.open('file').external_encoding. What you're doing when you opening your file and passing "r:UTF-8" is forcing the same external encoding that Ruby is using by default.
Chances are, your source document isn't in UTF-8 and those non-ascii characters aren't mapping cleanly to UTF-8 (if they were, you would either get the correct characters and no error, and if they mapped by incorrectly, you would get incorrect characters and no error). What you should do is try to determine the encoding of the source document, then have Ruby transcode the document on read, like so:
File.open(file, "r:windows-1251:utf-8").each_line { |line| puts line.strip(",") }
If you need help determining the encoding of the source, give this Python library a whirl. It's based on the automatic charset detection fallback that was in Seamonkey/Mozilla (and is possibly still in Firefox).
If you want to change your file encoding, you can use gem 'charlock holmes'
https://github.com/brianmario/charlock_holmes
$require 'charlock_holmes/string'
content = File.read('test2.txt')
if !content.is_utf8?
detection = CharlockHolmes::EncodingDetector.detect(content)
utf8_encoded_content = CharlockHolmes::Converter.convert content, detection[:encoding], 'UTF-8'
end
Then you can save your new content in a temp file and overwrite your original file.
Hope this help.

Resources