I am processing an ancient .xml log file schema that cannot be changed it is a legacy thing and it would be too expensive to fix the issues.
The issues are:
The file root element is but it may never be closed -> so I cant just set file => { ... delimiter => "" } to read this xml object
The children element all have unique names ... ... -> the best idea I have had by now is to create 11 file plugins one with the delimiter = "" and other 10 that would match "0>" ... "9>" it should work but I am concerned with the should part.
last issue is that after the last line of the element there is not "\n" so I cant just use multiline to merge the lines neither
My question is someone familiar with a way to flush the content of the logstash's internal buffer in which incomplete lines are stored and then manipulate that data, this would solve my issue in much better and more performant way.
Many thanks.
Related
I am trying to use Log Parser Plugin with Jenkins. Following is my rule file which I have taken from the sample given on the link.
# match line starting with 'error', case-insensitive
error /(?i)^error/
# list of warnings here...
warning /[Ww]arning/
warning /WARNING/
# create a quick access link to lines in the report containing 'INFO'
info /INFO/
# each line containing 'BUILD' represents the start of a section for grouping errors and warnings found after the line.
# also creates a quick access link.
start /BUILD/
I still see following at the end of the Parsed Console Output page:
NOTE: Some bad parsing rules have been found:
Bad parsing rule: , Error:1
Bad parsing rule: , Error:1
Bad parsing rule: , Error:1
I did come across this, but dint help as I am not using space anywhere.
Can someone help me resolving this issue?
It appears you have extra white-space somewhere in the file that the plugin is interpreting as you attempting to define a rule. Maybe try running it with the empty lines removed. That plugin has given me quite a bit of trouble as well, it's not very well documented (as is the case with many Jenkins plugins).
I had tried no spaces in the pattern, but that did not work. Turns out that the Parsing Rules files does not support empty lines in it. Once I removed the empty lines, I did not get this "Bad parsing rule: , Error:1".
I think since the line is empty - it doesn't echo any rule after the first colon. Would have been nice it the line number was reported where the problem is.
I posted the same to this thread too - Log parsing rules in Jenkins
Hopefully, it helps out other folks who may be wondering what is causing this.
Unfortunately this question may be a bit vague, I have a problem that I am finding difficult to describe, it is intermittent and I cannot reproduce it myself, I am just hoping that someone else has seen something like it before.
My application has quite a lot of text and ini files that get written when it closes down. Typically this would be in response to a Close event, but may also be triggered by a WM_ENDSESSION. Unfortunately at the moment I am not sure if both or only one of these events can result in the problem I am about to describe, because I have been unable to reproduce this problem myself.
The issue I have is that for some users some of the text and ini files end up being written as NULLs. The file sizes end up looking about right, but instead of text, every character is written as a x00. So instead of 500 bytes of regular ASCII text I end up with 500 x00's. I also have an application log file that can sometimes end up with nulls written to it also. However the logging of x00's to the log file does not necessarily correspond to the exact same time as x00's were written to the config files.
For my files I am using TmemIniFile or TstringList which means that ultimately a Tstrings.SaveToFile is being called for all of my config files.
sl:=TstringList.Create;
try
SourceList.GetSpecificSubset(sl);
AppLogLogLine('Commands: Saving Always Available list. List has '+inttostr(sl.Count)+' commands.');
sl.SaveToFile(fn);
finally
sl.Free;
end;
But then I also have instance where I would already have a TstringList in memory and I just call SaveToFile on it. For TmemIniFile the structure would look similar to above. In some instances I may have an outer loop to write multiple lists. Some of those will result in files being written correctly, some will be full of 00's.
EDIT: GetSpecificSubset is simply a function that will populate "sl" with a list of command names. I have "GetAllUsersCommands", "GetHiddenCommands", "GetAlwaysVisibleCommands" etc. Note that my log file also writes this kind of thing, as a check for how big those lists are:
16/10/2013 11:17:49 AM: Commands: Saving Any User list. List has 8 commands.
16/10/2013 11:17:49 AM: Commands: Saving Always Visible list. List has 17 commands.
16/10/2013 11:17:49 AM: Commands: Saving Always Hidden list. List has 2 commands.
I accidentally left the logging line out of the code above. So this log line is the last thing written before calling Tstrings.SaveToFile, and at that point it thinks it has data. Even if somehow each line of text were NULLs, I would still expect to see x13x10 in the files, but that is not happening.
Here's a screen cap from a HEX editor:
EDIT 2: I just realised I left off a very important piece of information. This is only intermittent. It works 99% of the time. When saving files at shutdown it might not even be all files. Even if I have a loop saving multiple similar files, some may work fine and others may fail.
Bet you didn't see this coming? ;)
So, a project of mine requires that I specifically read and make sense out of .htaccess files.
Sadly, searching on Google only yields the infinite woes of people trying to get their own .htaccess to work (sorry, couldn't resist the comment).
Anyway, I'm a bit scared of trying to get this thing out of open-source projects that use it. See, in the past few weeks, I ended up wasting a lot of time trying to fix my issues with this strategy, only to find out that I did better to read RFCs & specs and build the thing my way.
So, if you know about a library, or any (hopefully clean!) code that does this, please do share. In the mean time, if you know about any articles about .htaccess file format, I'm sure they'll be very handy. Thanks.
NB: I'm pretty much multilingual and could make use of any codebase, even though the end code will be Delphi. I know I'm asking too much, but I'd love to see less of C++. Just think of my mental health before sharing C++ code. :)
Edit: Well, I think I'm just going to do this manually myself. The file structure seems to be:
directive arg1 arg2 argN
<begin directive section>
</end directive section>
# single line comment
.htaccess grammar is actually the exact same as the Apache configuration itself, and example parsers do exist for it.
If you're looking to write your own, you are mostly correct on the format. Remember, section tags can be nested and can have parameters (like <Location />)
English method of parsing:
For each line in the file:
Strip whitespace from beginning and end of line.
If the line starts with a '#':
Parse it as a comment (or skip it)
Else, If the line starts with a '<':
If the next character is a '/', the line is a closing tag:
Seek to the next '>' to get the tag name, and pop it from the tag stack.
Else, the line is an opening tag:
Seek to the next '>' for the tag name.
If the tag, trimmed, contains whitespace:
Split on the first whitespace. The right side is params, left is the tag.
(IfModule, Location, etc use this)
Push the tag name to the tag stack.
Else, the line is a directive:
Split the line on whitespace. This is the directive and params.
Just add quote handling and you're set.
I'm having a slight annoyance with the javascript_I18n plugin, which generates js-friendly versions of the I18n translation tables, so that you can localise your javascript. It all works fine, but it works by calling to_json on each locale's translations hash and outputting the results into a file. When you call to_json on a hash the resulting string is all a single lne, which means you end up with files with an enormous line at the end.
This in turn is stopping git from being able to merge any changes, because the merge works on a line by line basis and can't deal with a single massive line with changes in the middle somewhere. It's not a massive problem because i can always just re-generate the js-friendly translation file with a rake task that comes with the plugin (which replaces the mid-merge file with a brand new one which i can just commit in), but it's just a bit annoying. It occurred to me that if the json was output on different lines, instead of all the same line, then it wouldn't be a problem, and it wouldn't even make the file that much bigger, just inserting two characters (\n) per line.
Before i try and hack the resulting string with a gsub to split it onto seperate lines, is there a nicer way to call to_json on a hash and output the results onto seperate lines? Or a nicer way to solve this problem generally? (i can't find much useful in the documentation for javascript_I18n).
Grateful for any advice - max
Not to answer you question, but to give a suggestion:
It would probably be easier to just ignore all the generated js translation files.
This is because even if you separate the translation into multiple lines, there are still chances of merge conflicts, which you probably already resolved once in your yaml file.
I would set it up this way:
1). gitignore all the js translation files.
2). In ActionController, add a before filter to generate the js files automatically on each load in development mode only.
3). Tweak your deploy.rb file to generate the js files after code update.
No more merge conflicts! (at least for js translation files) :D
Aaron Qian
Issues reading CSV file using OLEDB when filename have period.
I have a code in C# that reads CSV File using OleDBProvider. It works perfect with filenames in regular format such as Budget.csv but failed when i renamed the file into Budget.DKK.csv or Budget.USD.csv
I throws this exception:
he Microsoft Jet database engine could not find the object 'Budget.DKK.csv'. Make sure the object exists and that you spell its name and the path name correctly.
I have no idea so far why is this happening.
If this thread is to be believed, then it is a known problem that won't be fixed. It mentions a work-around that allows the name to be forced into the old style 8.3 format.
And just as a random suggestion if you haven't tried it, maybe delimit the filename with brackets [filename.stuff.txt]. I doubt it is that simple, though.