There appears to be a years-long-standing issue with large ( longer than 16KB) messages getting split into parts and appearing on Kibana in multiple lines. Such long messages typically include Java exception stack traces. The splitting makes parsing and therefore indexing impossible and messes things up completely for developers who need to read the logs.
By "message" I'm referring to the field with the label "message" that appears as part of the log entry that, of course, starts with the "timestamp" field.
As much as I have searched I have not found a filter that can concatenate those parts and make them appear as a whole on a single log entry, where the JSON block can be properly parsed and indexed. I have tried a few filters of my own, with little success.
Please help if you are aware of a solution.
Thanks
Related
Over the last several weeks, my Rails 6.1 app has been raising hundreds of exceptions like the following:
An ActionDispatch::Http::Parameters::ParseError occurred in regdevs#create:
859: unexpected token at
'utf8=%E2%9C%93&authenticity_token=fnrg63qtelQvkQx9NQx8SaZIp7mR500toEiWdaJe91
%2FOzsEvRbGD4Ow6NNADCtjw7H3EkDvFZVGP4gQkGZceEw%3D%3D&user%5Bemail%5D=jrmills91
%40hotmail.com&user%5Bfirst_name%5D=wJAhzNnPWbqMdfH&user%5Blast_name%5D=tKdp
IVXJlneZbiuA&user%5Bnickname%5D=lBdbyYgiaHt&user%5Bpassword%5D=0SgLpVuf5Wqs
%21&user%5Bpassword_confirmation%5D=0SgLpVuf5Wqs%21&user%5Bterms_of_service
%5D=1&commit=Complete+registration&g-recaptcha-response=dzghTWcBpmI'
The error occurs when "someone" tries to register for a new account (via Devise) with bogus credentials. It clearly looks like some sort of attack.
I have two questions:
How can I find out more about this specific 859 unexpected token error?
Should I be concerned? The recaptcha guard seems effective in blocking account creation, but I wonder if I am missing something
I partially decoded the response by hand mostly just to see what it looked like.
utf8=%E2%9C%93
authenticity_token=fnrg63qtelQvkQx9NQx8SaZIp7mR500toEiWdaJe91/OzsEvRbGD4Ow6NNADCtjw7H3EkDvFZVGP4gQkGZceEw==
user[email]=jrmills91#hotmail.com
user[first_name]=wJAhzNnPWbqMdfH
user[last_name]=tKdpIVXJlneZbiuA
user[nickname]=lBdbyYgiaHt
user[password]=0SgLpVuf5Wqs!
user[password_confirmation]=0SgLpVuf5Wqs!
user[terms_of_service]=1
commit=Complete+registration
g-recaptcha-response=dzghTWcBpmI
The first line is utf8= and then that is the unicode for checkmark. Perhaps that is part of the captcha?
The user names etc clearly look bogus. I bet jrmills is getting lots of weird email. :-)
The ParseError is defined in parameters.rb and you can see the call that created it inside parse_formatted_parameters. The only place that calls it is from request.rb.
From the comments in the code (and what I would expect), if the authenticity_token doesn't match, an ActionController::InvalidAuthenticityToken should be raised. So, the parse error (I'm guessing) is lower down but I can't spot why the JSON or Ruby code would be complaining. I think I would have written that piece of code differently: captured the error and included the original error message with the new error -- but... such is life.
I would attempt to track down the IP address where these are being sent from and block that IP address if possible or at least see if it is the same address or if they form a pattern.
Depending upon what you are protecting would alter my level of concern. It seems plausible that a validation on the first and last name could be written that would reject this particular style of name. i.e. capital letters in the middle of the name? You could also flag any email going to "bogus" places like hotmail, yahoo, gmail, etc and require them to go through a moderator before being approved.
It also seems plausible to rig up a test case using this response to recreate the issue. Then add debug code to the Rails code until the precise reason for the exception is better understood.
I'm updating a node and pushing it:
remote_graph.push(node)
push() seems to return nothing. How can I tell if the push works? In my test code, I ought to be violating a unique constraint in Neo4J.
How can I tell using py2neo? I expected an exception.
When I enter the equivalent cypher into the Neo4J web tool, I get the following exception:
Node 322184 already exists with label VERSION and property "version"=[1.436818928448956E9]
which is what I expected.
Edit -
What I expected to get back was an indicator of whether the operation worked or not. I think push() accepts an array of nodes, so an array of results would be sensible. I don't know what the indicator would have within it since I don't know what is available. An array of strings would be fine, with each string being a failure reason, or "OK".
Generally speaking, the design of this API is: if it returns OK, you can assume everything has worked as expected, if an error is raised, that error will contain details of what went wrong. Therefore, absence of error should usually be interpreted as a signal of success.
That said, if you believe that your push has failed and no error has been raised, there is a bug in py2neo. For debugging, you can check the state of the database after your push by using the browser and then if you're able to recreate this scenario in a standalone piece of code, please raise an issue on GitHub and I will be happy to fix it.
I recently wrote a mailing platform for one of our employees to use. The system runs great, scales great, and is fun to use. However, it is currently inoperable due to a bug that I can't figure out how to fix (fairly inexperienced developer).
The process goes something like this...
Upload a CSV file to a specific FTP directory.
Go to the import_mailing_list page.
Choose a CSV file within the FTP directory.
Name and describe what the list contains.
Associate file headings with database columns.
Then, the back-end loops over each line of the file, associating the values with a heading, and importing these values into a database.
This all works wonderfully, except in a specific case, when a raw CSV is not correctly formatted. For example...
fname, lname, email
Bob, Schlumberger, bob#bob.com
Bobbette, Schlumberger
Another, Record, goeshere#email.com
As you can see, there is a missing comma on line two. This would cause an error when attempting to pull "valArray[3]" (or valArray[2], in the case of every language but mine).
I am looking for the most efficient solution to keep this error from happening. Perhaps I should check the array length, and compare it to the index we're going to attempt to pull, before pulling it. But to do this for each and every value seems inefficient. Anybody have another idea?
Our stack is ColdFusion 8/9 and MySQL 5.1. This is why I refer to the array index as [3].
There's ArrayIsDefined(array, elementIndex), or ArrayLen(array)
seems inefficient?
You gotta code what you need to code, forget about inefficiency. Get it right before you get it fast (when needed).
I suppose if you are looking for another way of doing this (instead of checking the array length each time, although that really doesn't sound that bad to me), you could wrap each line insert attempt in a try/catch block. If it fails, then stuff the failed row in a buffer (including the line number and error message) that you could then display to the user after the batch has completed, so they could see each of the failed lines and why they failed. This has the advantages of 1) not having to explicitly check the array length each time and 2) catching other errors that you might not have anticipated beforehand (maybe a value is too long for your field, for example).
One feature/need for my application is that I have an audit trail for certain tables. This is something that I have done many times in the past using different languages. So far I have been able to come up with the idea of using an Observer to monitor changes to the model/table. This works exactly as expected. The issue I am having is with string manipulation/querying specific data.
I have a string such as this {{user.created_by}} changed {{user.id}}'s First Name from {{user.first_name_was}} to {{user.first_name}}
The main issue I am trying to solve is what would be the best way to convert a string to an object/attribute definition?
Summary:
I want to see more detailed XML/SVG parsing error messages. I want to know where the errors are happening. How can I do this?
Background:
I'm working with some complicated javascript-generated SVG in Firefox. As I'm developing, sometimes while hunting down a big I'll see errors in the Firefox error console (or firebug) "Unexpected value NaN parsing y attribute". This is pretty clear. However, there's no line number, no code shown in Firebug - basically no way to track down where this error occurs.
With simple JS, it's a matter of tracking down the bad code. However, as my JS gets more complicated, I really need to be able to see which of hundreds of potential lines is causing this.
Ideally, I'd like to see this parsing error the same way I see JS errors or HTML errors:
Unexpected value NaN parsing y attribute.
Line 103: svgElement.setAttribute('x', some_bad_js_variable);
Is there any way to do this? Even knowing which SVG element is being affected would help, anything besides "There was an error somewhere". Thanks!
Nearly three years later and using Firefox 29.0.1, I have the same difficulty. I ended up commenting out successive blocks of code until I found the offending line.
FWIW, in my case Firefox didn't like the fact that I had created a node with blank attributes:
<clipPath id="chart_area">
<rect x="" y="" width="" height=""/>
</clipPath>
Once I removed the attributes or set them to any value, the problem went away. I was surprised because I'd expected the error to be in the Javascript instead. I hope this helps someone else.
Raise a bug in bugzilla and ask for the element tag name to be added to the error message: https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=SVG
Adding a line number would be more difficult. If want that too then create another bug specifically for it as you're less likely to get it.