Repeated "859: unexpected token" exceptions in a Rails 6 application - ruby-on-rails

Over the last several weeks, my Rails 6.1 app has been raising hundreds of exceptions like the following:
An ActionDispatch::Http::Parameters::ParseError occurred in regdevs#create:
859: unexpected token at
'utf8=%E2%9C%93&authenticity_token=fnrg63qtelQvkQx9NQx8SaZIp7mR500toEiWdaJe91
%2FOzsEvRbGD4Ow6NNADCtjw7H3EkDvFZVGP4gQkGZceEw%3D%3D&user%5Bemail%5D=jrmills91
%40hotmail.com&user%5Bfirst_name%5D=wJAhzNnPWbqMdfH&user%5Blast_name%5D=tKdp
IVXJlneZbiuA&user%5Bnickname%5D=lBdbyYgiaHt&user%5Bpassword%5D=0SgLpVuf5Wqs
%21&user%5Bpassword_confirmation%5D=0SgLpVuf5Wqs%21&user%5Bterms_of_service
%5D=1&commit=Complete+registration&g-recaptcha-response=dzghTWcBpmI'
The error occurs when "someone" tries to register for a new account (via Devise) with bogus credentials. It clearly looks like some sort of attack.
I have two questions:
How can I find out more about this specific 859 unexpected token error?
Should I be concerned? The recaptcha guard seems effective in blocking account creation, but I wonder if I am missing something

I partially decoded the response by hand mostly just to see what it looked like.
utf8=%E2%9C%93
authenticity_token=fnrg63qtelQvkQx9NQx8SaZIp7mR500toEiWdaJe91/OzsEvRbGD4Ow6NNADCtjw7H3EkDvFZVGP4gQkGZceEw==
user[email]=jrmills91#hotmail.com
user[first_name]=wJAhzNnPWbqMdfH
user[last_name]=tKdpIVXJlneZbiuA
user[nickname]=lBdbyYgiaHt
user[password]=0SgLpVuf5Wqs!
user[password_confirmation]=0SgLpVuf5Wqs!
user[terms_of_service]=1
commit=Complete+registration
g-recaptcha-response=dzghTWcBpmI
The first line is utf8= and then that is the unicode for checkmark. Perhaps that is part of the captcha?
The user names etc clearly look bogus. I bet jrmills is getting lots of weird email. :-)
The ParseError is defined in parameters.rb and you can see the call that created it inside parse_formatted_parameters. The only place that calls it is from request.rb.
From the comments in the code (and what I would expect), if the authenticity_token doesn't match, an ActionController::InvalidAuthenticityToken should be raised. So, the parse error (I'm guessing) is lower down but I can't spot why the JSON or Ruby code would be complaining. I think I would have written that piece of code differently: captured the error and included the original error message with the new error -- but... such is life.
I would attempt to track down the IP address where these are being sent from and block that IP address if possible or at least see if it is the same address or if they form a pattern.
Depending upon what you are protecting would alter my level of concern. It seems plausible that a validation on the first and last name could be written that would reject this particular style of name. i.e. capital letters in the middle of the name? You could also flag any email going to "bogus" places like hotmail, yahoo, gmail, etc and require them to go through a moderator before being approved.
It also seems plausible to rig up a test case using this response to recreate the issue. Then add debug code to the Rails code until the precise reason for the exception is better understood.

Related

Missing or Invalid Url Parameter with Twitter API

I am using the Twitter API's Search Tweets endpoint (https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets) to search for tweets on a particular topic. My application is written in Java and uses Twitter4j. I follow the guidelines (i.e. no more than 180 calls per minute and no more than 500 characters per query) however I do have a test case that has 50 word exclusions (so the total character count is 397). My test program also runs 100 such searches in rapid succession though, like I said, I observe the rate limits strictly.
The odd behavior I'm seeing is that the test runs fine and gets results initially, but after an arbitrary period of time, I start getting the following error:
Message: Missing or invalid url parameter. 403:The request is understood, but it has been refused. An accompanying error message will explain why. This code is used when requests are being denied due to update limits (https://support.twitter.com/articles/15364-about-twitter-limits-update-api-dm-and-following).
message - Missing or invalid url parameter.
code - 195
The error message confuses me because I'm not trying to perform any of the update actions listed in the link provided. I'm just searching tweets. I'm also not sure what "missing or invalid url parameter" means. Is a query parameter missing from my request? The only required parameter is the query itself which I am definitely passing. The url is definitely correct unless Twitter4j is generating incorrect urls. So what does this message mean?
When I stop and restart the search (i.e. the max_id and since_id values get reset to -1, the searches start working again...for a while. As far as I know, nothing else about the search changes when it is restarted. Just those two ids.

How to return custom error codes in a Rails API?

Given a RESTful API, implemented in Rails, I want to include in the responses not only the errors messages, generated by ActiveModel::Validations, but also custom error codes. First of all I would like to point out that I am not talking about HTTP Status codes. I am talking about having an error of any type (from general errors like record not found to small validation errors like username can't be blank) be mapped to a unique numeric code, that is custom application-specific error codes. Let me give an example - given a list of error codes, like:
1: record not found
... some other errors
# Validation errors for User model between 1000 to 2000
1001: first name can't be blank
1002: first name must contain at least 3 characters
1003: last name can't be blank
1004: last name must contain at least 3 characters
...some other errors
If I have a form for a user and submit it with empty fields for first and last name, I want to have in the response body something like:
{error_codes: [1001, 1002, 1003, 1004]}
or something similar (for example I could have an array of error objects, each with a code, message for developer, message for user etc.). Let me give an example with the Twilio API, taken from RESTful API Design: what about errors?:
Here, 20003 is some custom Twilio-specific code. The question is - how can this be implemented in Rails? I see several difficult aspects:
how do I get a list of all possible errors that can be encountered. It is hard to get such a list even only for the validation errors, let alone the other types of errors that can occur.
how should this list be organized - maybe in a YAML file?
how do I access the list - maybe something similar to the way translations are accessed via I18n.t?
I will really appreciate any advice on the topic. Thank you.
P.S. I think this is a similar question.
ActiveModel built-in validators can be found here. Sometimes one validator can check for more than one thing and output different messages. The easiest way to see them all is, as you've guessed, in its I18n yaml file, which can be found here.
One way of doing what you want is overwriting those messages with your custom codes. Another way is passing a custom message when explicitly attaching a validator to your models.
validates :name, message: 'code:001 - my custom message'
Those two options won't help you with structure, though. You won't have a different key code on your json out of the box.
One way you can accomplish that is to can create a helper to parse the error messages and extract the codes after they have been assigned to a model instance. Something along the lines of:
def extract_error_codes(error_messages)
error_messages.map{ |message| message.match('^code:(\d+)\s-')[1] }
end
That would give you an array of error codes for that instance if you'd used the format code:001 - my custom message.
Another, much more involved way, is to tap into ActiveModel's Validator class and store an error code when a validation fails. That would require going into each validator to assign the code.

Decoding YouTube's error 500 page

This is YouTube's 500 page. Can anyone help decode this information?
<p>500 Internal Server Error<p>
Sorry, something went wrong.
<p>A team of highly trained monkeys has been dispatched to deal with this situation.<p>
If you see them, show them this information:
AB38WENgKfeJmWntJ8Y0Ckbfab0Fl4qsDIZv_fMhQIwCCsncG2kQVxdnam35
TrYeV6KqrEUJ4wyRq_JW8uFD_Tqp-jEO82tkDfwmnZwZUeIf1xBdHS_bYDi7
6Qh09C567MH_nUR0v93TVBYYKv9tHiJpRfbxMwXTUidN9m9q3sRoDI559_Uw
FVzGhjH5-Rd1GPLDrEkjcIaN_C3xZW80hy0VbJM3UI5EKohX35gZNK2aNi_8
Toi9z3L8lzpFTvz5GyHygFFBFEJpoRRJSu3CbH5S2OxXEVo4HgaaBTV7Dx_1
Zs1HZvPqhPIvXg9ifd4KZJiUJDFS8grPLE7bypFsRamyZw-OCVyUHsGQKBwu
77pTtRwpF3hOxYLxM4KnAyiY1N6yrASSWyaeumRDENAoEEe8i8MRxzifqHuR
leatvNMiwsg1pbSl7IIiaKljZaD9UkRms4Kvz1uYUNk4AwXnJ9-Wq44ufMPl
syiHp_LwaeqyuxXykJMl-SA9p05VrJc4kCETUW3Ybp0yTYvVrqggo56A0ofC
OiyAmifQA9pdYVGeumrQtbFlFyDyG9VKNpzn5lqutxFZPsS8xjiILfF3bETD
H4aUb5fT4iERFsEL7S-ClsXiA4yAJdAcNH-OhGg9ipAaIxRRTOR5P1MYx6s6
-OrqgpT5VEaEx2hMpS1afaMd2_F21sxvcz2d8sCpEceHHSfsntTth6talYeD
4l63aUTbbCKV1lHxKWxdUjACFKRobeAvIpcJPcdHSN3CNQI-LlIWIx9jeyBU
tDcL6S6GpRG_Z2of9fmw0LHpVU5hKlQ3lCPd4pVP6J02yrsBi0S9OLoE9jmM
T2FfCvU1sWUCsrZu4-UPflXMyRnFK8aN8DYiwWWE8OvnLQ-LIaRDhjp29u9a
LT6Lh4KxEmWF5XeZTwrzJxtuDLVomxVD5mpwFvK0YSoaz9dnPGXb0Fm2txSL
BvGssSrWBJ4FeR6eEEkd_UkQ-aUnPv2W-POox17n54wzTwLugYjslRenMzmk
I4_jlXcx9NpKmUg7Pa0qJuaElt-ZymPv6h0cXRUlyZtS0iT9-CQOHWLYMi3I
kKrYa6bKUCAj058JEderSnbXqGEMvwBeZ_xgJpAjJiSgMOxJPokhbS6ezIv4
1JNr_dvQyvu4vh-YQNZ37fNTqQcoDZtYflBsJjuGrJlmIcqBYufB9g6nUaOE
xPAKjPdvZ_z1Rn_8sWVf8NHNBBKGe5lgDgBxypsV0kIwVa9QOlehivOaieBI
tmqHNdQIfdob0XUTEBPSeLj9hmw3Bqplc3gqUfFhIvpHml6dOTbjBhfkq0TE
5yCRHL2VSe2Xt9_i8SPQA2yCtJVO8HP6pnohmxqlBWSTE8Xj87PI6quX7f9i
0W6PdtkMYaGJsd_Ly_4Ag-KmGNHN585tF9eC5HeQ8Gz-vHZWOUiM4OQAG9UA
31ENOAjHtYb--ketbUcdX_FdjGiPtI_GxYeBqEShICotcd-S-E3bEGO-77M2
CuUUdB1AUYVDZR81XejVG5kSWsrz-p1qZ-6sSpSHCp114C6PheQPCwRHEr_1
AS-DkZfIuZ-w8XAo6pHIwvnv0dORSo-hPFgw1rw2VE4aKsgeMc7ZoPUxby1d
Zr-o-0X4ZMgxoQHw_Ub27rTTHxS5Czt_vgBPq7k5OK5dm6b7JCs6Dbn2dsIA
AakPL26t4smr8IiPAnqNC2sn7vxSiAe9mTJ670eNc6C9dCSGwqzqSURiLHmT
kFyLhNSOdipttECmSSA1qh_E0K4LUhiOq7MFDEzg9CLD8kuJrqpEGgltYpD-
8lk7KEpyjMqbWFs-qeD8uJpsVfY2ac1C67OmyGzkERVoC245-YXuNCP8KUZH
LGzRm9jXwUP_piDETX0N5xj34VOCfUTffT1WlWHmB9WRPhwjIsYYy_kgR-uT
kIEDQ23NVUEGgDoryl-ymysIfwifjq-lPB3e85dz1PajNxawsCrKNeR_4hhq
zE_E4ete1EgXeAYoeH4UIgrPGXDD-KfoNoB6viNs0GzNU9czD9Avr-tDtARO
HBLSLIVRYq8caMA-jvpplTOMoDdmUMUWytf4Y_F5tKTpNtLPaAe1py1IgZBl
lfAGY9L_k5slelh_9gUBEkURxS2oMGf2gdSeDdRBxKKx5tF1b-cuMLK6JYZJ
vbGFYSsSENOkHrHEo9NdTwTi7NON9ZgRJgh7OaENK4TFCXrhKc4C6cyJs-V_
HZ0Q-B8XDyjL0qudg_0rJbjTNpNZajT_1WGsnhsTTAgMCGtTsj1T8vNx2LuX
lPQV30nUKpukdCP3zuiE9_aeJQ-nzf3dMQ-KnZU5APmGcIP_u2be6blieMWH
qVax1asKmuIjslh49ceM6lRt3Ia2bHUB8b1TMSjU4I79KPqc3clDnD8quNnU
cRkgfJ_8LCEoH7jml_2TNV0fLuH_9IOXF3jKjhT9K5f-e5N06GmPQLzdqzeQ
MnEtHuDcf4IizyKnB5GUXoNfQxbScQEzztQ_nHMYfF-E8KqoxxlK-Z0wfEDv
dJpL3mcNfFu_vz-_LJ0oI4dE0-vthsxbpTxVQkdI0E5XSi4nYfLqhXompk4j
gpxcHBjsXVbWcnelWhhQP15gCApj6Gz_ddRtk_uxiyiqZ44oUUDcl1KeWMTf
yhKDj7jgGNzTOkUsXZRPb9M77-ZYPuL2wR68E3b9PC_mS6HBHiUxQ7pXvkwS
Bi2CoFgd9SqBXk2O5I_BPaEoA8Aorazw4OvDrmTQrCk4OkGPKRukE4Ci2RMq
TZIYbBz-v3QxmOKHJoMXPNOfj93TRWpmlAd6iHCH6BVlSdfgfjdbHeD0b0ct
qXC_-S5fr1XFBuaZwaUTrBPxU-3IxWLp-dx7wpKcFykqKnByYpkzR3twKEXc
z--CZV79Qk3ZTMY9ATia4HbyhoAqY_hV9GKAHQdU_C-9qwYt0rliNUcizlBc
RHcwzoMyx30ciwbE8e9QsEH_AMa3E2ezuhjTqQlAG33_Gy5Bwe7fj6zNR0ud
jjpcNVf-wprWHEYxMcKwjCQvEHBtv6TnCHkgi_AOtPzzm3aYkMc_ysdNAnI7
DE_T9S5Mkcs6VdT2DWgUN_UC-oAw27xej0aTIn0GckXPDcLBgvrUhUPU3FRn
lW65syvFvxFmBOiCAEHD1q6Is1XhIf8vE6y0FMdNEWSMUW5rQG8f3KP_pjqG
XlUHnGrQPQysylrczHOj4E3WTT918xg2vrXVraDJbCYnJpaWp8m94iqZw2gJ
I_0UWOAZJMCYWz5jYf2DanCOBaGeZIO-UsWorP6YV3yHehirZ_Lc6KtaUorb
bm_BnnCqGVZypL4k6cDy-4GyO2GNohXzN-VWqbAIUQIat9w6RsDpzpS2DIap
96aMBDg24D73RhFTEgCSunPpbbGrDVU3GFkuTGFFBQWNfAU_F22XtoPr_ZB0
1zZPrBVXrEhebvrzp0Z31_sIT8zLop_oaSRvykbyJKKxucARfPee-d0xlgWN
WwKKtl49WVMhhu0OfDScH3knAVdv0LDAyt1fo3WF8jxdp7J9Hn3OWF3rcn0p
zw0gt6YV_6FRy_UbZmpLBvEhZQKUfKuxp6LK-SfHOilT29ERg9LJZhnyluTV
HELRtkJIcHzvphXupCIaIgZispYxHNSmAfze2cshWBYizGTBSKXWgrJeo7Q6
kEjim72yKaJ8JaLzMQFPxtQxhtvHRw94dCuXcajg3nE_r_9t7D8RicqF-CVV
tvp-rHMPhhizlgfixHWXrPB7reTtftT64pOSl5vUop8gTlbeW5Kg7WQAPNfp
zUH8YcAo0xDLJHA-FgTM4mYGih41rKaKKteWRFGU-fIyEzeO1s35tbGzlZ7R
btUG_fCpIbaJmucMZK9OzVBSfBgTBtFSesqKq6hIc8HctGcj5LPUfP9DRqqe
CrBi6bPjTlzrjaxJoU6oRq4ZtiBG38skOAaCUk61tpjilkq1fmWe2ByvLXhp
O2furZoiwNrizYmUmAW3ak3iSneScA64M-9apdZwhhEgpqyw5mUMYNT5SOOf
xZePlgXxhlL81t3KlofdbzT0w6tlbbT0NSbj9Q_zNkeZ8ar5aeMgTR-pJACg
baB20YVezziX-yboCF-uIptCTFNV
(Source: this post on HN)
The debug information contained in the (urlsafe-)base64 blob is likely encrypted.
Think about it from Google's perspective: You would want to display a stack trace, relevant headers of the http request and possibly some internal state of the user session to help a developer debug the situation. On the other hand all that information might contain sensitive information that you don't want the general public to see or that might endanger the user if he copy'n pastes it in a public support forum.
If I was to take a guess of the format I would imagine:
A public identifier of the key used for encryption (their servers could use different keys then)
The debug data encrypted using an authenticated encryption scheme
Additional data for error correction when OCR has to be used
For statistical analysis of the format it would be interesting to sample a lot of these error messages and see if some parts of the message are less random than you would expect from encrypted data (symmetrical encrypted data should follow a uniform distribution).
It looks like you are not the only one who is looking for some secret messages in YouTube error page. It seems that you can decode it using Base64.
Here is how:
http://www.cambus.net/decoding-youtube-http-error-500-message/
In a nutshell:
Sadly, contrary to my expectations, there doesn't seem to be any
hidden messageā€¦ Screw you, highly trained monkeys!
I guess it is just another Easter Egg similar to 'Goats Teleported' performance counter Google Chrome had:
https://plus.google.com/+RobertPitt/posts/PrqAX3kVapn
But I guess unless you look like this, you can't be 100% sure.
It's entirely possible that this is random padding to avoid the "friendly" IE error pages that show if your error page does not contain more than 512 bytes of HTML. It would be base64 encoded if it were simply random bytes.
Imho this is all about customer care.
Actually there would be no need to send the error/debug message to the customer, because, I guess, it's already handled internally.
So:
why do we see this?
and why do they crypt it?
and is there really no hidden message for us?
Although the error might be handled and resolved internally, this does not necessarily satisfy a customer, who is not able to use the product. They pretty much do crypt by a good reason as this debug message might reveal more than a typical admin is used to.
And also there is no need to hide a message for us. Why? Because we NEVER stop until we find something.
I think:
internally the error is dealt with
external users might have something in hand to tell a technician if necessary and in return can get an approximation of ongoing problem
All in all nothing special about it and i think linking e.g. to the inf. monkey theorem is a bit overspectulated...
Error 500 means google has a problem which can not resolve. So when reporting a bug the most important thing is to prepare reproduction steps. So I tried to find an answer of the question "When this happens?"
I found this post in reddit: https://www.reddit.com/r/youtube/comments/40k858/is_youtube_giving_you_500_internal_server_errors/?utm_source=amp&utm_medium=comment_list
As resume:
It happens on desktops (www...), it works ok on mobile version (m...)
It happens for authenticated users. For anonymous users is working fine.
The problem is resolved after cookies are cleaned.
So I would give a direction: try to find the key in the session cookie. I hope my 2 cents will help.

Customizing All errors symfony 1.4

Sorry for my English and writing through a translator.
I'm new here and I'm a bit lost in what I want, maybe you can help me out if they wish, as explained here:
I want to customize all the errors that may be produced (any status code that is a mistake. 404, 500, 204, 301 .... etc.) And also show suggestions based on the "module" and/or "action" from where the error occurred.
I also wish I could tell by the file redirection "setting.yml" application, something like this:
error_404_module: common
error_404_action: errors
error_500_module: common
error_500_action: errors
error_204_module: common
error_204_action: errors
error_301_module: common
error_301_action: errors
........
Within the "app/frontend/modules/common" in the "action.class.php" define "executeErrors" in which I receive the status code that has occurred and the module where the error occurred or except for well, getting partial show "app/frontend/modules/common/templates/_errorXXXSuccess.php" and the partial of the suggestion that I seek and extract the module "referer" where the error occurred.
Well, that's the idea, dynamic content for different types of error in different circumstances and working perfectly with the type codes 404 but not with the rest ...
I read that could place me in the middle of the event by creating a file "config/error.html.php" and "config/exception.html.php". Indeed if I produce such a 500 error and manually these files are executed. The problem is that from there I can not access variables from symfony. Because if so I could invoke "common/errors" and bad send as parameters the error status code and the module where it occurred, but it is not. From there all I can do is play with $ _SERVER and routing.yml but can not get good results because the environment is mixed with Development Production (/frontend_dev.php) and is something I also want to avoid.
That is, I want to get as in "setting.yml" does "error_404_module" and "error_404_action" but with different error codes. Thus allowing me to show my custom pages in production based on "error or exception" and/or "module or action" where the error has been released from production, and in case of being in development mode (/frontend_dev.php/....), show the symfony Dispatch.
Surely life is complicandome and has a simpler way to do what I want but after thousands of combinations do not get the right result. Even if symfony may not allow other types of error redirect from "setting.yml" is because they are reserved for him or something. I do not understand.
Please please help me because no documentation to find and secure here at least we left public for others who want to do something.
Many Thanks and greetings!
First of all - you can't get symfony to intercept all errors. Some of the errors may be server errors and the request won't even get to your application. You can reproduce some of them explicitly setting responses in symfony though, but I may not be necessary to deal with all of them.
As for the exact response: you should use a listener to hook your errors to a class that handles them according to your need. See: https://stackoverflow.com/a/7077818/580167 for implementation details.

Parsing a CSV for Database Insertion when Formatted Incorrectly

I recently wrote a mailing platform for one of our employees to use. The system runs great, scales great, and is fun to use. However, it is currently inoperable due to a bug that I can't figure out how to fix (fairly inexperienced developer).
The process goes something like this...
Upload a CSV file to a specific FTP directory.
Go to the import_mailing_list page.
Choose a CSV file within the FTP directory.
Name and describe what the list contains.
Associate file headings with database columns.
Then, the back-end loops over each line of the file, associating the values with a heading, and importing these values into a database.
This all works wonderfully, except in a specific case, when a raw CSV is not correctly formatted. For example...
fname, lname, email
Bob, Schlumberger, bob#bob.com
Bobbette, Schlumberger
Another, Record, goeshere#email.com
As you can see, there is a missing comma on line two. This would cause an error when attempting to pull "valArray[3]" (or valArray[2], in the case of every language but mine).
I am looking for the most efficient solution to keep this error from happening. Perhaps I should check the array length, and compare it to the index we're going to attempt to pull, before pulling it. But to do this for each and every value seems inefficient. Anybody have another idea?
Our stack is ColdFusion 8/9 and MySQL 5.1. This is why I refer to the array index as [3].
There's ArrayIsDefined(array, elementIndex), or ArrayLen(array)
seems inefficient?
You gotta code what you need to code, forget about inefficiency. Get it right before you get it fast (when needed).
I suppose if you are looking for another way of doing this (instead of checking the array length each time, although that really doesn't sound that bad to me), you could wrap each line insert attempt in a try/catch block. If it fails, then stuff the failed row in a buffer (including the line number and error message) that you could then display to the user after the batch has completed, so they could see each of the failed lines and why they failed. This has the advantages of 1) not having to explicitly check the array length each time and 2) catching other errors that you might not have anticipated beforehand (maybe a value is too long for your field, for example).

Resources