I am using the Twitter API's Search Tweets endpoint (https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets) to search for tweets on a particular topic. My application is written in Java and uses Twitter4j. I follow the guidelines (i.e. no more than 180 calls per minute and no more than 500 characters per query) however I do have a test case that has 50 word exclusions (so the total character count is 397). My test program also runs 100 such searches in rapid succession though, like I said, I observe the rate limits strictly.
The odd behavior I'm seeing is that the test runs fine and gets results initially, but after an arbitrary period of time, I start getting the following error:
Message: Missing or invalid url parameter. 403:The request is understood, but it has been refused. An accompanying error message will explain why. This code is used when requests are being denied due to update limits (https://support.twitter.com/articles/15364-about-twitter-limits-update-api-dm-and-following).
message - Missing or invalid url parameter.
code - 195
The error message confuses me because I'm not trying to perform any of the update actions listed in the link provided. I'm just searching tweets. I'm also not sure what "missing or invalid url parameter" means. Is a query parameter missing from my request? The only required parameter is the query itself which I am definitely passing. The url is definitely correct unless Twitter4j is generating incorrect urls. So what does this message mean?
When I stop and restart the search (i.e. the max_id and since_id values get reset to -1, the searches start working again...for a while. As far as I know, nothing else about the search changes when it is restarted. Just those two ids.
Related
Over the last several weeks, my Rails 6.1 app has been raising hundreds of exceptions like the following:
An ActionDispatch::Http::Parameters::ParseError occurred in regdevs#create:
859: unexpected token at
'utf8=%E2%9C%93&authenticity_token=fnrg63qtelQvkQx9NQx8SaZIp7mR500toEiWdaJe91
%2FOzsEvRbGD4Ow6NNADCtjw7H3EkDvFZVGP4gQkGZceEw%3D%3D&user%5Bemail%5D=jrmills91
%40hotmail.com&user%5Bfirst_name%5D=wJAhzNnPWbqMdfH&user%5Blast_name%5D=tKdp
IVXJlneZbiuA&user%5Bnickname%5D=lBdbyYgiaHt&user%5Bpassword%5D=0SgLpVuf5Wqs
%21&user%5Bpassword_confirmation%5D=0SgLpVuf5Wqs%21&user%5Bterms_of_service
%5D=1&commit=Complete+registration&g-recaptcha-response=dzghTWcBpmI'
The error occurs when "someone" tries to register for a new account (via Devise) with bogus credentials. It clearly looks like some sort of attack.
I have two questions:
How can I find out more about this specific 859 unexpected token error?
Should I be concerned? The recaptcha guard seems effective in blocking account creation, but I wonder if I am missing something
I partially decoded the response by hand mostly just to see what it looked like.
utf8=%E2%9C%93
authenticity_token=fnrg63qtelQvkQx9NQx8SaZIp7mR500toEiWdaJe91/OzsEvRbGD4Ow6NNADCtjw7H3EkDvFZVGP4gQkGZceEw==
user[email]=jrmills91#hotmail.com
user[first_name]=wJAhzNnPWbqMdfH
user[last_name]=tKdpIVXJlneZbiuA
user[nickname]=lBdbyYgiaHt
user[password]=0SgLpVuf5Wqs!
user[password_confirmation]=0SgLpVuf5Wqs!
user[terms_of_service]=1
commit=Complete+registration
g-recaptcha-response=dzghTWcBpmI
The first line is utf8= and then that is the unicode for checkmark. Perhaps that is part of the captcha?
The user names etc clearly look bogus. I bet jrmills is getting lots of weird email. :-)
The ParseError is defined in parameters.rb and you can see the call that created it inside parse_formatted_parameters. The only place that calls it is from request.rb.
From the comments in the code (and what I would expect), if the authenticity_token doesn't match, an ActionController::InvalidAuthenticityToken should be raised. So, the parse error (I'm guessing) is lower down but I can't spot why the JSON or Ruby code would be complaining. I think I would have written that piece of code differently: captured the error and included the original error message with the new error -- but... such is life.
I would attempt to track down the IP address where these are being sent from and block that IP address if possible or at least see if it is the same address or if they form a pattern.
Depending upon what you are protecting would alter my level of concern. It seems plausible that a validation on the first and last name could be written that would reject this particular style of name. i.e. capital letters in the middle of the name? You could also flag any email going to "bogus" places like hotmail, yahoo, gmail, etc and require them to go through a moderator before being approved.
It also seems plausible to rig up a test case using this response to recreate the issue. Then add debug code to the Rails code until the precise reason for the exception is better understood.
I would like to, in a single request, get a list of messages filtered by date and by content in the body (if it's possible, of course).
I'm perfectly able to receive an ok response (200) from the following endpoint:
https://graph.microsoft.com/v1.0/users/user#company.com/messages?search="body:substring"
However, this gives me a broad response, since I'm interested only in cases between a specific week. Also, it's impossible to use a filter parameter along with the search one.
Then, in regard to the date filter, I could use:
https://graph.microsoft.com/v1.0/users/user#company.com/messages?$filter=(sentDateTime ge 2000-01-01) and (sentDateTime le 2100-01-01)
The problem is: I cannot filter the body as well, since I receive an error 400 - 'Bad Request: Invalid filter clause':
https://graph.microsoft.com/v1.0/users/user#company.com/messages?$filter=contains(body, substring)
Right now, my work around is: I request all the messages from user#company.com regarding a specific week, and after receiving that response, I locally filter the body in my machine. This works, but is not as fast as a compound request.
You can include specific a date restriction within your KQL Search eg
https://graph.microsoft.com/v1.0/me/messages?$search="(sent>=2019-01-01 AND sent<=2019-02-01) AND (body:asx)"
That should yield better performance then a filter
I'm making a new iOS (swift) app to test some concepts, and I'm using the GitHub Serach API to retrieve a list of filtered repositories.
The request is working fine so far, but I'm having trouble understanding the pagination process and how to know I reached the end of the results.
For what I saw, the Search API returns a maximum of 1k results, broke in pages of 100 maximum results. But the field in the returned json with the total count shows way more available results (I imagine that it shows the total repositories that satisfy the query and not the maximum available for return in the API).
The only way I found so far to obtain information about the pages (and the pagination process) in GitHub Documentation comes in the header of the response, like:
Status: 200 OK
Link: <https://api.github.com/resource?page=2>; rel="next",
<https://api.github.com/resource?page=5>; rel="last"
X-RateLimit-Limit: 20
X-RateLimit-Remaining: 19
Anyone can suggest the best approach to detect the end of the pages in this case?
Should I try to parse the information from the header or infer it somehow based on the returned json? I even got the "Link" header value but don't know how to parse it.
Using graph explorer to execute the following message search by subject works as expected:
https://graph.microsoft.com/v1.0/me/messages?$search=%22subject%3Agoogle.com%3A443%2Fsearch%3Fq%3Da%2Btest*%22
However, this message search by body fails. The only difference between this and the previous search is that subject is replaced with body:
https://graph.microsoft.com/v1.0/me/messages?$search=%22body%3Agoogle.com%3A443%2Fsearch%3Fq%3Da%2Btest*%22
The error that I receive for the second search is "An internal server error occurred. The operation failed., Too few pops in query tree".
This search used to work. How do I fix it?
This is a 500 error - you won't make this work. I'll report this as an error. The error appears to be with the second colon in the search string.
$search="body:google.com:443/search?q=a+test*" - this 500s
$search="body:google.com443/search?q=a+test*" - this 200s
Conjecture
I suspect that the search feature now supports multiple fields as the following query works as expected:
https://graph.microsoft.com/v1.0/me/messages?$search="body:Minerva subject:pilot"
This would break your query as it appears that the colon is only considered for specifying the field to search on.
Potential workaround
You could break your query string into multiple parts like this:
https://graph.microsoft.com/v1.0/me/messages?$search="body:google.com AND body:443/search?q=a+test*"
It is not sufficient but may work for your scenario.
I have a script that connects via TCP/Sockets, Authenticates SSL, and then checks if the response from stream returns "OK".
It then sends a UID search command:
{tag} UID SEARCH (UNDELETED) (SENTSINCE "{RFC2060 Format Date}")
I then follow that with another OK check aswell as a * SEARCH stream response check.
When both of those are not true, I always end up with this as a result from the stream: xm005 NO UID SEARCH State error
Im not certain but is it possible this returns if the email has no UNDELETED inbox or something?
This seems to occur all the time on one of my chinese friends specific accounts on yeah.net (163-China related email service).
When I login to it with Windows 10's Mail App, I can see it has inboxes in CHINESE. Is it possible something to do with that is causing this issue?
I'm essentially wanting to search for every email within the sent-since date that has not been deleted, perhaps (UNDELETED) isn't a global declaration and is an actual inbox or something?
According to IMAPv4.1's RFC:
UNDELETED
Messages that do not have the \Deleted flag set.
So maybe its not to do with Inbox's? regardless its pretty odd that both emails this occurs on have Chinese Inbox's yet my English-Only one works splendid.
I removed (UNDELETED) and attempted running, and the same issue occurs, so it's not that.
The "state" in the error message could mean "your state does not include a mailbox". Make sure to issue a SELECT command before UID SEARCH.