I need to find all mails in IMAP mailbox which contains somestring in BODY and is FROM someone#me.com or TO someone#me.com.
Trying to do:
49:51.53 > JBPM3 SEARCH CHARSET utf-8 "BODY \"somestring\" (OR (TO \"someone#me.com\") (FROM \"someone#me.com\"))"
Receiving:
49:51.71 < JBPM3 BAD Could not parse command
How to make it work using GMail?
You may skip parenthesis '(' ')' to group logical expressions in IMAP.
Parenthesis are not needed in Polish Notation (see edit below):
A0001 SEARCH CHARSET utf-8 BODY "somestring" OR TO "someone#me.com" FROM "someone#me.com"
You could also use gmail search syntax (X-GM-RAW) command:
http://www.limilabs.com/blog/search-gmail-using-gmails-search-syntax
[Edit]
Parenthesis are sometimes required in IMAP SEARCH. This is because AND operator can have more than 2 operands and is not explicitly defined:
http://www.limilabs.com/blog/imap-search-requires-parentheses
Related
Im using a BG96 modem to connect to AWS iot over MQTT.
I'm trying to set my MQTT Last Will and Testament with the following AT-command:
+QMTCFG:"will",(0-5),(0,1),(0-2),(0,1),"willtopic","willmessage"
Which works great.
But now I'm trying to add a JSON formatted string to "willmessage", so I need to add "" (double quotes) in there, which means I need to escape them in my command. But I have no clue if I can escape them or what the escape character is.
Things I tried: \" (backslash) and "" (double double quotes)
I looked in all of the BG96 datasheets, and I don't see it mentioned anywhere.
I had the same issue while using MQTT commands on a SIMCOM SIM800c, and I noticed that the regular backslash (\) escapes the quotation marks (as it does in c) when communicating directly with the GSM unit via a USB to TTL converter. To implement this in software I printed the following string to the UART connected to the GSM Modem:
AT+SMPUB=\"testTopicPost\",0,1,\"{\x5c\x22Key\x5c\x22 : \x5c\x22Value\x5c\x22}\"
What this basically does is send the raw \ and " characters to the GSM unit. Hope this solution works for you as well.
Escaping of " within a string is covered in chapter 5.4.2.2 String constants in the V.250 standard - which is a MUST READ for anyone writing code handling AT commands (read at least all of chapter 5):
String constants shall consist of a sequence of ... except for the characters """ ... . String constants shall be bounded at the beginning and end by the double-quote character (""" ... . ... The double-quote character, used as the beginning and ending string delimiter, shall be represented within a string constant as "\22".
So the escape mechanism is \22 not \x22. This should be universally the case for all modems and not something that is implementation dependent.
I did not find reference to documentation of MQTT and BG96 and you did not link any of your "all of the BG96 datasheets" so I am just providing example syntax for an imaginary command to send a JSON payload of {"key": "value"}:
AT+SOMECOMMAND=...,"{\22key\22: \22value\22}"
Some special characters have different values between the ASCII character set and the GSM character set.
i.e.
In the ASCII character set: \ = 0x5C.
In the GSM character set: Ö = 0x5C.
Beyond this point, some special characters must be entered using a specific way, such as a 2-byte representation. I suggest you check the standard/version of AT commands implemented on your hardware (i.e. GSM 07.07, GSM 07.05, manufacturer specific set...).
i.e. I'm using a GPS+GPRS modem from Ai-Thinker called A9G. In this one, to use the AT+MQTTPUB command with data formatted in JSON style, I need to append \x5c\x32\x32. So the module will interpret this as \22 and the server as \".
i.g.
"{\x5c\x32\x32Key\x5c\x32\x32:\x5c\x32\x32Value\x5c\x32\x32}"
at the cloud it will be:
{"Key":"Value"}
I am trying to create a Lexer/Parser with ANTLR that can parse plain text with 'tags' scattered inbetween.
These tags are denoted by opening ({) and closing (}) brackets and they represent Java objects that can evaluate to a string, that is then replaced in the original input to create a dynamic template of sorts.
Here is an example:
{player:name} says hi!
The {player:name} should be replaced by the name of the player and result in the output i.e. Mark says hi! for the player named Mark.
Now I can recognize and parse the tags just fine, what I have problems with is the text that comes after.
This is the grammar I use:
grammar : content+
content : tag
| literal
;
tag : player_tag
| <...>
| <other kinds of tags, not important for this example>
| <...>
;
player_tag : BRACKET_OPEN player_identifier SEMICOLON player_string_parameter BRACKET_CLOSE ;
player_string_parameter : NAME
| <...>
;
player_identifier : PLAYER ;
literal : NUMBER
| STRING
;
BRACKET_OPEN : '{';
BRACKET_CLOSE : '}';
PLAYER : 'player'
NAME : 'name'
NUMBER : <...>
STRING : (.+)? /* <- THIS IS THE PROBLEMATIC PART !*/
Now this STRING Lexer definition should match anything that is not an empty string but the problem is that it is too greedy and then also consumes the { } bracket tokens needed for the tag rule.
I have tried setting it to ~[{}]+ which is supposed to match anything that does not include the { } brackets but that screws with the tag parsing which I don't understand either.
I could set it to something like [ a-zA-Z0-9!"§$%&/()= etc...]+ but I really don't want to restrict it to parse only characters available on the british keyboard (German umlaute or French accents and all other special characters other languages have must to work!)
The only thing that somewhat works though I really dislike it is to force strings to have a prefix and a suffix like so:
STRING : '\'' ~[}{]+ '\'' ;
This forces me to alter the form from "{player:name} says hi!" to "{player:name}' says hi!'" and I really desperately want to avoid such restrictions because I would then have to account for literal ' characters in the string itself and it's just ugly to work with.
The two solutions I have in mind are the following:
- Is there any way to match any number of characters that has not been matched by the lexer as a STRING token and pass it to the parser? That way I could match all the tags and say the rest of the input is just plain text, give it back to me as a STRING token or whatever...
- Does ANTLR support lookahead and lookbehind regex expressions with which I could match any number of characters before the first '{', after the last '}' and anything inbetween '}' and '{' ?
I have tried
STRING : (?<=})(.+)?(?={) ;
but I can't seem to get the syntax right because that won't compile at all, which leads me to believe that ANTLR does not support lookahead and lookbehind syntax, but I could not find a definitive answer on the internet to that question.
Any advice on what to do?
Antlr does not support lookahead or lookbehind. It does support non-greedy wildcard matches, but only when the .* non-greedy wildcard is followed in the rule with the termination sequence (which, as you say, is also contained in the match, although you could push it back into the input stream).
So ~[{}]* is correct. But there's a little problem: lexer rules are (normally) always active. So that lexer rule will be active inside the braces as well, which means that it will swallow the entire contents between the braces (unless there are nested braces or braces inside quotes or some such, and that's even worse).
So you need to define different lexical contents, called "lexical modes" in Antlr. There's a publically viewable example in the Antlr Definitive Reference, which shows a solution to a very similar problem: parsing HTML.
I am searching for all occurrences of an out of date email domain in my codebase on TFS. Specifically, I am searching for "#testexample.com". However, the search seems to completely ignore the "#" symbol.
If I search "#testexample.com", I get all occurrences of "testexample.com". I have also tried "*#testexample.com" but prefix wildcards aren't allowed in code search.
That's not supported.
Checked for some characters in code search. You can't use the symbol characters except * and ? as part of your search query, which including below characters: . , : ; / \ ` ' " # = ! # $ & + ^ | ~ < > ( ) { } [ ]. The search will simply ignore these symbols.
But you can use wildcard characters * and ? to broaden your search.
You can use wildcard characters anywhere in your search string except
as a prefix in a simple search string or a query that uses a code type
filter. For example, you cannot use a search query such as
*RequestHandler or class:?RequestHandler. However, you can use prefix wildcards with the other search filter functions; for example, the
search query strings file:*RequestHandler.cs and repo:?Handlers are
valid.
Please see Broaden your search with wildcards for details.
If you want to search the strings including these symbol exactly(such as '#' here), you can code search with other strings (eg, testexample.com here) to narrow down the scope first, then copy the specific code to text editor which support the symbols (eg, Notepad++), then search stings with the symbol characters.
Besides, if you are using Git, another workaround is using the code search tool Hound: a lightning fast code search tool, it supports the symbol characters. Reference this thread to use it: How can I publish source code (Visual Studio) on a intranet?
I have been trying to figure out how exactly \x00, \n, \r, \, or \x1a can cause an SQL Injection (as it is mentioned at http://nl3.php.net/manual/en/function.mysql-real-escape-string.php)
I understand the idea of single quote and double quotes, but how and why I need to take care of the other items to make my query safe?
I was wondering about the same question and I found the answer in the C API documentation of MySQL, it states:
Characters encoded are “\”, “'”, “"”, NUL (ASCII 0), “\n”, “\r”, and
Control+Z (\x1a). Strictly speaking, MySQL requires only that backslash and
the quote character used to quote the string in the query be escaped.
mysql_real_escape_string() quotes the other characters to make them
easier to read in log files.
It is also explained in String Literals that:
The mysql client truncates quoted strings containing NUL characters if
they are not escaped, and Control+Z may be taken for END-OF-FILE on
Windows if not escaped.
The NUL character represents the end of a string in C language, so this can falsely terminate the input argument of the mysql client program. Same thing for \x1a, it marks the end-of-file under Windows (try type test.txt in a command prompt with a \x1a character in the middle of the file).
The main point is that an admin can miss important information in a log file if his log file reader doesn't show the data beyond one of these characters. But who still uses precarious type command or equivalent under Windows to read a log file anyway?
In other terms, there is no danger with \n, \r, \0 or \x1a in PHP, other than potentially making a log file difficult to read.
As for the backslash, \' OR 1==1 would be converted to \\' OR 1==1 if it was not escaped too, cancelling the effect of the escaping of the quote.
let's assume you have
$SQL="select * from mytable where myfield='$uservalue'"
\ -> \:
try \' or 1=1; --', after escaping the quote, you would get \\' or 1=1; --' and the SQL would be select * from mytable where myfield='\\' or 1=1; --'
\x00
Not important for PHP, but for C
Sorry, too lazy for the rest.
Background (question further down)
I've been Googling this back and forth reading RFCs and SO questions trying to crack this, but I still don't got jack.
So I guess we just vote for the "best" answer and that's it, or?
Basically it boils down to this.
3.4. Query Component
The query component is a string of information to be interpreted by the resource.
query = *uric
Within a query component, the characters ";", "/", "?", ":", "#", "&", "=", "+", ",", and "$" are reserved.
The first thing that boggles me is that *uric is defined like this
uric = reserved | unreserved | escaped
reserved = ";" | "/" | "?" | ":" | "#" | "&" | "=" | "+" | "$" | ","
This is however somewhat clarified by paragraphs such as
The "reserved" syntax class above refers to those characters that are allowed within a URI, but which may not be allowed within a particular component of the generic URI syntax; they are used as delimiters of the components described in Section 3.
Characters in the "reserved" set are not reserved in all contexts. The set of characters actually reserved within any given URI component is defined by that component. In general, a character is reserved if the semantics of the URI changes if the character is replaced with its escaped US-ASCII encoding.
This last excerpt feels somewhat backwards, but it clearly states that the reserved character set depends on context. Yet 3.4 states that all the reserved characters are reserved within a query component, however, the only things that would change the semantics here is escaping the question mark (?) as URIs do not define the concept of a query string.
At this point I've given up on the RFCs entirely but found RFC 1738 particularly interesting.
An HTTP URL takes the form:
http://<host>:<port>/<path>?<searchpart>
Within the <path> and <searchpart> components, "/", ";", "?" are reserved. The "/" character may be used within HTTP to designate a hierarchical structure.
I interpret this at least with regards to HTTP URLs that RFC 1738 supersedes RFC 2396. Because the URI query has no notion of a query string also the interpretation of reserved doesn't really let allow me to define query strings as I'm used to doing by now.
Question
This all started when I wanted to pass a list of numbers together with the request of another resource. I didn't think much of it, and just passed it as a comma separated values. To my surprise though the comma was escaped. The query page.html?q=1,2,3 encoded turned into page.html?q=1%2C2%2C3 it works, but it's ugly and didn't expect it. That's when I started going through RFCs.
My first question is simply, is encoding commas really necessary?
My answer, according to RFC 2396: yes, according to RFC 1738: no
Later I found related posts regarding the passing of lists between requests. Where the csv approach was poised as bad. This showed up instead, (haven't seen this before).
page.html?q=1;q=2;q=3
My second question, is this a valid URL?
My answer, according to RFC 2396: no, according to RFC 1738: no (; is reserved)
I don't have any issues with passing csv as long as it's numbers, but yes you do run into the risk of having to encode and decode values back and forth if the comma suddenly is needed for something else. Anyway I tried the semi-colon query string thing with ASP.NET and the result was not what I expected.
Default.aspx?a=1;a=2&b=1&a=3
Request.QueryString["a"] = "1;a=2,3"
Request.QueryString["b"] = "1"
I fail to see how this greatly differs from a csv approach as when I ask for "a" I get a string with commas in it. ASP.NET certainly is not a reference implementation but it hasn't let me down yet.
But most importantly -- my third question -- where is specification for this? and what would you do or for that matter not do?
That a character is reserved within a generic URL component doesn't mean it must be escaped when it appears within the component or within data in the component. The character must also be defined as a delimiter within the generic or scheme-specific syntax and the appearance of the character must be within data.
The current standard for generic URIs is RFC 3986, which has this to say:
2.2. Reserved Characters
URIs include components and subcomponents that are delimited by characters in the "reserved" set. These characters are called "reserved" because they may (or may not) be defined as delimiters by the generic syntax, by each scheme-specific syntax, or by the implementation-specific syntax of a URI's dereferencing algorithm. If data for a URI component would conflict with a reserved character's purpose as a delimiter [emphasis added], then the conflicting data must be percent-encoded before the URI is formed.
reserved = gen-delims / sub-delims
gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "#"
sub-delims = "!" / "$" / "&" / "'" / "(" / ")"
/ "*" / "+" / "," / ";" / "="
3.3. Path Component
[...]
pchar = unreserved / pct-encoded / sub-delims / ":" / "#"
[...]
3.4 Query Component
[...]
query = *( pchar / "/" / "?" )
Thus commas are explicitly allowed within query strings and only need to be escaped in data if specific schemes define it as a delimiter. The HTTP scheme doesn't use the comma or semi-colon as a delimiter in query strings, so they don't need to be escaped. Whether browsers follow this standard is another matter.
Using CSV should work fine for string data, you just have to follow standard CSV conventions and either quote data or escape the commas with backslashes.
As for RFC 2396, it also allows for unescaped commas in HTTP query strings:
2.2. Reserved Characters
Many URI include components consisting of or delimited by, certain
special characters. These characters are called "reserved", since
their usage within the URI component is limited to their reserved
purpose. If the data for a URI component would conflict with the
reserved purpose, then the conflicting data must be escaped before
forming the URI.
Since commas don't have a reserved purpose under the HTTP scheme, they don't have to be escaped in data. The note from § 2.3 about reserved characters being those that change semantics when percent-encoded applies only generally; characters may be percent-encoded without changing semantics for specific schemes and yet still be reserved.
I think the real question is: "What characters should be encoded in a query string?" And that depends mainly on two things: The validity and the meaning of a character.
Validity according to the RFC standard
In RFC3986 we can find which special characters are valid and which are not inside a query string:
// Valid:
! $ & ' ( ) * + , - . / : ; = ? # _ ~
% (should be followed by two hex chars to be completely valid (e.g. %7C))
// Invalid:
" < > [ \ ] ^ ` { | }
space
# (marks the end of the query string, so it can't be a part of it)
extended ASCII characters (e.g. °)
Deviations from the standard
Browsers and web frameworks do not always strictly follow the RFC standard. Below are some examples:
[, ] are not valid, but Chrome and Firefox do not encode these characters inside a query string. The reasoning given by Chrome devs is simply: "If other browsers and an RFC disagree, we will generally match other browsers." QueryHelpers.AddQueryString from ASP.NET Core on the other hand will encode these characters.
Other invalid characters that are not encoded by Chrome and Firefox are:
\ ^ ` { | }
' is a valid character inside a query string but will be encoded by Chrome, Firefox and QueryHelpers nevertheless. The explanation given by Firefox devs is that they knew that they don't have to encode it according to the RFC standard, but did it to reduce vulnerabilities.
Special meaning
Some characters are valid and also don't get encoded by browsers, but should still be encoded in certain cases.
+: Spaces are normally encoded as %20 but alternatively they can be encoded as +. So + inside a query string means it's an encoded space. If you want to include a character that's actually supposed to literally mean plus, then you have to use the encoded version of + which is %2B.
~: Some old Unix systems interpreted URI parts that started with ~ as a path to a home directory. So it's a good idea to encode ~ if it's not meant to denote the start of a Unix home directory path for an old system (so nowadays probably always encode).
=, &: Usually (although RFC doesn't specify that this is required) query strings contain parameters in the format "key1=value1&key2=value2". If that's the case and =s or &s should be part of the parameter key or the parameter value instead of giving them the role of separating the key and value or separating the parameters, then you have to encode those =s and &s. So if a parameter value should for some reason consist of the string "=&" then it has to be encoded as %3D%26 which then can be used for the full key and value: "weirdparam=%3D%26".
%: Usually web frameworks figure out that %s that are not followed by two hex characters simply mean the % itself, but it's still a good idea to always encode % when it's supposed to only mean % and not indicate the start of an encoded character (e.g. %7C) because RFC3986 specifies that % is only valid when followed by two hex characters. So don't use "percentageparam=%" use "percentageparam=%25" instead.
Encoding guidelines
Encode every character that is otherwise invalid* according to RFC3986 and every character that can have special meaning but should only be interpreted in a literal way without giving it a special meaning. You can also encode things that aren't required to be encoded, like '. Why? Because it doesn't hurt to encode more than necessary. Servers and web frameworks when parsing a query string will decode every encoded character, no matter if it was really necessary to previously encode that character or not.
The only characters of a query string that shouldn't be encoded are those that can have a special meaning and shouldn't lose that special meaning, e.g. don't encode the = of "key1=value1". For that to achieve don't apply an encoding method to the whole query string (and also not to the whole URI) but apply it only and separately to the query parameter keys and values. For example, with JS:
var url = "http://example.com?" + encodeURIComponent(myKey1) + "=" + encodeURIComponent(myValue1) + "&" + encodeURIComponent(myKey2)...;
Note that encodeURIComponent encodes a lot more characters than necessary meaning characters that are valid in a query string and don't have special meaning there e.g. /, ?, ...
The reason is that encodeURIComponent wasn't created for query strings alone but instead encodes characters that have special meaning outside of the query string as well, e.g. / for the path URI component. QueryHelpers.AddQueryString works in a similar manner. Under the hood it uses System.Text.Encodings.Web.DefaultUrlEncoder which is not just meant for query strings but also for isegment, ipath-noscheme and ifragment.
* You could probably get away with only regarding those characters as invalid that are both not allowed by the RFC and that are also always encoded by Chrome for instance. This would be Space " < >. But it's probably better to be on the safer side and encode at least everything that RFC3986 considers invalid.
OP's questions
My first question is simply, is encoding commas really necessary -> No it's not necessary, but it doesn't hurt (except ugliness) and will happen with default encoding methods e.g. encodeURIComponent and decoding and query string parsing should work nevertheless.
My second question, is this a valid URL (page.html?q=1;q=2;q=3)? -> It's RFC valid, but your server / web framework might have a hard time parsing the query string when it might expect the typical "key1=value1&key2=value2" format for query strings.
Where is specification for this? -> There isn't a single specification that covers everything because some things are implementation specific. For instance there are different ways of specifying arrays inside of query strings.
Just use ?q=1+2+3
I am answering here a fourth question :) that did not ask but all started with: how do i pass list of numbers a-la comma-separated values? Seems to me the best approach is just to pass them space-separated, where spaces will get url-form-encoded to +. Works great, as longs as you know the values in the list contain no spaces (something numbers tend not to).
page.html?q=1;q=2;q=3
is this a valid URL?
Yes. The ; is reserved, but not by an RFC. The context that defines this component is the definition of the application/x-www-form-urlencoded media type, which is part of the HTML standard (section 17.13.4.1). In particular the sneaky note hidden away in section B.2.2:
We recommend that HTTP server implementors, and in particular, CGI implementors support the use of ";" in place of "&" to save authors the trouble of escaping "&" characters in this manner.
Unfortunately many popular server-side scripting frameworks including ASP.NET do not support this usage.
I would like to note that page.html?q=1&q=2&q=3 is a valid url as well. This is a completely legitimate way of expressing an array in a query string. Your server technology will determine how exactly that is presented.
In Classic ASP, you check Response.QueryString("q").Count and then use Response.QueryString("q")(0) (and (1) and (2)).
Note that you saw this in your ASP.NET, too (I think it was not intended, but look):
Default.aspx?a=1;a=2&b=1&a=3
Request.QueryString["a"] = "1;a=2,3"
Request.QueryString["b"] = "1"
Notice that the semicolon is ignored, so you have a defined twice, and you got its value twice, separated by a comma. Using all ampersands Default.aspx?a=1&a=2&b=1&a=3 will yield a as "1,2,3". But I am sure there is a method to get each individual element, in case the elements themselves contain commas. It is simply the default property of the non-indexed QueryString that concatenates the sub-values together with comma separators.
I had the same issue. The URL that was hyperlinked was a third party URL and was expecting a list of parameters in format page.html?q=1,2,3 ONLY and the URL page.html?q=1%2C2%2C3 did not work. I was able to get it working using javascript. May not be the best approach but can check out the solution here if it helps anyone.
If you are sending the ENCODED characters to FLASH/SWF file, then you should ENCODE the character twice!! (because of Flash parser)