I don't understand why this token doesn't work. The token is this:
TEXT = [a-zA-Z0-9-_,;. \t\r\n\r\n]+
The token TEXT must recognize some text, with whitespace and new lines. For example the following token works:
TEXT = [a-zA-Z0-9-_,;. \t]+
How can i do to add the possibility of new lines in the token TEXT?
(Answered as a Community Wiki so that question shows an answer: See Question with no answers, but issue solved in the comments (or extended in chat) )
#user3640434 reported he found the correct regular expression for TEXT. It recognizes whitespace and new lines:
TEXT= ((\r\n|\r|\n)*[a-zA-Z0-9\-_,;. \t](\r\n|\r|\n)*)*
#Rob11311 Noted:
If you have a separate token rule WHITE =[ \t\n\r]+ does that work to match 1 or more whitespace items? Can you explain why you've added \r & \n twice into the char class? BTW you might want \f, \b and \v as whitespace to (formfeeds, backspace, vertical tab). Does char class help? [[:alnum:]-_,;.[:space:]] ?
Related
I have JSON strings that may contain \n, \t, which I don't want to save into database. strip_tags helps only with simple strings. I am using gsub(/(\\n)|(\\t)/, "").
I wonder if there is another Rails helper method or a better way to achieve this.
e.g
"[{\"type\":\"checkbox-group\",\"label\":\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nFill in\\nthe Gap (Please\\nfill in the blank box with correct wor\",\"name\":\"checkbox-group-1527245153706\",\"values\":[{\"label\":\"Option 1\",\"value\":\"option-1\",\"selected\":true}]},{\"type\":\"text\",\"label\":\"\\n\\t\\t\\n\\t\\n\\t\\n\\t\\t\\n\\t\\t\\t\\n\\t\\t\\t\\t\\n\\t\\t\\t\\t\\t\\n\\t\\t\\t\\t\\t\\t\\n\\t\\t\\t\\t\\t\\t\\t\\n\\t\\t\\t\\t\\t\\t\\t\\tWhat are the unique features of\\ne-commerce, digital markets, and\\ndigital goods? \\n\\t\\t\\t\\t\\t\\t\\t\\n\\t\\t\\t\\t\\t\\t\\n\\t\\t\\t\\t\\t\\n\\t\\t\\t\\t\\n\\t\\t\\t\\n\\t\\t\",\"className\":\"form-control\",\"name\":\"text-1527245426509\",\"subtype\":\"text\"}]"
You can make use of squish or squish!
" Some text \n\n and a tab \t\t with new line \n\n ".squish
#=> "Some text and a tab with new line"
Squish removes all the whitespace chars on both ends and grouping remaining whitespace chars (\n, \t, space) in one space
I think this one may help you,
JSON.parse(string).map{ |a| a['label'] = a['label'].squish; a}
I'm trying to figure out how to allow posting of special commands during statuses/update. the API documentation stats that if post tweets stating with "D ", "M " it will send a direct message to the user after the command.
I tried replacing the first character with a "no-break space" (%A0). However twitter translates these characters to �.
The only other option I can think of is creating a regex which strips out all potentially hazardous SMS code. This isn't ideal since I would prefer to keep the raw tweet. Such as tweeting: "D Day was an alright movie." would result in "Day was an alright movie."
I also tried entering a space before the post but it just got ignored and still sent a direct message. Is there some sort of escape character or cheat to make twitter ignore these commands?
Looks like I found the solution. There were a few issues. 1st was my percent encoding function wasn't accepting unicode characters so twitter was responding with a �. To resolve this I used the following code in VB.Net:
Regex.Replace(Uri.EscapeDataString(s), "[\!\*\'\(\)]", Function(m) Uri.HexEscape(Convert.ToChar(m.Value(0).ToString())))
The second issue is twitter treats all white space characters the same. So you can't cheat and use a character such as u+00A0 (No-break space) or any other character listed in:
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf (6.4)
http://en.wikipedia.org/wiki/Whitespace_character
Luckily twitter does not include zero-width spaces as a white space character. As well twitter actually allows for 141 characters to be posted through the API if using a zero-width space. This means you can add U+200B after the SMS command and before the space to prevent the SMS command from running. Such as:
Dim s as String = "M" & ChrW(8203) & " Username 140 character tweet posted here as a test. This is actually 141 characters because of the zero-width space. But accepted anyway."
Hope this helps!
Update:
For those who are looking for the regex code to do this, it's a 1 liner:
Regex.Replace(s, "^(ON|OFF|FOLLOW|F|UNFOLLOW|LEAVE|L|STOP|QUIT|END|CANCEL|UNSBSCRIBE|ARRET|D|M|RETWEET|RT|SET|WHOIS|W|GET|G|FAV|FAVE|FAVORITE|FAVORITE|\*|STATS|SUGGEST|SUG|S|WTF|HELP|INFO|AIDE|BLOCK|BLK|REPORT|REP)( )", Function(m) m.ToString.Replace(" ", ChrW(8203) & " "))
I accidentally hit the spacebar and wrote this:
lTTEvent .CustUpdateStatus := usUnchanged;
and was surprised to see that the compiler accepted the space in front of the dot (actually, any number of spaces).
Is the dot such a special character that the parser can interpret it correctly? How would that work in Pascal?
The parser first translates text to tokens. So the text:
lTTEvent .CustUpdateStatus := usUnchanged;
Is translated to the tokens:
identifier
period
identifier
becomes
identifier
semicolon
The space is a whitespace and it can have three functions:
separator between tokens (for example between an identifier and a keyword).
a literal space (in that case it is included in a string.
cosmetic.
The first and last function spaces are lost in the translation to tokens.
An identifier and a period don't have any characters in common so there is no way those can be confused so a space is not required but it still can be used.
short answer
'lTTEvent' and '.' are tokens. Tokens can (sometimes) be separated by whitespace.
I have some text with hard line breaks in it like this:
This should all be on one line
since it's one sentence.
This is a new paragraph that
should be separate.
I want to remove the single newlines but keep the double newlines so it looks like this:
This should all be on one line since it's one sentence.
This is a new paragraph that should be separate.
Is there a single regular expression to do this? (or some easy way)
So far this is my only solution which works but feels hackish.
txt = txt.gsub(/(\r\n|\n|\r)/,'[[[NEWLINE]]]')
txt = txt.gsub('[[[NEWLINE]]][[[NEWLINE]]]', "\n\n")
txt = txt.gsub('[[[NEWLINE]]]', " ")
Replace all newlines that are not followed by or preceded by a newline:
text = <<END
This should all be on one line
since it's one sentence.
This is a new paragraph that
should be separate.
END
p text.gsub /(?<!\n)\n(?!\n)/, ' '
#=> "This should all be on one line since it's one sentence.\n\nThis is a new paragraph that should be separate. "
Or, for Ruby 1.8 without lookarounds:
txt.gsub! /([^\n])\n([^\n])/, '\1 \2'
text.gsub!(/(\S)[^\S\n]*\n[^\S\n]*(\S)/, '\1 \2')
The two (\S) groups serve the same purposes as the lookarounds ((?<!\s)(?<!^) and(?!\s)(?!$)) in #sln's regexes:
they confirm that the linefeed really is in the middle of a sentence, and
they ensure that the [^\S\n]*\n[^\S\n]* part consumes any other whitespace surrounding the linefeed, making it possible for us to normalize it to a single space.
They also make the regex easier to read, and (perhaps most importantly) they work in pre-1.9 versions of Ruby that don't support lookbehinds.
There is more to formatting (turning off word wrap) than you think.
If the output is a result of a formatting operation, then you should go by
those rules to reverse engineer the original.
For instance, the test you have there is
This should all be on one line
since it's one sentence.
This is a new paragraph that
should be separate.
If you removed just the single newlines only, it would look like this:
This should all be on one line since it's one sentence.
This is a new paragraph thatshould be separate.
Also, other formatting such as intentional newlines will be lost, so something like:
This is Chapter 1
Section a
Section b
Turns into
This is Chapter 1 Section a Section b
Finding the newline in question is easy /(?<!\n)\n(?!\n)/
but, what do you replace it with.
Edit: Actually, its not that easy even to find standalone newlines, because visually they sit amongst hidden from view (horizontal) whitespaces.
There are 4 ways to go.
Remove newline, keep the surrounding formatting
$text =~ s/(?<!\s)([^\S\n]*)\n([^\S\n]*)(?!\s)/$1$2/g;
Remove newline and formatting, substitute a space
$text =~ s/(?<!\s)[^\S\n]*\n[^\S\n]*(?!\s)/ /g;
Same as above but ignore newline at beginning or end of string
$text =~ s/(?<!\s)(?<!^)[^\S\n]*\n[^\S\n]*(?!$|\s)/ /g;
$text =~ s/(?<!\s)(?<!^)([^\S\n]*)\n([^\S\n]*)(?!$|\s)/$1$2/g;
Example breakdown of regex (this is the minimum required just to isolate a single newline):
(?<!\s) # Not a whitespace behind us (text,number,punct, etc..)
[^\S\n]* # 0 or more whitespaces, but no newlines
\n # a newline we want to remove
[^\S\n]* # 0 or more whitespaces, but no newlines
(?!\s)/ # Not a whitespace in front of us (text,number,punct, etc..)
Well, there is this:
s.gsub /([^\n])\n([^\n])/, '\1 \2'
It won't do anything to leading or trailing newlines. If you don't need leading or trailing white space at all, then you will win with this variation:
s.gsub(/([^\n])\n([^\n])/, '\1 \2').strip
$ ruby -00 -pne 'BEGIN{$\="\n\n"};$_.gsub!(/\n+/,"\0")' file
This should all be on one line since it's one sentence.
This is a new paragraph thatshould be separate.
let's say that I have an XML file containing this :
<description><![CDATA[
<h2>lorem ipsum</h2>
<p>some text</p>
]]></description>
that I want to get and parse in ActionScript 2 as HTML text, and setting some CSS before displaying it. Problem is, Flash takes those whitespaces (line feed and tab) and display it as it is.
<some whitespace here>
lorem ipsum
some text
where the output I want is
lorem ipsum
some text
I know that I could remove the whitespaces directly from the XML file (the Flash developer at my workplace also suggests this. I guess that he doesn't have any idea on how to do this [sigh]). But by doing this, it would be difficult to read the section in the XML file, especially when lots of tags are involved and that makes editing more difficult.
So now, I'm looking for a way to strip those whitespaces in ActionScript. I've tried to use PHP's str_replace equivalent (got it from here). But what should I use as a needle (string to search) ? (I've tried to put in "\t" and "\r", don't seem to be able to detect those whitespaces).
edit :
now that I've tried to throw in newline as a needle, it works (meaning that newline successfully got stripped).
mystring = str_replace(newline, '', mystring);
But, newlines only got stripped once, meaning that in every consecutive newlines, (eg. a newline followed by another newline) only one newline can be stripped away.
Now, I don't see that this as a problem in the str_replace function, since every consecutive character other than newline get stripped away just fine.
Pretty much confused about how stuff like this is handled in ActionScript. :-s
edit 2:
I've tried str_replace -ing everything I know of, \n, \r, \t, newline, and tab (by pressing tab key). Replacing \n, \r, and \t seem to have no effect whatsoever.
I know that by successfully doing this, my content can never have real line breaks. That's exactly my intention. I could format the XML the way I want without Flash displaying any of the formatting stuff. :)
Several ways to approach this. Perhaps the simplest answer is, in one sense your Flash developer is probably right, and you should move your whitespace outside of the CDATA container. The reason being, many people (me at least) tend to assume that everything inside a CDATA is "real data", as opposed to markup. On the other hand, whitespace outside a CDATA is normally assumed to be irrelevant, so data like this:
<description>
<![CDATA[<h2>lorem ipsum</h2>
<p>some text</p>]]>
</description>
would be easier to understand and to work with. (The flash developer can use the XML.ignoreWhite property to ignore the whitespace outside the CDATA.)
With that said, if you're editing the XML by hand, then I can see why it would be easier to use the formatting you describe. However, if the extra whitespace is inside the CDATA, then it will inevitable be included in the String data you extract, so your only option is to grab the content of the CDATA and remove the whitespace afterwards.
Then your question reduces to "how do I strip leading/trailing whitespace from a String in AS2?". And unfortunately, since AS2 doesn't support RegEx there's no simple way to do this. I think your best option would be to parse through from the beginning and end to find the first/last non-white character. Something along these lines (untested pseudocode):
myString = stuffFromXML;
whitespace = " " + "\t" + "\n" + "\r" + newline;
start = 0;
end = myString.length;
while ( testString( myString.substr(start,1), whitespace ) ) { start++; }
while ( testString( myString.substr(end-1,1), whitespace ) ) { end--; }
trimmedString = myString.substring( start, end );
function testString( needle, haystack ) {
return ( haystack.indexOf( needle ) > -1 );
}
Hope that helps!
Edit: I notice that in your example you'd also need to remove tabs and whitespace within your text data. This would be tricky, unless you can guarantee that your data will never include "real" tabs in addition to the ones for formatting. No matter what you do with the CDATA tags, it would probably be wiser not to insert extraneous formatting inside your real content and then remove it programmatically afterward. That's just making your own life difficult.
Second edit: As for what character to remove to get rid of newlines, it depends partially on what characters are actually in the XML to begin with (which probably depends on what OS is running where the file is generated), and partially on what character the client machine (that's showing the flash) considers a newline. Lots of gory details here. In practice though, if you remove \r, \n, and \r\n, that usually does the trick. That's why I added both \r and \n to the "whitespace" string in my example code.
its been a while since I've tinkered with AS2.
someXML = new XML();
someXML.ignoreWhite = true;
if you wanted to str_replace try '\n'
Is there a reason that you are using cdata? Admittedly I have no idea what the best practice for this sort of this is, but I tend to leave them out and just have the HTML sit there inside the node.
var foo = node.childnodes.join("") parses it out just fine and I never seem to come across these whitespace problems.
I'm reading this over and over again, and if I'm interpreting you right, all you want to know how to do is strip certain characters (tabs and newlines) from a string in AS2, right? I cannot believe no one has given you the simple one line answer yet:
myString = myString.split("\n").join("");
That's it. Repeat that for \r, \n, and \t and all newlines and tabs will be gone. If you want it as an easy function, then do this:
function stripWhiteSpace(str: String) : String
{
return str.split("\r").join("").split("\n").join("").split("\t").join("");
}
That function won't modify your old string, it will return a new one without \r, \n, or \t. To actually modify the old string use that function like this:
myString = stripWhiteSpace(myString);