I am working on an email client, and currently trying to render message body. In those cases when MIME object is of type ALTERNATIVE, we have two representations of the same data, namely HTML and plain text representation.
Now, there are several questions regarding this.
Question 1
What factors do we consider when deciding which form - HTML or plain text - to render?
Question 2
Are there any devices that are not able to render html or plain text?
Question 3
Can we say that we should always look for HTML and try to render it, and only in case of failure turn to plain text alternative?
The sender sends alternatives in order of preference, best last.
Yes, particularly if you ask about rendering well enough to be pleasantly usable.
"We" can say whatever "we" want, and epecially so if the subject is as fuzzy as what constitutes a "rendering failure".
Related
Why Spring Security doesn't provide any XSS filter to clean the form input values?
Accordingly to this ticket, such XSS filter is a low priority:
https://jira.spring.io/browse/SEC-2167?jql=text%20~%20%22xss%22
(although the ticket speaks only about URL querystring. Sanitizing POST params would be also required)
In my opinion it would be really useful that spring would provide such a filter instead of building your own. This filter it's a recurrent problem.
XSS is best handled at output stage via the use of encoding. That is, store everything in your database as is, and yes storing <script> is fine, however once output, encode correctly for the context it is output in. For HTML this would be <script>, however if your output context was plain text you would just output as is <script> (assuming the same character set encoding is used). Side note: Use parameterised queries or equivalent for storing in your database to avoid SQL injection, however the text stored should exactly match what was entered.
Microsoft attempts to block inputs that look like XSS via their request validation feature in ASP.NET. However, this isn't very effective and flaws are found quite often. Similar approaches from other frameworks are doomed to fail.
The reason that this is much better is that it makes things much more simple. Imagine if StackOverflow didn't allow HTML or script tags - the site would not be functional as a place for people to post code snippets.
You can use input validation as second line of defence. For example, if you are asking the user to enter their car registration you would only want to allow alphanumerics and space to be entered. However, for more complex fields it is often difficult to restrict input to a safe set as output context is unknown at this stage.
Say your language filtered < and > characters. However you were outputting user input into the following context.
<img src="foo.jpg" alt="USER-INPUT" />
An XSS attack is possible by entering " onmouseover="alert('xss') because it would be rendered as
<img src="foo.jpg" alt="" onmouseover="alert('xss')" />
Similar problems would ensue if you were outputting to JavaScript server-side. This is why it should be up to the developer to select the correct encoding type when using user controlled values.
There is a webpage parser, which takes a page contains several tags, in a certain structure, where divs are badly nested. I need to extract a certain div element, and copy it and all its content to a new html file.
Since I am new to lua, I may need basic clarification for things might seem simple.
Thanks,
The ease of extraction of data is going to largely depend on the page itself. If the page uses the exact same tag information throughout its entirety, it'll be much more difficult to extract than it would if it has named tags.
If you're able to find a version of the page that returns json format, then you're that much better off. Here's a snippet of code on something I wrote to grab definitions from a webpage that did not have json format:
local actualword, definition = string.match(wayup,"<html.-<td class='word'>%c(.-)%c</td>.-<div class=\"definition\">(.-)</div>")
Essentially, this code searched down the page until it found the class "word", and took the word after it (%c is the pattern for control characters). It continued on to "definition" and captured that, as well.
As you can see, it's a bit convoluted, but I had the luck of having specifically named tags for what I wanted.
This is edited to fit your comment. As a side note that I should have mentioned before, if you're familiar with regular expressions, you can use its model to capture what you need. In this case, it's capturing the string in its totality:
local data = string.match(page, "(<div id=\"aa\"><div>.-</div>.-</div>)")
It's rarely the fault of the language, but rather the webpage itself, that makes it hard to data mine anything. Since webpages could literally have hundreds of lines of code, it's hard to pinpoint exactly what you want without coming across garbage information. It's why I prefer a simplified result such as json, since Lua has a json module that can encode/decode and you can get your precise information.
In my rails 4 app i want to add comments to my articles, but i want to add functional as most forum-engines do (like SMF), and i need to add bb-code for it.
Are there any good gem for it? With rails 4 support? How then in controller i can translate [quote] to some div with some style?
Also how is it good to store html data in database?
For example if i use haml, and somebody post comment as
- current_user.id
or something similar to this, how to secure my app from "bad boys" ? Sure i can change comments system to something like: quote_parent_id, but if i have multiple quotes in one comment? so it is hard to realise, better is to store html, but to secure it somehow.
Could i do this? And how? Please give good ideas, tutorials, gem-links.
Look into https://github.com/veger/ruby-bbcode
Since it converts to HTML and does not excecute user input as Ruby code - you'll be fairly safe. However, I havent tried the gem and its possible it introduces some XSS vulnerabilities.
Have you considered Markdown as an option?
You should also look into https://github.com/asceth/bbcoder ( I should note I am the original author ).
In the controller, changing a string such as "[quote=user]My post of epic importance[/quote]" into a div etc is just doing:
# assume params[:comment] is the text you are converting
params[:comment].bbcode_to_html
As for storing html in a database, there is no right or wrong answer. If you want to allow users to edit their posts later then I would lean towards not storing the html version but storing their original bbcode version. This way when you allow them to edit you aren't having to convert html back to bbcode.
To make sure you aren't open to XSS and other attacks I recommend combining other gems like sanitize.
Sanitize.clean(text.to_s).bbcode_to_html
Some more notes:
Multiple tags and nested tags are parsed as they are seen without any additional steps required. So a comment or post with lots of bbcode tags, multiple quotes, b tags or anything else is dealt with by just calling bbcode_to_html on the variable/string.
If a user tries to use haml in their post it should appear as-is. haml shouldn't try to eval the string unless you specifically tell it to which I'm not even sure how to do that unless haml as a special filter or operator.
My Rails app processes incoming emails by splitting them into multiple lines. This is what I currently use on the plain text version of the body: lines = email.body.split("\n")
This works well unless the sentences are longer than ~74 characters as most email clients will automatically add a line break per RFC 2822.
Example email: https://gist.github.com/marckohlbrugge/39c17b928eb17d330d63
Looking at the plain text part there seems to be no way to discern between a line break added by the user versus the email client. You could ignore any line break happening at the 75th position, but I think there might be a chance of false positives. (I could be wrong.)
The HTML part has all the information we need, but I'm not sure about a universal way to process this. Is replacing every div and br with a newline and then stripping al other HTML elements enough? What about all the other block-element tags? What about inline elements styled as block-elements? What if an email doesn't have an HTML part?
I did find some interesting code examples in Convert HTML to plain text (with inclusion of s), but replacing a list of html tags with newlines doesn't seem like a complete (exhaustive) solution.
Is it worth looking at something like this mail library as they've probably already thought about the edge cases? ;)
Can I use ActionView::Helpers::SanitizeHelper#sanitize on user-entered text that I plan on showing to other users? E.g., will it properly handle all cases described on this site?
Also, the documentation mentions:
Please note that sanitizing
user-provided text does not guarantee
that the resulting markup is valid
(conforming to a document type) or
even well-formed. The output may still
contain e.g. unescaped ’<’, ’>’, ’&’
characters and confuse browsers.
What's the best way to handle this? Pass the sanitized text through Hpricot before displaying?
Ryan Grove's Sanitize goes a lot farther than Rails 3 sanitize. It ensures the output HTML is well-formed and has three built-in whitelists:
Sanitize::Config::RESTRICTED
Allows only very simple inline formatting markup. No links, images, or block elements.
Sanitize::Config::BASIC
Allows a variety of markup including formatting tags, links, and lists. Images and tables are not allowed, links are limited to FTP, HTTP, HTTPS, and mailto protocols, and a attribute is added to all links to mitigate SEO spam.
Sanitize::Config::RELAXED Allows an even wider variety of markup than BASIC, including images and tables. Links are still limited to FTP, HTTP, HTTPS, and mailto protocols, while images are limited to HTTP and HTTPS. In this mode, is not added to links.
Sanitize is certainly better than the "h" helper. Instead of escaping everything, it actually allows the html tags that you specify. And yes, it does prevent cross-site scripting because it removes javascript from the mix entirely.
In short, both will get the job done. Use "h" when you don't expect anything other than plaintext, and use sanitize when you want to allow some, or you believe people may try to enter it. Even if you disallow all tags with sanitize, it'll "pretty up" the code by removing them instead of escaping them as "h" does.
As for incomplete tags: You could run a validation on the model that passes html-containing fields through hpricot, but I think this is overkill in most applications.
The best course of action depends on two things:
Your rails version (2.x or 3.x)
Whether your users are supposed to enter any html at all on the input or not.
As a general rule, I don't allow my users to input html - instead I let them input textile.
On rails 3.x:
User input is sanitized by default. You don't have to do anything, unless you want your users to be able to send some html. In that case, keep reading.
This railscast deals with XSS attacks on rails 3.
On rails 2.x:
If you don't allow any html from your users, just protect your output with the h method, like this:
<%= h post.text %>
If you want your users to send some html: you can use rails' sanitize method or HTML::StathamSanitizer