I am using DDMathParser in my app, and have recently come across the need to get occurrences of any group of numbers within a () parentheses bracket thingy (very highly technical!). For example, I would need to get (6+5) out of 6+7/8(6+5). Specifically, I would like to be able to do this so that I can make (56+9)sqrt compile just as well as sqrt(56+9). Any help?
P.S. I know that the maker of DDMathParser is often sighted in this neck of the woods. I am secretly hoping that he will come to the rescue and either fix my problem so I can implement it myself or him make it part of DDMathParser! :)
So, I've thought a lot about this question since you posted it a month ago. From what I understand, you're constructing a string as the user clicks/taps buttons.
I think this is your problem.
As the user taps buttons, you should be constructing (or modifying) DDExpression objects. This is the "pure" format of a math expression, whereas a string is lossy and difficult to manipulate. The string you show to the user should be generated from the DDExpression tree you're building.
This is a complex problem, and I'm still not entirely sure how I would go about implementing this, but this is the root of how I'd do it. I would not just construct a string based on what the user types.
Related
For those who are not familiar with what a homophone is, I provide the following examples:
our & are
hi & high
to & too & two
While using the Speech API included with iOS, I am encountering situations where a user may say one of these words, but it will not always return the word I want.
I looked into the [alternativeSubstrings] (link) property wondering if this would help, but in my testing of the above words, it always comes back empty.
I also looked into the Natural Language API, but could not find anything in there that looked useful.
I understand that as a user adds more words, the Speech API can begin to infer context and correct for these, but my use case will not work well with this since it will often only want one or two words at most, limiting the effectiveness of context.
An example of contextual processing:
Using the words above on their own, I get these results:
are
hi
to
However, if I put together the following sentence, you can see they are all wrong:
I am too high for our ladder
Ideally, I would either get a list back containing [are, our], [to, too, two], [hi, high] for each transcription segment, or would have a way to compare a string against a function that supports homophones.
An example of this would be:
if myDetectedWord == "to" then { ... }
Where myDetectedWord can be [to, too, two], and this function would return true for each of these.
This is a common NLP dilemma, and I'm not so sure what might be your desired output in this application. However, you may want to bypass this problem in your design/architecture process, if possible and if you could. Otherwise, this problem is to turn into a challenge.
Being said that, if you wish to really get into it, I like this idea of yours:
string against a function
This might be more efficient and performance friendly.
One way, I'd be liking to solve this problem would be though RegEx processing, instead of using endless loops and arrays. You could maybe prototype loops and arrays to begin with and see how it works, then you might want to use regular expression for gaining performance.
You could for instance define fixed arrays in regular expressions and quickly check against your string (word by word, maybe using back-referencing) and you can add many boundaries in your expressions for string processing, as you wish.
Your fixed arrays also can be designed based on probabilities of occurring certain words in certain part of a string. For instance,
^I
vs
^eye
The probability of I being the first word is much higher than that of eye.
The probability of I in any part of a string is higher than that of eye, also.
You might want to weight words based on that.
I'd say the key would be that you'd narrow down your desired outputs as focused as possible and increase accuracy, [maybe even with 100 words if possible], if you wish to have a good/working application.
Good project though, I hope you like/enjoy the challenge.
I currently work on a personal writing project which has ended up with me maintaining a few different versions due to the differences of the relevant platforms and output formats I want to support that are not trivially solved. After several instances of me glancing at pandoc and the sheer forest that it represents, I have concluded mere templates don't do what I need, and worse, that I seem to need a combination of a custom filter and writer... suffice to say: messing with the AST is where I feel way out of my depth. Enough so that, rather than asking specific questions of 'how do I do X' here, this is a question of 'is X the right way to go about it, or what is the proper way to do it, and can you give an example of how it ties together?'... so if this question is rather lengthy: my apologies.
My current goal is to have custom markup like the following which is supposed to 'track' which character says something:
<paul|"Hi there">
If I convert to HTML, I'd want something similar to:
<span class="speech paul">"Hi there"</span>
to pop out (and perhaps the <p> tags), whereas if it is just pure markdown / plain text, I'd want it to silently disappear:
"Hi there"
Looking at the JSON AST structures I've studied, it would make sense that I'd want a new structure type similar to the 'Emph' tag called 'Speech' which allows whole blobs of text to be put inside of it with a bit of extra information attached (the person speaking). So something like this:
{"t":"Speech","speaker":"paul","c":[ ... ] }
Problem #1: At the point a lua-filter sees the document, it is obviously already distilled to an AST. This means replacing the items in a manner similar to what most macro expander samples do cannot really work since it would require reading forward. With this method, I just replace bits and pieces in place (<NAME| becomes a StartSpeech and the first solitary > that follows becomes an EndSpeech, but that would make malformed input a bigger potential problem because of silent-ish failures. Additionally, these tags would be completely out of sorts with how an AST is supposed to look.
To complicate matters even further, some of my characters end up learning a secondary language throughout the story, for which I apply a different format that contains a simplified understanding of the spoken text with perspective-characters understanding of what was said. Example:
<paul|"Heb je goed geslapen?"|"Did you ?????">
I could probably add a third 'UnderstoodSpeech' group to my filter, but (problem #2) at this point, the relationship between the speaker, the original speech, and the understood translation is completely gone. As long as the final documents need these values in these respective orders and only in these orders, it is fine... but what if I want my HTML version to look like
"Did you?????"
with a tool-tip / hover-over effect containing the original speech? That would be near impossible to achieve because the AST does not contain that kind of relational detail.
Whatever kind of AST I create in the filter is what I need to understand in my custom writer. Ideally, I want to re-use as much stock functionality of pandoc as possible for the writer, but I don't even know if that is feasible at this point.
So now my question: could someone with great pandoc understanding please give me an example on how to keep relevant data-bits together and apply them in the correct manner? By this I mean show a basic example of what needs to be put in the lua-filter and lua-writer scripts in the following toolchain
[CUSTOMIZED MARKDOWN INPUT] -> lua-filter -> lua-writer -> [CUSTOMIZED HTML5 OUTPUT]
I'm working on an iOS app that involves user input, and I'd like to keep it kid-friendly. One of the main features of the app is that user inputted titles and phrases can be shown to everyone who uses the app.
When a user creates a new title I want to verify that it is safe-for-work. My initial thought was just to have a list of all profane words and verify that none of them exist in the title:
for bad_word in list_of_bad_words:
if bad_word in user_inputted_title:
// Complain to user!
// Title is okay.
I imagine that there must be libraries or best practices for doing this. People could easily substitute numbers for letters, and I'm sure there are sequences of SFW words that create inappropriate phrases.
Can anyone suggest a better way of doing this? Specifically, if there are any Swift tools that would be awesome!
There are some cocoapods for this:
https://github.com/IslandOfDoom/IODProfanityFilter
https://github.com/MaxKramer/SCRProfanityChecker
I haven't used either of these personally, but I hope these can be a good starting point. The first one replaces any profanity with asterisks, and the second can give you the range of the profanity so you can replace it with your own filler. Good luck.
I am looking to write a basic profanity filter in a Rails based application. This will use a simply search and replace mechanism whenever the appropriate attribute gets submitted by a user. My question is, for those who have written these before, is there a CSV file or some database out there where a list of profanity words can be imported into my database? We are submitting the words that we will replace the profanities with on our own. We more or less need a database of profanities, racial slurs and anything that's not exactly rated PG-13 to get triggered.
As the Tin Man suggested, this problem is difficult, but it isn't impossible. I've built a commercial profanity filter named CleanSpeak that handles everything mentioned above (leet speak, phonetics, language rules, whitelisting, etc). CleanSpeak is capable of filtering 20,000 messages per second on a low end server, so it is possible to build something that works well and performs well. I will mention that CleanSpeak is the result of about 3 years of on-going development though.
There are a few things I tell everyone that is looking to try and tackle a language filter.
Don't use regular expressions unless you have a small list and don't mind a lot of things getting through. Regular expressions are relatively slow overall and hard to manage.
Determine if you want to handle conjugations, inflections and other language rules. These often add a considerable amount of time to the project.
Decide what type of performance you need and whether or not you can make multiple passes on the String. The more passes you make the slow your filter will be.
Understand the scunthrope and clbuttic problems and determine how you will handle these. This usually requires some form of language intelligence and whitelisting.
Realize that whitespace has a different meaning now. You can't use it as a word delimiter any more (b e c a u s e of this)
Be careful with your handling of punctuation because it can be used to get around the filter (l.i.k.e th---is)
Understand how people use ascii art and unicode to replace characters (/ = v - those are slashes). There are a lot of unicode characters that look like English characters and you will want to handle those appropriately.
Understand that people make up new profanity all the time by smashing words together (likethis) and figure out if you want to handle that.
You can search around StackOverflow for my comments on other threads as I might have more information on those threads that I've forgotten here.
Here's one you could use: Offensive/Profane Word List from CMU site
Based on personal experience, you do understand that it's an exercise in futility?
If someone wants to inject profanity, there's a slew of words that are innocent in one context, and profane in another so you'll have to write a context parser to avoid black-listing clean words. A quick glance at CMU's list shows words I'd never consider rude/crude/socially unacceptable. You'll see there are many words that could be proper names or nouns, countries, terms of endearment, etc. And, there are myriads of ways to throw your algorithm off using L33T speak and such. Search Wikipedia and the internets and you can build tables of variations of letters.
Look at CMU's list and imagine how long the list would be if, in addition to the correct letter, every a could also be 4, o could be 0 or p, e could be 3, s could be 5. And, that's a very, very, short example.
I was asked to do a similar task and wrote code to generate L33T variations of the words, and generated a hit-list of words based on several profanity/offensive lists available on the internet. After running the generator, and being a little over 1/4 of the way through the file, I had over one million entries in my DB. I pulled the plug on the project at that point, because the time spent searching, even using Perl's Regex::Assemble, was going to be ridiculous, especially since it'd still be so easy to fool.
I recommend you have a long talk with whoever requested that, and ask if they understand the programming issues involved, and low-likelihood of accuracy and success, especially over the long-term, or the possible customer backlash when they realize you're censoring them.
I have one that I've added to (obfuscated a bit) but here it is: https://github.com/rdp/sensible-cinema/blob/master/lib/subtitle_profanity_finder.rb
I'm making a web app where the point is to change a given word by one letter. For example, if I make a post by selecting the word: "best," then the first reply could be "rest," while the one after that should be "rent," "sent", etc. So, the word a user enters must have changed by one letter from the last submitted word. It would be constantly evolving.
Right now you can make a game and respond just by typing a word. I coded up a custom validation using functionality from the Amatch gem:
http://flori.github.com/amatch/doc/index.html
Posts have many responses, and responses belong to a post.
here's the code:
def must_have_changed_by_one_letter
m = Amatch::Sellers.new(title.strip)
errors.add_to_base("Sorry, you must change the last submitted word by one letter")
if m.match(post.responses.last.to_s.strip) != 1.0
end
When I try entering a new response for a test post I made (original word "best", first response is "rest") I get this:
ActiveRecord::RecordInvalid in ResponsesController#create
Validation failed: Sorry, you must change the last submitted word by one letter
Any thoughts on what might be wrong?
Thanks!
Looks like there are a couple of potential issues here.
For one, is your if statement actually on a separate line than your errors.add_to_base... statement? If so, your syntax is wrong; the if statement needs to be in the same line as the statement it's modifying. Even if it is actually on the correct line, I would recommend against using a trailing if statement on such a long line; it will make it hard to find the conditional.
if m.match(post.responses.last.to_s.strip) != 1.0
errors.add_to_base("Sorry, you must change the last submitted word by one letter")
end
Second, doing exact equality comparison on floating point numbers is almost never a good idea. Because floating point numbers involve approximations, you will sometimes get results that are very close, but not quite exactly equal, to a given number that you are comparing against. It looks like the Amatch library has several different classes for comparing strings; the Sellers class allows you to set different weights for different kinds of edits, but given your problem description, I don't think you need that. I would try using the Levenshtein or Hamming distance instead, depending on your exact needs.
Finally, if neither of those suggestions work, try writing out to a log or in the response the exact values of title.strip and post.responses.last.to_s.strip, to make sure you are actually comparing the values that you think you're comparing. I don't know the rest of your code, so I can't tell you whether those are correct or not, but if you print them out somewhere, you should be easily able to check them yourself.