Is it possible for a person working with statistic to replace his specialized programs by F#? I'm thinking about SAS/SPSS mainly?
Any native support for it in F#?
I am not talking about the trivial things as standard deviation and the likes, but for example item-response modeling.
UPDATE : Dont't let the item-response modeling put you of! I don't even know it, just an example of things I know they do with SPSS to clarify it's about more advanced features.
Short : is there a way to use F# as your main statistical tool and replace SPSS all together?
Sadly, nothing comporable to combination of
R + PostgreSQL + Python/Java/Groovy/Scala/... + VisAD
Of course, there is nice http://www.codeplex.com/vslab instead of gnuplot
and some c# statistics code packaged in http://ta-lib.org/ http://www.alglib.net/
You can use R within F# with the type providers for R
see
http://blogs.msdn.com/b/dsyme/archive/2013/01/30/twelve-type-providers-in-pictures.aspx
and see
http://techblog.bluemountaincapital.com/2012/08/01/announcing-the-f-r-type-provider/
Here at BlueMountain we like to perform statistical analysis of data. The stats package R is great for doing that. We also like to use the data retrieval and processing capabilities of F#. F#’s interactive environment lends itself pretty well to data exploration, and we can also easily access our existing .NET-based libraries. Once we are done, we can build and release production-supportable applications.
Here is something maybe-useful:
http://fsmathtools.codeplex.com/
or
http://mathnetnumerics.codeplex.com/
Related
I am trying to build an iOS application. In one of the screens the user can type something in a search bar and I have to take same action for different spellings of the same word.
For eg: User can type "elephant" or "alephant" or "elefant". I have to take same action for all these three words.
Is there any library that identifies these words as similar ones ? I cannot use spellchecker as I need this in languages other than english also ..
I did some research and I found that there are some phonetic algorithms like Text::soundex for achieving this on server side. Wondering if any libraries there for iOS ?
Thanks in advance !!
A better alternative to Soundex would be Double Metaphone or, even better, Metaphone 3. You don't say what language you are using, but both of these algorithms are available in C++, C#, and Java
There's no soundex available in for example NSString, but if that's what you want, it's fairly easy to implement. Here's a—albeit horribly formatted—soundex NSString category from CocoaDev.
You could also use the Levenstein Distance algorithm to catch simple spelling errors. Also easy to implement (read the Wikipedia article for the details), but here's a NSString category for that.
Before you use these algorithms, normalize the input. There's the amazing CFStringTransform class in Core Foundation (see this great article about it on NSHipster—especially the last part about normalization) that automatically can transform different language inputs into normalized forms.
I'm trying to do some algorithm comparison for plagiarism. I've found many TEXT comparison for plagiarism.
But in an algorithm it's very different. Let's say that some algorithm uses an huge number of variables, functions and user defined structures. If some guy copy the source code from someone, he'll at least, change the variables and functions names. With an simple text comparison algorithm this difference in functions and variables letters will count as an "difference" making the algorithm gives an "false" for plagiarism.
What I want to do is "generalize" (I don't know if that's the right word) all the variables, functions and user-defined structures names in an C++ source code. So the varibles will be named like "a", "b", the same for functions "... fa(...)", "... fb(...)".
I have the c++ source algorithms in strings variables in PHP to be compared.
I know that many other things should be analysed for an accurate source code comparison, but that will be enough to me.
It's an interesting question. Depending on how complex the algorithm, however, it might be that variable names are what gives the plagiarism away. How many ways can you really code up a tree traversal for example?
I think there was a paper a few years ago on identifying coders through their style - looking at all the little things like whitespace, where {}s are placed, etc. Who knows but maybe that is the way to go, look for a negative match to the student's previous style rather than positive match to the known sources. Saying that, students aren't likely to have developed a very personal coding style at an early stage of learning.
One thought - what language are the examples written in? Can it be compiled? If you compile C and then do a binary comparison on the executables, then will identical programs with different local variable names have the exact same binary? (Global vars and functions wouldn't, though).
I've used MOSS in the past: http://theory.stanford.edu/~aiken/moss/ to detect plagiarized code. Since it works on a semantic level, it will detect the situations you presented above. The tool is language-aware, so comments are not considered in the analysis, and it goes a long way in detecting code that has been modified through simple search-and-replace of variable and/or function names.
Note: I used the tool a few years ago when I taught computer science in grad school, and it worked wonderfully in detecting code that had been yanked from the internet. Here is a well-documented account of similar application: http://fie2012.org/sites/fie2012.org/history/fie99/papers/1110.pdf
If you google "measure software similarity", you should find a few more useful hits: http://www.ics.heacademy.ac.uk/resources/assessment/plagiarism/detectiontools_sourcecode.html
is there a way to use calculations made in matrix language (matrix-end matrix) as macro variables later in calculations?
Let say I calculate chi^2 and pvalue in matrix language and then I want to use them as my new macro variables for, let say, printing information about if the statistics is significant or not.
Of course I can use OMS to solve my problem but I want to find out if there is a possible way to get variables from matrix language to syntax later on.
You might want to look into Python programmability instead of macro. It is much more powerful and flexible. You can read about it in the books and articles section of the SPSS Community website (www.ibm.com/developerworks/spssdevcentral). The site also provides the materials for getting started with programmability.
MATRIX can write datasets, which Python can read and manipulate - and it can even generate macro values from them.
HTH,
Jon Peck
Is there existing software for discriminative reranking, such as that used by the Charniak NLP parser, Shen, Sarkar, and Och's parser or Shen and Joshi's techniques? I'd like something that I can easily adapt for my own uses, which are similar to parse reranking.
Charniak-Johnson Reranking
The source code for the Charniak-Johnson (CJ) reranking parser is freely available, you can download a copy here.
The reranker is a separate code module that takes as input n-best lists of parses, so it's trivial to decouple it from the parsing front end.
SVM-rank
Alternatively, the package SVM-rank, from Thorsten Joachim's lab at Cornell, is a general purpose ranker. It might be easier to go with this package, if what you want to do deviates significantly from what's being done by the Charniak-Johnson parser.
I need to perform some basic OLS regression using F#. To do this I need some Linear Algebra functions, but I'm confused as to what's out there. I can't find any way to invert a matrix. There is some documentation for a library called Microsoft.FSharp.Math.LinearAlgebra, but I don't know if that exists anymore.
If you add the FSharp Powerpack to your project (in .NET references), you can use various functionality of the matrix library
edit: you also need to add the experimental library Fsharp.Powerpack.MathProviders, then you can call as follows
open Microsoft.FSharp.Math
let m = Matrix.create 10 10 1.2
let m2 = Experimental.LinearAlgebra.Inverse m
FlyingFrog do a Numerics library which contains Matrix inversion amongst many other functions.
Not sure which is preferable, that or the (apparently deprecated) 'experimental' code from the PowerPack. I guess you could always keep the source code for the managed bit of the PowerPack version in a safe place, still available here:
C:\Program Files\FSharp-1.9.6.2\source\fsharp\FSharp.PowerPack\math\lapack\linear_algebra_managed.fs.
I don't know; in the 1.9.6 version of F# I don't see anything offhand, the docs are here
http://research.microsoft.com/en-us/um/cambridge/projects/fsharp/manual/namespaces.html
and there is Matrix stuff in the Microsoft.FSharp.Math namespace in the FSharp.Powerpack.dll, but I don't see 'invert' offhand, and I don't know about the 'LinearAlgebra' stuff (deprecated? web search suggests it disappeared a few releases back).
Have you checked out this. It might help.