How to detect recurrent patterns in a textfile - parsing

Question

To simplify a function with many terms, a program would be useful that searches for patterns in a file and arranges them in a ranking list. I can imagine that this is an elaborate process, but I'm sure there are people who have built something like this.
An example of a text:
sin(t1)*cos(t1)*t1+t1-sin(t1)*sin(t1-pi)
This should give me such an output like this (min. 2 letters):
6x: "t1"
4x: "(t1"
3x: "n(t1"
3x: "sin"
3x: "sin("
2x: "sin(t1)"
etc.
Does this problem have a name (which I don't know)? Is there a known algorithm that could solve the problem for me?

Answer 1

I have written a small program with QT, which fulfills the task. The approach is to try everything. To solve my problem, it will probably take a few days, because the text files are very large.
If I have the following text as input ("text.txt"):
sin(t1)*cos(t1)*t1+t1-sin(t1)*sin(t1-pi)
I have with the parameters: length 2-5, minimum occurrence: 3
following result:
t1 6
(t 4
(t1 4
si 3
in 3
n( 3
1) 3
)* 3
sin 3
in( 3
n(t 3
t1) 3
1)* 3
sin( 3
in(t 3
n(t1 3
(t1) 3
t1)* 3
sin(t 3
in(t1 3
(t1)* 3
Code:
#include <QCoreApplication>
#include <qdebug.h>
#include <qstring.h>
#include <qfile.h>
#include <qtextstream.h>
int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
QString * wholefile = new QString;
uint64_t minchar = 2;
uint64_t maxchar = 5;
uint64_t min_occur = 3;
QFile file("text.txt");
if(!file.open(QIODevice::ReadOnly)) {
qDebug()<<"error reading file";
}
QTextStream in(&file);
while(!in.atEnd()) {
QString line = in.readLine();
wholefile->append(line);
}
file.close();
QStringList * allpatterns = new QStringList;
for(uint64_t i=minchar; i<=maxchar;i++){
for(uint64_t pos=0; pos<wholefile->length()-i;pos++){
QString pattern = wholefile->mid(pos,i);
if(allpatterns->contains(pattern)==0){
allpatterns->append(pattern);
}
}
}
uint64_t * strcnt = new uint64_t[allpatterns->length()];
uint64_t maximum_cnt = 0;
QStringList * interestingpatterns = new QStringList;
uint64_t nr_of_patterns = 0;
for(uint64_t i=0; i<allpatterns->length();i++){
QString str = allpatterns->at(i);
strcnt[nr_of_patterns] = wholefile->count(str);
if(strcnt[nr_of_patterns]>=min_occur){
if(strcnt[nr_of_patterns]>maximum_cnt){
maximum_cnt = strcnt[nr_of_patterns];
}
interestingpatterns->append(str);
nr_of_patterns++;
}
}
/* display result*/
QFile file2("out.txt");
if (!file2.open(QIODevice::WriteOnly | QIODevice::Text))
qDebug()<<"error writing file";
QTextStream out(&file2);
uint64_t current_max = maximum_cnt;
while(current_max>=min_occur){
for(uint64_t i=0; i<interestingpatterns->length();i++){
if(strcnt[i]==current_max){
QString str = interestingpatterns->at(i);
qDebug()<<str<<strcnt[i];
out <<str<<" "<< QString::number(strcnt[i])<<"\n";
}
}
current_max--;
}
file2.close();
return a.exec();
}