? value handling in Weka - machine-learning

How can I replace the '?' values in Weka.
I have a dataset. There are nominal values in a column which also have some values '?'. I tried to replace missing values with replacemissingvalues filter in Weka. This filter replaced those cells which did not have any values like blank cells but it could not replace '?'. How could I replace ? in Weka?
Screenshot of values with ? symbol.
Thank you.

The unofficial Weka package missing-values-imputation allows you to replace and inject missing values with various schemes.
In your case, you should be able to use the Regex injection scheme with .*\?.* as the regular expression (and updateHeader turned on), to replace the question marks with leading or trailing blanks (got added in release 2021.10.28).
Here is the full command-line of the MissingValuesInjection filter using the Regex injection:
weka.filters.unsupervised.attribute.MissingValuesInjection -algorithm "weka.filters.unsupervised.attribute.missingvaluesinjection.Regex -expression .*\\?.* -update-header -R first-last"

Its quite bit tricky with '?' values especially if any blank spaces before or after question mark ' ?' or '? ' won't be recognized as missing values by Weka. However you can try
At attribute level, use the option(when you right click) Replace values with... where you can replace '?' with blank value, prior applying missing values filter or directly replace '?' with the constant you have applied for filling the missing values.
Sometimes the value can have blank spaces ' ?' or '? ', so replace accordingly.
Use the filter Replace with missing value to replace '?' with blank cells. use this filter before using filter ReplaceMissingValues filter. However, from what I have seen you can randomly replace with missing value. Am not sure if specific values like '?', ' ?','? ' can be replaced with missing value(blank cell)
Best method is to opt for the first one, this you tube link provides 3 different ways to deal with missing value and shows the way how you can access Replace values with...

Related

How to remove a textual variable contained in parentheses in a filtered result on Google Sheets?

I would like to improve this filter so that it can remove a variable text contained in parentheses, as well as the parentheses themselves.
Like this :
Example
The starting point of my formula is :
=filter(C3:C;ISBLANK(B3:B))
I thought a "Substitute" could do the trick but I failed to use it properly, and I don't know if this function can be applied to variable text ?
Thanks in advance for your answers !
Try with this formula that removes every text in between brackets:
=INDEX(REGEXREPLACE(FILTER(C3:C;ISBLANK(B3:B)),"\(+.\) ",""))
you can use the SUBSTITUTE function in combination with a filter function. Here's how:
Select the cells you want to filter and remove the text contained in parentheses.
In a new column next to the data you want to filter, use the SUBSTITUTE function to remove the text contained in parentheses. The syntax for the SUBSTITUTE function is:
=SUBSTITUTE(A1, "(" & ")", "")
where A1 is the cell containing the data you want to filter and "(" & ")" is the text contained in parentheses that you want to remove.
Filter the data using the new column that contains the filtered text.
To remove the original data and keep only the filtered data, you can copy the filtered data and paste it over the original data, or delete the original data and keep only the filtered data.
Try with REGEXEXTRACT.
=FILTER(REGEXEXTRACT(C3:C;"\)(.*)");ISBLANK(B3:B))
Thank you for your answers ; unfortunately the formulas return errors when I apply them :
=SUBSTITUTE(A1, "(" & ")", "")
and
=INDEX(REGEXREPLACE(FILTER(C3:C;ISBLANK(B3:B)),"\(+.\) ",""))
gives this (In this two cases the error is not identified)
This formula has a different result:
=FILTER(REGEXEXTRACT(C3:C;"\)(.*)");ISBLANK(B3:B))
it gives that (Here the text variable has been deleted as desired, but the rest of the data is no longer filtered correctly)

How do you convert text like `'$71.4145` into a number you can do arithmetic with

The title says it all. One thing I want to avoid is long formulas. If it's more than a single function, something is clearly wrong since this should be a common use case.
I've tried TO_PURE_NUMBER and VALUE
Your question suggests that the value $71.4145 is not a number but a text string. That can happen if your spreadsheet locale is such that it expects comma as decimal mark, or expects a different currency symbol. It will also happen if you have formatted the value as plain text rather than currency.
To convert the text string $71.4145 into the number 71.4145 (seventy-one and change), use regexextract(), like this:
=iferror( value( regexextract( to_text(A2), "[\d.]+" ) ) )
Just use -- to suppress text to equivalent number values. Try-
=--A1
try:
=SUBSTITUTE(A1; "$"; )*1

Extract substring after '-' character in Google Sheets

I am using the following formula to extract the substring venue01 from column C, the problem is that when value string in column C is shorter it only extracts the value 1 I need it to extract anything straight after the - (dash) no matter the length of the value text in column c
={"VenueID";ARRAYFORMULA(IF(ISBLANK(A2:A),"",RIGHT(C2:C,SEARCH("-",C2:C)-21)))}
There is a much simpler solution using regular expressions.
=REGEXEXTRACT(A1,".*-(.*)")
In case you are no familiar with Regular Expressions what this means is, get me every string of characters ((.*)) after a dash (-).
Example
Reference
REGEXTRACT
Test regular expressions
Cheat sheet for regular expressions
To answer bomberjackets question in the comment of Raserhin:
To select the part of the string before the "-"
=REGEXEXTRACT(A1,"(.*)-.*")
EXAMPLE
example of code
Adding to your original formula. I think if you'd use RIGHT and inside it reverse the order of the string with ARRAY then that may work.
=Right(A1,FIND("-",JOIN("",ARRAYFORMULA(MID(A1,LEN(A1)-ROW(INDIRECT("1:"&LEN(A1)))+1,1))))-1)
It takes string from the right side up to X number of characters.
Number of character is fetched from reversing the text, then finding
the dash "-".
It adds one more +1 of the text as it will take out so it accounts
for the dash itself, if no +1 is added, it will show the dash on
the extracted string.
The REGEX on the other answer works great too, however, you can control a number of character to over or under trim. E.g. if there is a space after the dash and you would like to always account for one more char.

replace everything after a character in Google spreadsheet

I did some searching and in openoffice and excel it looks like you can simply add an * at the beginning or end of a character to delete everything before and after it, but in Google spreadsheet this isn't working. Does it support this feature? So if I have:
keyword USD 0078945jg .12 N N 5748 8
And I want to remove USD and everything after it what do I use? I have tried:
USD* and (USD*) with regular expressions checked
But it doesn't work. Any ideas?
The * quantifier just needs to be applied to a dot (.) which will match any character.
To clarify: the * wildcard used in certain spreadsheet functions (eg COUNTIF) has a different usage to the * quantifier used in regular expressions.
In addition to options that would be available in Excel (LEFT + FIND) pointed out by pnuts, you can use a variety of regex tools available in Google Sheets for text searching / manipulation
For example, RegexReplace:
=REGEXREPLACE(A1,"(.*)USD.*","$1")
(.*) <- capture group () with zero or more * of any character .
USD.* <- exact match on USD followed by zero or more * of any character .
$1 <- replace with match in first capture group
Please try:
and also have a look at.
For spaces within keyword I suggest a helper column with a formula such as:
=left(A1,find("USD",A1)-1)
copied down to suit. The formula could be converted to values and the raw data (assumed to be in ColumnA) then deleted, if desired.
To add to the answers here, you can get into trouble when there are special characters in the text (I have been struggling with this for years).
You can put a frontslash \ in front of special characters such as ?, + or . to escape them. But I still got stuck when there were further special characters in the text. I finally figured it out after reading find and replace in google sheets with regex.
Example: I want to remove the number, period and space from the beginning of a question like this: 1. What is your name?
Go to Edit → Find and replace
In the Find field, enter the following: .+\. (note: this includes a space at the end).
Note: In the Find and replace dialogue box, be sure to check "Search using regular expressions" and "match case". Leave the Replace field blank.
The result will be this text only: What is your name?

extract number from cell in openoffice calc

I have a column in open office like this:
abc-23
abc-32
abc-1
Now, I need to get only the sum of the numbers 23, 32 and 1 using a formula and regular expressions in calc.
How do I do that?
I tried
=SUMIF(F7:F16,"([:digit:].)$")
But somehow this does not work.
Starting with LibreOffice 6.4, you can use the newly added REGEX function to generically extract all numbers from a cell / text using a regular expression:
=REGEX(A1;"[^[:digit:]]";"";"g")
Replace A1 with the cell-reference you want to extract numbers from.
Explanation of REGEX function arguments:
Arguments are separated by a semicolon ;
A1: Value to extract numbers from. Can be a cell-reference (like A1) or a quoted text value (like "123abc"). The following regular expression will be applied to this cell / text.
"[^[:digit:]]": Match every character which is not a decimal digit. See also list of regular expressions in LibreOffice
The outer square brackets [] encapsulate the list of characters to search for
^ adds a NOT, meaning that every character not included in the search list is matched
[:digit:] represents any decimal digit
"": replace matching characters (every non-digit) with nothing = remove them
"g": replace all matches (don't stop after the first non-digit character)
Unfortunately Libre-Office only supports regex in find/replace and in search.
If this is a once-only deal, I would copy column A to column to B, then use [data] [text to columns] in B and use the - as a separator, leaving you with all the text in column B and the numbers in column C.
Alternatively, you could use =Right(A1,find("-",A1,1)+1) in column B, then sum Column C.
I think that this is not exactly what do you want, but maybe it can help you or others.
It is all about substring (in Calc called [MID][1] function):
First: Choose your cell (for example with "abc-23" content).
Secondly: Enter the start length ("british" --> start length 4 = tish).
After that: To print all remaining text, you can use the [LEN][2] function (known as length) with your cell ("abc-23") in parameter.
Code now looks like this:
D15="abc-23"
=MID(D15; 5; LEN(D15))
And the output is: 23
When you edit numbers (in this example 23), no problem. However, if you change anything before (text "abc-"), the algorithm collapses because the start length is defined to "5".
Paste the string in a cell, open search and replace dialog (ctrl + f) extended search option mark regular expression search for ([\s,0-9])([^0-9\s])+ and replace it with $1
adjust regex to your needs
I didn't figure out how to do this in OpenOffice/LibreOffice directly. After frustrations in searching online and trying various formulas, I realised my sheet was a simple CSV format, so I opened it up in vim and used vim's built-in sed-like feature to find/replace the text in vim command mode:
:%s/abc-//g
This only worked for me because there were no other columns with this matching text. If there are other columns with the same text, then the solution would be a bit more complex.
If your sheet is not a CSV, you could copy the column out to a text file and use vim to find/replace, and then paste the data back into the spreadsheet. For me, this was a lot less frustrating than trying to figure this out in LibreOffice...
I won't bother with a solution without knowing if there really is interest, but, you could write a macro to do this. Extract all the numbers and then implement the sum by checking for contained numbers in the text.

Resources