Please advice how to transform the original dataset to the expected output as shown below on SPSS Modeler?
It is expected to merge the values of all rows shared the same ID No.
If SPSS syntax is availabel to you, the following aggregation command should do the trick:
aggregate outfile=* /break=ID /date_1 from_1 to_1 date_2 from_2 to_2=
max(date_1 from_1 to_1 date_2 from_2 to_2)
Related
I am trying to plot the TP99 for all the models that do not start with the word "test".
So I created the following query:
histogram_quantile(0.99,
sum by (m_application)(
rate(
model_execution_duration_seconds_percentile_bucket{
m_account="test-account-number", kubernetes_cluster="cluster01", m_application!~"^(test).*", method="POST"
}[5m]
)
)
by (le, m_application)
)
I am certain that the inner query is correct which plots the rate for all the models that do not start with the pattern test* (has been verified).
The error I am receiving is "1:279: parse error: unexpected <by>". Not exactly sure what I should do to fix this issue. Could someone help me with this issue?
you have two by clauses in your query:
histogram_quantile(0.99,
sum by (m_application)( <------------ here is the first
rate(
model_execution_duration_seconds_percentile_bucket{
m_account="test-account-number", kubernetes_cluster="cluster01", m_application!~"^(test).*", method="POST"
}[5m]
)
)
by (le, m_application) <------------ here is the second
)
Please note, you also need to aggregate by le in order to calculate the histogram. So removing the first one will solve your problem.
I'm trying to understand how to visualise the sum of column percentages in some tabulations of multiple variables.
suppose that i have defined the variable $q12 as a multiple response set of categorical values of the variables sq12m1 sq12m2 sq12m3 sq12m4 sq12m5.
i could have cases with values only in sq12m1 or cases with values in all of those.
if i want to see how many times any brand appear in any of those sq12m1 to sq12m5 i am using this:
CTABLES
/VLABELS VARIABLES=$q12 DISPLAY=DEFAULT
/TABLE $q12 [C][COUNT F40.0, COLPCT.COUNT PCT40.1]
/CATEGORIES VARIABLES=$q12 ORDER=A KEY=VALUE EMPTY=INCLUDE TOTAL=YES POSITION=AFTER
MISSING=EXCLUDE.
and it will generate this:
how can i sum the column percentages? using this syntax the total is always 100%, i would like to visualise the sum (which in this case is 215.10%) which represents the average number of mentions...
do you know how to do it?
thanks!!!
Only one thing you need to change in your syntax, in the /TABLE sub-command:COLPCT.RESPONSES.COUNT instead of COLPCT.COUNT:
CTABLES
/VLABELS VARIABLES=$q12 DISPLAY=DEFAULT
/TABLE $q12 [C][COUNT F40.0, COLPCT.RESPONSES.COUNT PCT40.1]
/CATEGORIES VARIABLES=$q12 ORDER=A KEY=VALUE EMPTY=INCLUDE TOTAL=YES POSITION=AFTER
MISSING=EXCLUDE.
I am trying collapse rows of a dataframe based on a key. My file is big and pandas throws a memory error. I am currently trying to use dask. I am attaching the snippet of the code here.
def f(x):
p = x.groupby(id).agg(''.join).reset_index()
return p
metadf = pd.DataFrame(columns=['c1','p1','pd1','d1'])
df = df.groupby(idname).apply(f, meta=metadf).reset_index().compute()
p has the same structure as metadf. The shape of both the dataframes are same.
When I execute this, I get the following error:
"ValueError: Length mismatch: Expected axis has 6 elements, new values have 5 elements"
What am I missing here? Is there any other way to collapse rows based on a key in dask?
The task in hand, to do the following sample in a dask dataframe
Input csv file :
key,c1,c2,c3......,cn
1,car,phone,cat,.....,kite
2,abc,def,hij,.......,pot
1,yes,no,is,.........,hello
2,hello,yes,no,......,help
Output csv file:
key,c1,c2,c3,.......,cn
1,caryes,phoneno,catis,.....,kitehello
2,abchello,defyes,hijno,....,pothelp
In this case meta= corresponds to the output of df.groupby(...).apply(f) and not just to the output of f. Perhaps these differ in some subtle way?
I would address this by first not providing meta= at all. Dask.dataframe will give you a warning asking you to be explicit but things should hopefully progress anyway if it is able to determine the right dtypes and columns by running some sample data through your function.
When I run this syntax in SPSS:
output modify
/select all except (Tables)
/deleteobject delete=yes.
my custom tables still get deleted. Do you have any idea whether this is a bug or I am doing something wrong?
Many thanks in advance!
TABLES is a generic term for all objects of type table, which includes custom tables output. You can do what you want with OMS using syntax like this.
oms select all /exceptif subtypes='Custom Table'/destination viewer=no.
CTABLES
/VLABELS VARIABLES=educ DISPLAY=DEFAULT
/TABLE educ [C][COUNT F40.0]
/CATEGORIES VARIABLES=educ ORDER=A KEY=VALUE EMPTY=INCLUDE MISSING=EXCLUDE
/CRITERIA CILEVEL=95.
DESCRIPTIVES VARIABLES=bdate educ id jobcat jobtime
/STATISTICS=MEAN STDDEV MIN MAX.
omsend.
I have the following syntax to merge two datasets. I expect that the resulting dataset (test1) contains 5 cases with 4 of them (2 to 5) a value in variable set2.
The result I am getting is dataset test1 with 5 cases but only 1 of them (case with id 5) has a value in variable set2.
Do I need to contact my ICT department, or am I misunderstanding something about merging data in SPSS. I am used to working with SAS, R and SQL, but need to help someone with a data merging within SPSS
INPUT PROGRAM.
LOOP id=1 to 5.
END CASE.
END LOOP.
END FILE.
END INPUT PROGRAM.
COMPUTE set1 = RV.NORMAL(1,1).
EXECUTE.
DATASET NAME test1.
INPUT PROGRAM.
LOOP id=2 to 5.
END CASE.
END LOOP.
END FILE.
END INPUT PROGRAM.
COMPUTE set2 = RV.NORMAL(1,1).
EXECUTE.
DATASET NAME test2.
DATASET ACTIVATE test1.
STAR JOIN
/SELECT t0.set1, t1.set2
/FROM * AS t0
/JOIN 'test2' AS t1
ON t0.id=t1.id
/OUTFILE FILE=*.
results in:
id set1 set2
1,00 1,74
2,00 1,58
3,00 1,01
4,00 ,12
5,00 2,52 ,79
SPSS version 21
When I run the syntax you provide I get the desired results (and not what you indicate):
If it continues to fail (after contacting SPSS support), try using MATCH FILES:
DATASET ACTIVATE test1.
SORT CASES BY ID.
DATASET ACTIVATE test2.
SORT CASES BY ID.
MATCH FILES FILE=test1 /FILE=test2 /BY ID.
DATASET NAME Restult.