What is the best approach to achieve a linear regression using CEP ?. We have tried two different options.
We do want to have the algorithm working in real time.
Basic code for both approach :
create context IntervalSpanning3Seconds start #now end after 30 sec;
create schema measure (
temperature float,
water float,
_hours float,
persons float,
production float
);
#Name("gattering_measures")
insert into measure
select
cast(getNumber(m,"measurement.bsk_mymeasurement.temperature.value"),
float) as temperature,
cast(getNumber(m, "measurement.bsk_mymeasurement.water.value"), float) as water,
cast(getNumber(m, "measurement.bsk_mymeasurement._hours.value"), float) as _hours,
cast(getNumber(m, "measurement.bsk_mymeasurement.persons.value"), float) as persons,
cast(getNumber(m, "measurement.bsk_mymeasurement.production.value"),float) as production
from MeasurementCreated m
where m.measurement.type = "bsk_mymeasurement";
1. Using the function stat:linest
#Name("get_data")
context IntervalSpanning3Seconds
select * from measure.stat:linest(water,production,_hours,persons,temperature)
output snapshot when terminated;
EDIT: The problem here is that it seems like the "get_data" is getting execute by each measurement and not by the entire collection of measurement.
2. Get data and passed a javascript function.
create expression String exeReg(data) [
var f = f(data)
function f(d){
.....
// return the linear regression as a string
}
return f
];
#Name("get_data")
insert into CreateEvent
select
"bsk_outcome_linear_regression" as type,
exeReg(m) as text,
....
from measure m;
EDIT: Here, I would like to know what is the type of the variable that is passed to the exeReg() function and how I should iterate it ? example would be nice.
I'll appreciate any help.
Using JavaScript would mean that the script computes a new result (recomputes) for each collection it receives. Instead of recomputing it the #lineest data window is a good choice. Or you can add a custom aggregation function or custom data window to the engine if there is certain code your want to use. Below is how the script can receive multiple events for the case when a script is desired.
create expression String exeReg(data) [
.... script here...
];
select exeReg(window(*)) ....
from measure#time(10 seconds);
Related
I wonder if there is any way to make functions defined within the main function be local, in a similar way to local variables. For example, in this function that calculates the gradient of a scalar function,
grad(var,f) := block([aux],
aux : [gradient, DfDx[i]],
gradient : [],
DfDx[i] := diff(f(x_1,x_2,x_3),var[i],1),
for i in [1,2,3] do (
gradient : append(gradient, [DfDx[i]])
),
return(gradient)
)$
The variable gradient that has been defined inside the main function grad(var,f) has no effect outside the main function, as it is inside the aux list. However, I have observed that the function DfDx, despite being inside the aux list, does have an effect outside the main function.
Is there any way to make the sub-functions defined inside the main function to be local only, in a similar way to what can be made with local variables? (I know that one can kill them once they have been used, but perhaps there is a more elegant way)
To address the problem you are needing to solve here, another way to compute the gradient is to say
grad(var, e) := makelist(diff(e, var1), var1, var);
and then you can say for example
grad([x, y, z], sin(x)*y/z);
to get
cos(x) y sin(x) sin(x) y
[--------, ------, - --------]
z z 2
z
(There isn't a built-in gradient function; this is an oversight.)
About local functions, bear in mind that all function definitions are global. However you can approximate a local function definition via local, which saves and restores all properties of a symbol. Since the function definition is a property, local has the effect of temporarily wiping out an existing function definition and later restoring it. In between you can create a temporary function definition. E.g.
foo(x) := 2*x;
bar(y) := block(local(foo), foo(x) := x - 1, foo(y));
bar(100); /* output is 99 */
foo(100); /* output is 200 */
However, I don't this you need to use local -- just makelist plus diff is enough to compute the gradient.
There is more to say about Maxima's scope rules, named and unnamed functions, etc. I'll try to come back to this question tomorrow.
To compute the gradient, my advice is to call makelist and diff as shown in my first answer. Let me take this opportunity to address some related topics.
I'll paste the definition of grad shown in the problem statement and use that to make some comments.
grad(var,f) := block([aux],
aux : [gradient, DfDx[i]],
gradient : [],
DfDx[i] := diff(f(x_1,x_2,x_3),var[i],1),
for i in [1,2,3] do (
gradient : append(gradient, [DfDx[i]])
),
return(gradient)
)$
(1) Maxima works mostly with expressions as opposed to functions. That's not causing a problem here, I just want to make it clear. E.g. in general one has to say diff(f(x), x) when f is a function, instead of diff(f, x), likewise integrate(f(x), ...) instead of integrate(f, ...).
(2) When gradient and Dfdx are to be the local variables, you have to name them in the list of variables for block. E.g. block([gradient, Dfdx], ...) -- Maxima won't understand block([aux], aux: ...).
(3) Note that a function defined with square brackets instead of parentheses, e.g. f[x] := ... instead of f(x) := ..., is a so-called array function in Maxima. An array function is a memoizing function, i.e. if f[x] is called two or more times, the return value is only computed once, and then returned every time thereafter. Sometimes that's a useful optimization when the domain of the function comprises a finite set.
(4) Bear in mind that x_1, x_2, x_3, are distinct symbols, not related to each other, and not related to x[1], x[2], x[3], even if they are displayed the same. My advice is to work with subscripted symbols x[i] when i is a variable.
(5) About building up return values, try to arrange to compute the whole thing at one go, instead of growing the result incrementally. In this case, makelist is preferable to for plus append.
(6) The return function in Maxima acts differently than in other programming languages; it's a little hard to explain. A function returns the value of the last expression which was evaluated, so if gradient is that last expression, you can just write grad(var, f) := block(..., gradient).
Hope this helps, I know it's obscure and complex. The Maxima programming language was not designed before being implemented, and some of the decisions are clearly questionable at the long interval of more than 50 years (!) later. That's okay, they were figuring it out as they went along. There was not a body of established results which could provide a point of reference; the original authors were contributing to what's considered common knowledge today.
I would like to add units, long_name, and maybe a description to a variable while using the to_netcdf command. Let me know if you know how.
Here is my code that work:
filename = path+'file.nc'
ds = xr.Dataset({'sla': (('time_counter','x', 'y'), SLA)}, coords={'time_counter':time_counter,'nav_lon':(('x','y'),lon),'nav_lat':(('x','y'),lat)})
ds.to_netcdf(filename, 'w')
Supplementary informations if you want to use this:
'sla' is the name I give while saving the variable SLA
SLA has 3 dimensions; I give them the names 'time_counter', 'x', and 'y'
I defined coordinates, one of which ('time_counter') is directly a dimension of SLA, but also it is possible to have a coordinate with multiple dimensions (e.g., 'nav_lon' and 'nav_lat' have 2 dimensions.
Here is the link that explain the function: http://xarray.pydata.org/en/stable/generated/xarray.Dataset.to_netcdf.html
You can set the attributes of each variable before saving the Dataset to NetCDF, for example (after creating your ds):
ds['sla'].attrs = {'units': 'something'}
After the to_netcdf() step I get (part of the ncdump -h):
double sla(time_counter, x, y) ;
...
sla:units = "something" ;
I am surprising to notice that it is somehow difficult to obtain a correct fit of interaction function from gam().
To be more specific, I want to estimate an additive function:
y=m_1(x)+m_2(z)+m_{12}(x,z)+u,
where m_1(x)=x^2, m_2(z)=z^2,m_{12}(x,z)=xz. The following code generate this model:
test1 <- function(x,z,sx=1,sz=1) {
#--m1(x) function
m.x<-x^2
m.x<-m.x-mean(m.x)
#--m2(z) function
m.z<-z^2
m.z<-m.z-mean(m.z)
#--m12(x,z) function
m.xz<-x*z
m.xz<-m.xz-mean(m.xz)
m<-m.x+m.z+m.xz
return(list(m=m,m.x=m.x,m.z=m.z,m.xz=m.xz))
}
n <- 1000
a=0
b=2
x <- runif(n,a,b)/20
z <- runif(n,a,b)
u <- rnorm(n,0,0.5)
model<-test1(x,z)
y <- model$m + u
So I use gam() by fitting the model as
b3 <- gam(y~ ti(x) + ti(z) + ti(x,z))
vis.gam(b3);title("tensor anova")
#---extracting basis matrix
B.f3<-model.matrix.gam(b3)
#---extracting series estimator
b3.hat<-b3$coefficients
Question: when I plot the estimated function by gam()above against its true function, I end up with
par(mfrow=c(1,3))
#---m1(x)
B.x<-B.f3[,c(2:5)]
b.x.hat<-b3.hat[c(2:5)]
plot(x,B.x%*%b.x.hat)
points(x,model$m.x,col='red')
legend('topleft',c('Estimate','True'),lty=c(1,1),col=c('black','red'))
#---m2(z)
B.z<-B.f3[,c(6:9)]
b.z.hat<-b3.hat[c(6:9)]
plot(z,B.z%*%b.z.hat)
points(z,model$m.z,col='red')
legend('topleft',c('Estimate','True'),lty=c(1,1),col=c('black','red'))
#---m12(x,z)
B.xz<-B.f3[,-c(1:9)]
b.xz.hat<-b3.hat[-c(1:9)]
plot(x,B.xz%*%b.xz.hat)
points(x,model$m.xz,col='red')
legend('topleft',c('Estimate','True'),lty=c(1,1),col=c('black','red'))
However, the function estimate of m_1(x) is largely different from x^2, and the interaction function estimate m_{12}(x,z) is also largely different from xz defined in test1 above. The results are the same if I use predict(b3).
I really can't figure it out. Can anybody help me out by explaining why the results end up with this? Greatly appreciate it!
First, the problem of the above issue is not due to the package, of course. It is closely related to the identification conditions of the smooth functions. One common practice is to impose the assumptions that E(mj(.))=0 for all individual function j=1,...,d, and E(m_ij(x_i,x_j)|x_i)=E(m_ij(x_i,x_j)|x_j)=0 for i not equal to j. Those conditions require one to employ centered basis function in series estimator, which has been done already in GAM package. However, in my case above, function m(x,z)=x*z defined in test1 does not satisfy the above identification assumptions, since the integral of x*z with respect to either x or z is not zero when x and z have range from zero to two.
Furthermore, series estimator allows the individual and interaction function to be identified if one impose m(0)=0 or m(0,x_j)=m(x_i,0)=0. This can be readily achieved if we center the basis function around zero. I have tried both cases, and they work well whenever DGP satisfies the identification conditions.
I have some scientific project. There are vectors / square matrices of various lengths there. Obviously (for example) a vector of length 2 cannot be added to a vector of length 3 (and so on and so forth). There are several NET libraries, which deal with vectors / matrices. All of them either have generic vectors / matrices OR have some very specific vectors / matrices, which do not suite the needs.
Most, if not all, of these libraries can create a vector from a list or array. Unfortunately, If I mistakenly give an input array of the wrong length, then I will get a vector of the wrong length and then everything will blow up at run time!
I wonder if it is possible to check array length at compile time so that to get a compile error if, let’s say, I try to pass a 5-element array to a vector of length 2 “constructor”. After all, printfn does almost that!
F# type providers come to mind, but I am not sure how to apply them here.
Thanks a lot!
Thanks to the OP for an interesting question. My answer frequency has dropped not because of unwillingness to help but rather that there a few questions that tickles my interest.
We don't have dependent types in F# and F# doesn't support generics with numerical type arguments (like C++).
However we could create distinct types for different dimensions like Dim1, Dim2 and so on and provide them as type arguments.
This would allow us to have a type signature for apply that applies a vector a matrix like this:
let apply (m : Matrix<'R, 'C>) (v : Vector<'C>) : Vector<'R> = …
The code won't compile unless the columns of the matrix matches the length of the vector. In addition; the resulting vector has the length that is rows of the columns.
One way to do this is defining an interface IDimension and some concrete implementions representing the different dimensions.
type IDimension =
interface
abstract Size : int
end
type Dim1 () = class interface IDimension with member x.Size = 1 end end
type Dim2 () = class interface IDimension with member x.Size = 2 end end
The vector and the matrix can then be implemented like this
type Vector<'Dim when 'Dim :> IDimension
and 'Dim : (new : unit -> 'Dim)
> () =
class
let dim = new 'Dim()
let vs = Array.zeroCreate<float> dim.Size
member x.Dim = dim
member x.Values = vs
end
type Matrix<'RowDim, 'ColumnDim when 'RowDim :> IDimension
and 'RowDim : (new : unit -> 'RowDim)
and 'ColumnDim :> IDimension
and 'ColumnDim : (new : unit -> 'ColumnDim)
> () =
class
let rowDim = new 'RowDim()
let columnDim = new 'ColumnDim()
let vs = Array.zeroCreate<float> (rowDim.Size*columnDim.Size)
member x.RowDim = rowDim
member x.ColumnDim = columnDim
member x.Values = vs
end
Finally this allows us to write code like this:
let m76 = Matrix<Dim7, Dim6> ()
let v6 = Vector<Dim6> ()
let v7 = apply m76 v6 // Vector<Dim7>
// Doesn't compile because v7 has the wrong dimension
let vv = apply m76 v7
If you need a wide range of dimensions (because you have an algebra increments/decrements the dimensions of vectors/matrices) you could support that using some smart variant of church numerals.
If this is usable or not is entirely up the reader I think.
PS.
Perhaps unit of measures could have been used for this as well if they applied to more types than floats.
The general term for what you're looking for is dependent types, but F# does not support them.
I've seen an experiment in using type providers to mimic one particular flavor of dependent types (constraining the domain of a primitive type), but I wouldn't expect it to be possible to achieve what you want using type providers in their current form. They seem to be too whimsical for that.
Print format strings appear to be doing that (and in fact printers are a "Hello World" application for dependent types), but actually they work because they get special treatment by the compiler, and the mechanism for that is not extensible.
You're doomed to ensure correct lengths at runtime.
My best bet would be to use structs to encode actual vectors and ensure correctness on the API level that way, map them to arrays at the point where you're interacting with those matrix algebra libraries, then map the results back to structs with ample assertions when done.
The comment from #Justanothermetaprogrammer qualifies as an answer. Here is how it works in the real example. The matrix implementation in the example is based on MathNet.Numerics.LinearAlgebra:
open MathNet.Numerics.LinearAlgebra
type RealMatrix2x2 =
| RealMatrix2x2 of Matrix<double>
static member private createInternal (a : #seq<#seq<double>>) =
matrix a |> RealMatrix2x2
static member create
(
(a11, a12),
(a21, a22)
) =
RealMatrix2x2.createInternal
[|
[| a11; a12|]
[| a21; a22|]
|]
let m2 =
(
(1., 2.),
(3., 4.)
)
|> RealMatrix2x2.create
The tuple signatures and "re-mapping" into #seq<#seq<double>> can be easily code-generated using, for example, Excel or any other convenient tool for as many dimensions as necessary. In fact, the whole class along with any other necessary operator overrides (like multiplication of RealMatrix2x2 by RealMatrix2x2, ...) can be code generated for all necessary dimensions.
I have a simple F# function cost receiving a single parameter amount which is used for some calculations. It is a float so I need to pass in something like cost 33.0 which in math is the same as cost 33. The compiler complaints about it, and I understand why, but I would like being able to call it like that, I tried to create another function named the same and used type annotation for both of them and I also get compiler warnings. Is there a way to do this like C# does?
There are two mechanisms in F# to achieve this, and both do not rely on implicit casts "like C#":
(A) Method overloading
type Sample =
static member cost (amount: float) =
amount |> calculations
static member cost (amount: int) =
(amount |> float) |> calculations
Sample.cost 10 // compiles OK
Sample.cost 10. // compiles OK
(B) Using inlining
let inline cost amount =
amount + amount
cost 10 // compiles OK
cost 10. // compiles OK
F# doesn't allow overloading of let-bound functions, but you can overload methods on classes like in C#.
Sometimes, you can change the model to work on a Discriminated Union instead of a set of overloaded primitives, but I don't think it would be particularly sensible to do just to be able to distinguish between floats and integers.
if you want to use an int at call site but have a float inside the function body ; why not simply cast it ?
let cost amount =
// cast amount from to float (reusing the name amount to shadow the first one)
let amount = float amount
// rest of your function