I'm trying to run an optimization with SNOPT.
Right now as I run it I consistently get exit condition 41.
I've added the following parameters to the solver:
prog.SetSolverOption(solver.solver_id(),"Function Precision","1e-6")
prog.SetSolverOption(solver.solver_id(),"Major Optimality Tolerance","1e-3")
prog.SetSolverOption(solver.solver_id(),"Superbasics limit","600")
#print("Trying to Solve")
result = solver.Solve(prog)
print(result.is_success())
But I still contently get the same exit condition.
The error seems to be from the interpolation function I'm using. (When I remove it I no longer get the error).
t, c, k = interpolate.splrep(sref, kapparef, s=0, k=3)
kappafnc = interpolate.BSpline(t, c, k, extrapolate=False)
Here is the function I think I determined was causing the issue:
def car_continous_dynamics(self, state, force, steering):
beta = steering
s = state[0]
#print(s)
n = state[1]
alpha = state[2]
v = state[3]
#State = s, n, alpha , v
#s= distance along the track, n = perpendicular distance along the track
#alpha = velocity angle relative to the track
#v= magnitude of the velocity of the car
s_val = 0
if s.value() >0:
s_val = s.value()
Cm1 = 0.28
Cm2 = 0.05
Cr0 = 0.011
Cr2 = 0.006
m = 0.043
phi_dot = v*beta*15.5
Fxd = (Cm1 - Cm2 * v) * force - Cr2 * v * v - Cr0 *tanh(5 * v)
s_dot = v*cos(alpha+.5*beta)/(1)##-n*self.kappafunc(s_val))
n_dot= v*sin(alpha+.5*beta)
alpha_dot = phi_dot #-s_dot*self.kappafunc(s_val)
v_dot=Fxd/m*cos(beta*.5)
# concatenate velocity and acceleration
#print(s_dot)
#print(n_dot)
#print(alpha_dot)
#print(v_dot)
state_dot = np.array((s_dot,n_dot,alpha_dot,v_dot))
#print("State dot")
#print(state_dot.dtype.name)
#print(state_dot)
#print(state_dot)
return state_dot
``
Debugging INFO 41 in SNOPT is generally challenging, it is often caused by some bad gradient (but could also due to the problem being really hard to optimize).
You should check if your gradient is well-behaved. I see you have
s_val = 0
if s.value() >0:
s_val = s.value()
There are two potential problems
Your s carries the gradient (you could check s.derivatives(), it has non-zero gradient), but when you compute s_val, this is a float object with no gradient. Hence you lose the gradient information.
s_val is non-differentiable at 0.
I would use s instead of s_val in your function (without the branch if s.value() > 0). And also add a constraint s>=0 by
prog.AddBoundingBoxConstraint(0, np.inf, s)
With this bounding box constraint, SNOPT guarantees that in each iteration, the value of is is always non-negative.
I'd like to return a dask dataframe from an overlapping dask array computation, where each block's computation returns a pandas dataframe. The example below shows one way to do this, simplified for demonstration purposes. I've found a combination of da.overlap.overlap and to_delayed().ravel() as able to get the job done, if I pass in the relevant block key and chunk information.
Edit:
Thanks to a #AnnaM who caught bugs in the original post and then made it general! Building off of her comments, I'm including an updated version of the code. Also, in responding to Anna's interest in memory usage, I verified that this does not seem to take up more memory than naively expected.
def extract_features_generalized(chunk, offsets, depth, columns):
shape = np.asarray(chunk.shape)
offsets = np.asarray(offsets)
depth = np.asarray(depth)
coordinates = np.stack(np.nonzero(chunk)).T
keep = ((coordinates >= depth) & (coordinates < (shape - depth))).all(axis=1)
data = coordinates + offsets - depth
df = pd.DataFrame(data=data, columns=columns)
return df[keep]
def my_overlap_generalized(data, chunksize, depth, columns, boundary):
data = data.rechunk(chunksize)
data_overlapping_chunks = da.overlap.overlap(data, depth=depth, boundary=boundary)
dfs = []
for block in data_overlapping_chunks.to_delayed().ravel():
offsets = np.array(block.key[1:]) * np.array(data.chunksize)
df_block = dask.delayed(extract_features_generalized)(block, offsets=offsets,
depth=depth, columns=columns)
dfs.append(df_block)
return dd.from_delayed(dfs)
data = np.zeros((2,4,8,16,16))
data[0,0,4,2,2] = 1
data[0,1,4,6,2] = 1
data[1,2,4,8,2] = 1
data[0,3,4,2,2] = 1
arr = da.from_array(data)
df = my_overlap_generalized(arr,
chunksize=(-1,-1,-1,8,8),
depth=(0,0,0,2,2),
columns=['r', 'c', 'z', 'y', 'x'],
boundary=tuple(['reflect']*5))
df.compute().reset_index()
-- Remainder of original post, including original bugs --
My example only does xy overlaps, but it's easy to generalize. Is there anything below that is suboptimal or could be done better? Is anything likely to break because it's relying on low-level information that could change (e.g. block key)?
def my_overlap(data, chunk_xy, depth_xy):
data = data.rechunk((-1,-1,-1, chunk_xy, chunk_xy))
data_overlapping_chunks = da.overlap.overlap(data,
depth=(0,0,0,depth_xy,depth_xy),
boundary={3: 'reflect', 4: 'reflect'})
dfs = []
for block in data_overlapping_chunks.to_delayed().ravel():
offsets = np.array(block.key[1:]) * np.array(data.chunksize)
df_block = dask.delayed(extract_features)(block, offsets=offsets, depth_xy=depth_xy)
dfs.append(df_block)
# All computation is delayed, so downstream comptutions need to know the format of the data. If the meta
# information is not specified, a single computation will be done (which could be expensive) at this point
# to infer the metadata.
# This empty dataframe has the index, column, and type information we expect in the computation.
columns = ['r', 'c', 'z', 'y', 'x']
# The dtypes are float64, except for a small number of columns
df_meta = pd.DataFrame(columns=columns, dtype=np.float64)
df_meta = df_meta.astype({'c': np.int64, 'r': np.int64})
df_meta.index.name = 'feature'
return dd.from_delayed(dfs, meta=df_meta)
def extract_features(chunk, offsets, depth_xy):
r, c, z, y, x = np.nonzero(chunk)
df = pd.DataFrame({'r': r, 'c': c, 'z': z, 'y': y+offsets[3]-depth_xy,
'x': x+offsets[4]-depth_xy})
df = df[(df.y > depth_xy) & (df.y < (chunk.shape[3] - depth_xy)) &
(df.z > depth_xy) & (df.z < (chunk.shape[4] - depth_xy))]
return df
data = np.zeros((2,4,8,16,16)) # round, channel, z, y, x
data[0,0,4,2,2] = 1
data[0,1,4,6,2] = 1
data[1,2,4,8,2] = 1
data[0,3,4,2,2] = 1
arr = da.from_array(data)
df = my_overlap(arr, chunk_xy=8, depth_xy=2)
df.compute().reset_index()
First of all, thanks for posting your code. I am working on a similar problem and this was really helpful for me.
When testing your code, I discovered a few mistakes in the extract_features function that prevent your code from returning correct indices.
Here is a corrected version:
def extract_features(chunk, offsets, depth_xy):
r, c, z, y, x = np.nonzero(chunk)
df = pd.DataFrame({'r': r, 'c': c, 'z': z, 'y': y, 'x': x})
df = df[(df.y >= depth_xy) & (df.y < (chunk.shape[3] - depth_xy)) &
(df.x >= depth_xy) & (df.x < (chunk.shape[4] - depth_xy))]
df['y'] = df['y'] + offsets[3] - depth_xy
df['x'] = df['x'] + offsets[4] - depth_xy
return df
The updated code now returns the indices that were set to 1:
index r c z y x
0 0 0 0 4 2 2
1 1 0 1 4 6 2
2 2 0 3 4 2 2
3 1 1 2 4 8 2
For comparison, this is the output of the original version:
index r c z y x
0 1 0 1 4 6 2
1 3 1 2 4 8 2
2 0 0 1 4 6 2
3 1 1 2 4 8 2
It returns lines number 2 and 4, two times each.
The reason why this happens is three mistakes in the extract_features function:
You first add the offset and subtract the depth and then filter out the overlapping parts: the order needs to be swapped
df.y > depth_xy should be replaced with df.y >= depth_xy
df.z should be replaced with df.x, since it is the x dimension that has an overlap
To optimize this even further, here is a generalized version of the code that would work for an arbitrary number of dimension:
def extract_features_generalized(chunk, offsets, depth, columns):
coordinates = np.nonzero(chunk)
df = pd.DataFrame()
rows_to_keep = np.ones(len(coordinates[0]), dtype=int)
for i in range(len(columns)):
df[columns[i]] = coordinates[i]
rows_to_keep = rows_to_keep * np.array((df[columns[i]] >= depth[i])) * \
np.array((df[columns[i]] < (chunk.shape[i] - depth[i])))
df[columns[i]] = df[columns[i]] + offsets[i] - depth[i]
del coordinates
return df[rows_to_keep > 0]
def my_overlap_generalized(data, chunksize, depth, columns):
data = data.rechunk(chunksize)
data_overlapping_chunks = da.overlap.overlap(data, depth=depth,
boundary=tuple(['reflect']*len(columns)))
dfs = []
for block in data_overlapping_chunks.to_delayed().ravel():
offsets = np.array(block.key[1:]) * np.array(data.chunksize)
df_block = dask.delayed(extract_features_generalized)(block, offsets=offsets,
depth=depth, columns=columns)
dfs.append(df_block)
return dd.from_delayed(dfs)
data = np.zeros((2,4,8,16,16))
data[0,0,4,2,2] = 1
data[0,1,4,6,2] = 1
data[1,2,4,8,2] = 1
data[0,3,4,2,2] = 1
arr = da.from_array(data)
df = my_overlap_generalized(arr, chunksize=(-1,-1,-1,8,8),
depth=(0,0,0,2,2), columns=['r', 'c', 'z', 'y', 'x'])
df.compute().reset_index()
I am working on a client's site, and I'm writing an amortization schedule calculator in in ruby on rails. For longer loan term calculations, it doesn't seem to be breaking when the balance reaches 0
Here is my code:
def calculate_amortization_results
p = params[:price].to_i
i = params[:rate].to_d
l = params[:term].to_i
j = i/(12*100)
n = l * 12
m = p * (j / (1 - (1 + j) ** (-1 * n)))
#loanAmount = p
#rateAmount = i
#monthlyAmount = m
#amort = []
#interestAmount = 0
while p > 0
line = Hash.new
h = p*j
c = m-h
p = p-c
line["interest"] = h
line["principal"] = c
if p <= 0
line["balance"] = 0
else
line["balance"] = p
end
line["payment"] = h+c
#amort.push(line)
#interestAmount += h
end
end
And here is the view:
- #amort.each_with_index do |a, i|
%li
.m
= i+1
.i
= number_to_currency(a["interest"], :unit => "$")
.p
= number_to_currency(a["principal"], :unit => "$")
.pp
= number_to_currency(a["payment"], :unit => "$")
.b
= number_to_currency(a["balance"], :unit => "$")
What I am seeing is, in place of $0.00 in the final payment balance, it shows "-$-inf", iterates one more loop, then displays $0.00, but shows "-$-inf" for interest. It should loop until p gets to 0, then stop and set the balance as 0, but it isn't. Any idea what I've done wrong?
The calculator is here. It seems to work fine for shorter terms, like 5 years, but longer terms cause the above error.
Edit:
Changing the while loop to n.times do
and then changing the balance view to
= number_to_currency(a["balance"], :unit => "$", :negative_format => "$0.00")
Is a workaround, but i'd like to know why the while loop wouldn't work correctly
in Ruby the default for numerical values is Fixnum ... e.g.:
> 15 / 4
=> 3
You will see weird rounding errors if you try to use Fixnum values and divide them.
To make sure that you use Floats, at least one of the numbers in the calculation needs to be a Float
> 15.0 / 4
=> 3.75
> 15 / 4.0
=> 3.75
You do two comparisons against 0 , which should be OK if you make sure that p is a Float.
As the other answer suggests, you should use "decimal" type in your database to represent currency.
Please try if this will work:
def calculate_amortization_results
p = params[:price].to_f # instead of to_i
i = params[:rate].to_f # <-- what is to_d ? use to_f
l = params[:term].to_i
j = i/(12*100.0) # instead of 100
n = l * 12
m = p * (j / (1 - (1 + j) ** (-1 * n))) # division by zero if i==0 ==> j==0
#loanAmount = p
#rateAmount = i
#monthlyAmount = m
#amort = []
#interestAmount = 0.0 # instead of 0
while p > 0
line = Hash.new
h = p*j
c = m-h
p = p-c
line["interest"] = h
line["principal"] = c
if p <= 0
line["balance"] = 0
else
line["balance"] = p
end
line["payment"] = h+c
#amort.push(line)
#interestAmount += h
end
end
If you see "inf" in your output, you are doing a division by zero somewhere.. better check the logic of your calculation, and guard against division by zero.
according to Wikipedia the formula is:
http://en.wikipedia.org/wiki/Amortization_calculator
to improve rounding errors, it's probably better to re-structure the formula like this:
m = (p * j) / (1 - (1 + j) ** (-1 * n) # these are two divisions! x**-1 == 1/x
which is equal to:
m = (p * j) + (p * j) / ((1 + j) ** n) - 1.0)
which is equal to: (use this one)
q = p * j # this is much larger than 1 , so fewer rounding errors when dividing it by something
m = q + q / ((1 + j) ** n) - 1.0) # only one division
I think it has something to do with the floating point operations precision. It has already been discussed here: Ruby number precision with simple arithmetic and it would be better to use decimal format for financial purposes.
The answer could be computing the numbers in the loop, but with precomputed number of iterations and from the scratch.
Consider the following code:
let dl = 9.5 / 11.
let min = 21.5 + dl
let max = 40.5 - dl
let a = [ for z in min .. dl .. max -> z ] // should have 21 elements
let b = a.Length
"a" should have 21 elements but has got only 20 elements. The "max - dl" value is missing. I understand that float numbers are not precise, but I hoped that F# could work with that. If not then why F# supports List comprehensions with float iterator? To me, it is a source of bugs.
Online trial: http://tryfs.net/snippets/snippet-3H
Converting to decimals and looking at the numbers, it seems the 21st item would 'overshoot' max:
let dl = 9.5m / 11.m
let min = 21.5m + dl
let max = 40.5m - dl
let a = [ for z in min .. dl .. max -> z ] // should have 21 elements
let b = a.Length
let lastelement = List.nth a 19
let onemore = lastelement + dl
let overshoot = onemore - max
That is probably due to lack of precision in let dl = 9.5m / 11.m?
To get rid of this compounding error, you'll have to use another number system, i.e. Rational. F# Powerpack comes with a BigRational class that can be used like so:
let dl = 95N / 110N
let min = 215N / 10N + dl
let max = 405N / 10N - dl
let a = [ for z in min .. dl .. max -> z ] // Has 21 elements
let b = a.Length
Properly handling float precision issues can be tricky. You should not rely on float equality (that's what list comprehension implicitely does for the last element). List comprehensions on float are useful when you generate an infinite stream. In other cases, you should pay attention to the last comparison.
If you want a fixed number of elements, and include both lower and upper endpoints, I suggest you write this kind of function:
let range from to_ count =
assert (count > 1)
let count = count - 1
[ for i = 0 to count do yield from + float i * (to_ - from) / float count]
range 21.5 40.5 21
When I know the last element should be included, I sometimes do:
let a = [ for z in min .. dl .. max + dl*0.5 -> z ]
I suspect the problem is with the precision of floating point values. F# adds dl to the current value each time and checks if current <= max. Because of precision problems, it might jump over max and then check if max+ε <= max (which will yield false). And so the result will have only 20 items, and not 21.
After running your code, if you do:
> compare a.[19] max;;
val it : int = -1
It means max is greater than a.[19]
If we do calculations the same way the range operator does but grouping in two different ways and then compare them:
> compare (21.5+dl+dl+dl+dl+dl+dl+dl+dl) ((21.5+dl)+(dl+dl+dl+dl+dl+dl+dl));;
val it : int = 0
> compare (21.5+dl+dl+dl+dl+dl+dl+dl+dl+dl) ((21.5+dl)+(dl+dl+dl+dl+dl+dl+dl+dl));;
val it : int = -1
In this sample you can see how adding 7 times the same value in different order results in exactly the same value but if we try it 8 times the result changes depending on the grouping.
You're doing it 20 times.
So if you use the range operator with floats you should be aware of the precision problem.
But the same applies to any other calculation with floats.
If I am given a floating point number but do not know beforehand what range the number will be in, is it possible to scale that number in some meaningful way to be in another range? I am thinking of checking to see if the number is in the range 0<=x<=1 and if not scale it to that range and then scale it to my final range. This previous post provides some good information, but it assumes the range of the original number is known beforehand.
You can't scale a number in a range if you don't know the range.
Maybe what you're looking for is the modulo operator. Modulo is basically the remainder of division, the operator in most languages is is %.
0 % 5 == 0
1 % 5 == 1
2 % 5 == 2
3 % 5 == 3
4 % 5 == 4
5 % 5 == 0
6 % 5 == 1
7 % 5 == 2
...
Sure it is not possible. You can define range and ignore all extrinsic values. Or, you can collect statistics to find range in run time (i.e. via histogram analysis).
Is it really about image processing? There are lots of related problems in image segmentation field.
You want to scale a single random floating point number to be between 0 and 1, but you don't know the range of the number?
What should 99.001 be scaled to? If the range of the random number was [99, 100], then our scaled-number should be pretty close to 0. If the range of the random number was [0, 100], then our scaled-number should be pretty close to 1.
In the real world, you always have some sort of information about the range (either the range itself, or how wide it is). Without further info, the answer is "No, it can't be done."
I think the best you can do is something like this:
int scale(x) {
if (x < -1) return 1 / x - 2;
if (x > 1) return 2 - 1 / x;
return x;
}
This function is monotonic, and has a range of -2 to 2, but it's not strictly a scaling.
I am assuming that you have the result of some 2-dimensional measurements and want to display them in color or grayscale. For that, I would first want to find the maximum and minimum and then scale between these two values.
static double[][] scale(double[][] in, double outMin, double outMax) {
double inMin = Double.POSITIVE_INFINITY;
double inMax = Double.NEGATIVE_INFINITY;
for (double[] inRow : in) {
for (double d : inRow) {
if (d < inMin)
inMin = d;
if (d > inMax)
inMax = d;
}
}
double inRange = inMax - inMin;
double outRange = outMax - outMin;
double[][] out = new double[in.length][in[0].length];
for (double[] inRow : in) {
double[] outRow = new double[inRow.length];
for (int j = 0; j < inRow.length; j++) {
double normalized = (inRow[j] - inMin) / inRange; // 0 .. 1
outRow[j] = outMin + normalized * outRange;
}
}
return out;
}
This code is untested and just shows the general idea. It further assumes that all your input data is in a "reasonable" range, away from infinity and NaN.