I would like to use dask.array.map_overlap to deal with the scipy interpolation function. However, I keep meeting errors that I cannot understand and hoping someone can answer this to me.
Here is the error message I have received if I want to run .compute().
ValueError: could not broadcast input array from shape (1070,0) into shape (1045,0)
To resolve the issue, I started to use .to_delayed() to check each partition outputs, and this is what I found.
Following is my python code.
Step 1. Load netCDF file through Xarray, and then output to dask.array with chunk size (400,400)
df = xr.open_dataset('./Brazil Sentinal2 Tile/' + data_file +'.nc')
lon, lat = df['lon'].data, df['lat'].data
slon = da.from_array(df['lon'], chunks=(400,400))
slat = da.from_array(df['lat'], chunks=(400,400))
data = da.from_array(df.isel(band=0).__xarray_dataarray_variable__.data, chunks=(400,400))
Step 2. declare a function for da.map_overlap use
def sumsum2(lon,lat,data, hex_res=10):
hex_col = 'hex' + str(hex_res)
lon_max, lon_min = lon.max(), lon.min()
lat_max, lat_min = lat.max(), lat.min()
b = box(lon_min, lat_min, lon_max, lat_max, ccw=True)
b = transform(lambda x, y: (y, x), b)
b = mapping(b)
target_df = pd.DataFrame(h3.polyfill( b, hex_res), columns=[hex_col])
target_df['lat'] = target_df[hex_col].apply(lambda x: h3.h3_to_geo(x)[0])
target_df['lon'] = target_df[hex_col].apply(lambda x: h3.h3_to_geo(x)[1])
tlon, tlat = target_df[['lon','lat']].values.T
abc = lNDI(points=(lon.ravel(), lat.ravel()),
values= data.ravel())(tlon,tlat)
target_df['out'] = abc
print(np.stack([tlon, tlat, abc],axis=1).shape)
return np.stack([tlon, tlat, abc],axis=1)
Step 3. Apply the da.map_overlap
b = da.map_overlap(sumsum2, slon[:1200,:1200], slat[:1200,:1200], data[:1200,:1200], depth=10, trim=True, boundary=None, align_arrays=False, dtype='float64',
)
Step 4. Using to_delayed() to test output shape
print(b.to_delayed().flatten()[0].compute().shape, )
print(b.to_delayed().flatten()[1].compute().shape)
(1065, 3)
(1045, 0)
(1090, 3)
(1070, 0)
which is saying that the output from da.map_overlap is only outputting 1-D dimension ( which is (1045,0) and (1070,0) ), while in the da.map_overlap, the output I am preparing is 2-D dimension ( which is (1065,3) and (1090,3) ).
In addition, if I turn off the trim argument, which is
c = da.map_overlap(sumsum2,
slon[:1200,:1200],
slat[:1200,:1200],
data[:1200,:1200],
depth=10,
trim=False,
boundary=None,
align_arrays=False,
dtype='float64',
)
print(c.to_delayed().flatten()[0].compute().shape, )
print(c.to_delayed().flatten()[1].compute().shape)
The output becomes
(1065, 3)
(1065, 3)
(1090, 3)
(1090, 3)
This is saying that when trim=True, I cut out everything?
because...
#-- print out the values
b.to_delayed().flatten()[0].compute()[:10,:]
(1065, 3)
array([], shape=(1045, 0), dtype=float64)
while...
#-- print out the values
c.to_delayed().flatten()[0].compute()[:10,:]
array([[ -47.83683837, -18.98359832, 1395.01848583],
[ -47.8482856 , -18.99038681, 2663.68391094],
[ -47.82800624, -18.99207069, 1465.56517187],
[ -47.81897323, -18.97919009, 2769.91556363],
[ -47.82066663, -19.00712956, 1607.85927095],
[ -47.82696896, -18.97167714, 2110.7516765 ],
[ -47.81562653, -18.98302933, 2662.72112163],
[ -47.82176881, -18.98594465, 2201.83205114],
[ -47.84567 , -18.97512514, 1283.20631652],
[ -47.84343568, -18.97270783, 1282.92117225]])
Any thoughts for this?
Thank You.
I guess I got the answer. Please let me if I am wrong.
I am not allowing to use trim=True is because I change the shape of output array (after surfing the internet, I notice that the shape of output array should be the same with the shape of input array). Since I change the shape, the dask has no idea how to deal with it so it returns the empty array to me (weird).
Instead of using trim=False, since I didn't ask cutting-out the buffer zone, it is now okay to output the return values. (although I still don't know why the dask cannot concat the chunked array, but believe is also related to shape)
The solution is using delayed function on da.concatenate, which is
delayed(da.concatenate)([e.to_delayed().flatten()[idx] for idx in range(len(e.to_delayed().flatten()))])
In this case, we are not relying on the concat function in map_overlap but use our own concat to combine the outputs we want.
I came across an error during execute stereoCalibrate in Opencv 2.4.11, which is says :
OpenCV Error: Assertion failed (!fixedSize() || ((Mat*)obj)->size.operator()() == Size(cols, rows)) in cv::_OutputArray::create,
I think this must be some size error between these parameters, which go through them one by one. But there is still error. I hope someone awesome could find the error from the assembly code below. Here is the method call in my code.
double error = cv::stereoCalibrate(
objPoints, cali0.imgPoints, cali1.imgPoints,
camera0.intr.cameraMatrix, camera0.intr.distCoeffs,
camera1.intr.cameraMatrix, camera1.intr.distCoeffs,
cv::Size(1920,1080), m.rvec, m.tvec, m.evec, m.fvec,
cv::TermCriteria(CV_TERMCRIT_ITER + CV_TERMCRIT_EPS, 100, 1e-5)
,CV_CALIB_FIX_INTRINSIC + CV_CALIB_USE_INTRINSIC_GUESS
);
In my code, m.rvec is (3,3,CV_64F), m.tvec is (3,1,CV_64F), m.evec and m.fvec are not preallocated which is same with the stereoCalibrate example. And intr.cameraMatrix is (3,3,CV_64F) and intr.distCoeffs is (8,1,CV_64F), objPoints is computed from the checkerboard which stores the 3d position of corners and all z value for point is zero.
After reading advice from #Josh, I modify the code as plain output mat object which are in CV_64F, but it still throws this assertion.
cv::Mat R, t, e, f;
double error = cv::stereoCalibrate(
objPoints, cali0.imgPoints, cali1.imgPoints,
camera0.intr.cameraMatrix, camera0.intr.distCoeffs,
camera1.intr.cameraMatrix, camera1.intr.distCoeffs,
cali0.imgSize, R, t, e, f,
cv::TermCriteria(CV_TERMCRIT_ITER + CV_TERMCRIT_EPS, 100, 1e-5));
Finally I solved this problem, as a reminder, make sure the camera parameters you passed in are not const type....
Why go for assembly? OpenCV is open source and you can check the code you're calling here: https://github.com/opencv/opencv/blob/master/modules/calib3d/src/calibration.cpp#L3523
If you get assertion fails in OpenCV it's usually because you've passed a matrix with an incorrect shape. OpenCV is extremely picky. The assertion fail is on an OutputArray, so checking the function signature there are four possible culprits:
OutputArray _Rmat, OutputArray _Tmat, OutputArray _Emat, OutputArray _Fmat
The sizing is done inside cv::stereoCalibrate here:
https://github.com/opencv/opencv/blob/master/modules/calib3d/src/calibration.cpp#L3550
_Rmat.create(3, 3, rtype);
_Tmat.create(3, 1, rtype);
<-- snipped -->
if( _Emat.needed() )
{
_Emat.create(3, 3, rtype);
p_matE = &(c_matE = _Emat.getMat());
}
if( _Fmat.needed() )
{
_Fmat.create(3, 3, rtype);
p_matF = &(c_matF = _Fmat.getMat());
}
The assertion is being triggered in one of these calls, the code is here:
https://github.com/opencv/opencv/blob/master/modules/core/src/matrix.cpp#L2241
Try passing in plain Mat objects without preallocating their shape.
I am new to Prolog
I am trying in Prolog a rule that gives me a given path from a node to another and also gives me the total weight of the path.
I have succeeded to get all the edges of the path but I am not able to show the weight of the path. I debbuged it and it is seen that variable S adds up to the whole weight of the path but in the way back, deletes all the elements. My idea is to add the total weight to P.
Code:
notIn(A,[]).
notIn(A,[H|T]):- A\==H,notIn(A,T).
path(X,X,_,[], S, P).
path(X,Y,[X|Cs], S, P) :-
path(X,Y,[X],Cs, S, P), P is S+W.
path(X,Y,Visited,[Z|Cs], S, P) :-
connection(X,Z,W),
notIn(Z,Visited),
path(Z,Y,[Z|Visited],Cs, S+W, P).
? path(ori, dest, X, 0, P).
Your predicate almost works. There are only two issues and some details I'd like to address. Firstly it would aid readability greatly to separate predicates with different arities. So let's put the one rule of path/5 in front of the two rules of path/6 like so:
path(X,Y,[X|Cs], S, P) :-
path(X,Y,[X],Cs, S, P),
P is S+W. % <-(1)
path(X,X,_,[], S, P).
path(X,Y,Visited,[Z|Cs], S, P) :-
connection(X,Z,W),
notIn(Z,Visited),
path(Z,Y,[Z|Visited],Cs, S+W, P). % <-(2)
Looking at your example query path/5 seems to be the predicate you want to call to find paths. In the second goal of its single rule (marked as % <-(1)) you are using the built-in is/2 with the expression S+W on the right hand side. The variable W appears here for the first time and thus is unbound. This leads to an instantiation error as illustrated by the following example:
?- X is 1+W.
ERROR!!
INSTANTIATION ERROR- in arithmetic: expected bound value
However, since you are only using path/5 to call path/6 there is no need for that goal. Secondly, in the second rule of path/6, in the last goal you are passing S+W as argument instead of evaluating it first. To see what happens, let's remove the goal marked % <-(1) from path/5 and add an example graph to your code:
connection(ori,a,2).
connection(a,b,5).
connection(b,a,4).
connection(b,dest,1).
Now consider your example query with an additional goal:
?- path(ori, dest, X, 0, P), Weight is P.
P = 0+2+5+1,
Weight = 8,
X = [ori,a,b,dest] ? ;
no
As you see the argument S+W leads to the final weight being an expression rather than a value. Consider adding a goal S1 is S+W before the recursive goal and pass S1 as an argument. Thirdly you are using the built-in (\==)/2 in your predicate notIn/2. This comparison succeeds or fails without side effect or unification. This is fine as long as both arguments are bound to values but are problematic when used with unbound variables. Consider the following queries:
?- X=Y, X\==Y.
no
fails as expected but:
?- X\==Y, X=Y.
X = Y
succeeds as X\==Y has no effect to the variables, so they can be unified in the next goal. It is a good idea to use dif/2 instead:
?- X=Y, dif(X,Y).
no
?- dif(X,Y), X=Y.
no
Lastly, two minor suggestions: First, since you are using the 4th argument of path/5 to pass 0 as a start-value for the weight, you might as well do that in the single goal of the rule, thereby simplifying the interface to path/4. Second, it would be nice to have a more descriptive name for the predicate that reflects its declarative nature, say start_end_path_weight/4. So your code would then look something like this:
notIn(A,[]).
notIn(A,[H|T]):-
dif(A,H),
notIn(A,T).
start_end_path_weight(X,Y,[X|Cs], P) :-
path(X,Y,[X],Cs, 0, P).
path(X,X,_,[], P, P).
path(X,Y,Visited,[Z|Cs], S, P) :-
connection(X,Z,W),
notIn(Z,Visited),
S1 is S+W,
path(Z,Y,[Z|Visited],Cs, S1, P).
With these modifications your example query looks like this:
?- start_end_path_weight(ori,dest,X,W).
W = 8,
X = [ori,a,b,dest] ? ;
no
Here's how to improve upon #tas's answer by using clpfd for arithmetics instead of (is)/2:
:- use_module(library(clpfd)).
start_end_path_weight(X,Y,[X|Cs], P) :-
path(X,Y,[X],Cs, 0, P).
path(X,X,_,[], P, P).
path(X,Y,Visited,[Z|Cs], S, P) :-
connection(X,Z,W),
notIn(Z,Visited)
maplist(dif(Z),Visited),
S1 is S+W
S1 #= S+W, S1 #=< P,
path(Z,Y,[Z|Visited],Cs, S1, P).
Limiting the maximum costs? Piece of cake!
Consider the following InterRail subset ...
... translated to Prolog ...
connection(X,Y,D) :- to_fro_dt(X,Y,D).
connection(X,Y,D) :- to_fro_dt(Y,X,D).
to_fro_dt(aberdeen,edinburgh,140). to_fro_dt(amsterdam,berlin,370). to_fro_dt(amsterdam,brussels,113). to_fro_dt(amsterdam,cologne,158). to_fro_dt(amsterdam,copenhagen,675). to_fro_dt(ancona,igoumenitsa,900). to_fro_dt(athens,patras,215). to_fro_dt(athens,/* for consistency */piraeus,5). to_fro_dt(athens,thessaloniki,265). to_fro_dt(bar,belgrade,572). to_fro_dt(barcelona,madrid,170). to_fro_dt(barcelona,marseille,280). to_fro_dt(barcelona,sevilla,330). to_fro_dt(barcelona,valencia,175). to_fro_dt(bari,igoumenitsa,570). to_fro_dt(bari,rome,240). to_fro_dt(belfast,dublin,240). to_fro_dt(belgrade,bucharest,730). to_fro_dt(belgrade,budapest,450). to_fro_dt(belgrade,sarajevo,540). to_fro_dt(belgrade,skopje,525). to_fro_dt(belgrade,sofia,485). to_fro_dt(bergen,oslo,405). to_fro_dt(berlin,cologne,260). to_fro_dt(berlin,hamburg,95). to_fro_dt(berlin,munich,345). to_fro_dt(berlin,prague,275). to_fro_dt(berlin,warsaw,365). to_fro_dt(bern,frankfurt,235). to_fro_dt(bern,lyon,230). to_fro_dt(bern,milan,240). to_fro_dt(birmingham,edinburgh,265). to_fro_dt(birmingham,holyhead,245). to_fro_dt(birmingham,london,105). to_fro_dt(bologna,florence,37). to_fro_dt(bologna,milan,60). to_fro_dt(bordeaux,lyon,375). to_fro_dt(bordeaux,madrid,660). to_fro_dt(bordeaux,paris,180). to_fro_dt(bristol,london,105). to_fro_dt(brussels,cologne,107). to_fro_dt(brussels,frankfurt,190). to_fro_dt(brussels,london,140). to_fro_dt(brussels,paris,85). to_fro_dt(bucharest,budapest,830). to_fro_dt(bucharest,sofia,540). to_fro_dt(bucharest,zagreb,365). to_fro_dt(budapest,ljubljana,540). to_fro_dt(budapest,vienna,165). to_fro_dt(budapest,warsaw,680). to_fro_dt(budapest,zagreb,365). to_fro_dt(catania,naples,450). to_fro_dt(cologne,frankfurt,82). to_fro_dt(copenhagen,hamburg,270). to_fro_dt(copenhagen,oslo,520). to_fro_dt(copenhagen,stockholm,315). to_fro_dt(cork,dublin,165). to_fro_dt(dublin,holyhead,195). to_fro_dt(dublin,westport,210). to_fro_dt(edinburgh,glasgow,50). to_fro_dt(faro,lisbon,230). to_fro_dt(florence,rome,95). to_fro_dt(florence,venice,123). to_fro_dt(frankfurt,hamburg,220). to_fro_dt(frankfurt,munich,190). to_fro_dt(frankfurt,paris,235). to_fro_dt(hamburg,munich,350). to_fro_dt(helsinki,rovaniemi,570). to_fro_dt(helsinki,turku,110). to_fro_dt(heraklion,piraeus,390). to_fro_dt(igoumenitsa,patras,360). to_fro_dt(istanbul,sofia,775). to_fro_dt(istanbul,thessaloniki,720). to_fro_dt(kiruna,stockholm,960). to_fro_dt(lisbon,madrid,610). to_fro_dt(lisbon,porto,165). to_fro_dt(ljubljana,venice,540). to_fro_dt(ljubljana,zagreb,140). to_fro_dt(london,paris,135). to_fro_dt(london,penzance,305). to_fro_dt(lyon,marseille,100). to_fro_dt(lyon,paris,115). to_fro_dt(madrid,'málaga',165). to_fro_dt(madrid,pamplona,180). to_fro_dt(madrid,santander,270). to_fro_dt(madrid,santiago,425). to_fro_dt(madrid,sevilla,155). to_fro_dt(madrid,valencia,105). to_fro_dt(marseille,montpellier,140). to_fro_dt(marseille,nice,155). to_fro_dt(milan,munich,465). to_fro_dt(milan,nice,310). to_fro_dt(milan,venice,155). to_fro_dt(munich,prague,365). to_fro_dt(munich,venice,425). to_fro_dt(munich,vienna,250). to_fro_dt(naples,rome,70). to_fro_dt(oslo,stockholm,380). to_fro_dt(paris,rennes,120). to_fro_dt(piraeus,rhodes,710). to_fro_dt(prague,vienna,270). to_fro_dt(prague,warsaw,520). to_fro_dt(sarajevo,zagreb,550). to_fro_dt(skopje,sofia,540). to_fro_dt(skopje,thessaloniki,240). to_fro_dt(sofia,thessaloniki,400). to_fro_dt(split,zagreb,335). to_fro_dt(stockholm,/* added by hand */turku,725). to_fro_dt(stockholm,'östersund',420). to_fro_dt(trondheim,'östersund',230). to_fro_dt(venice,vienna,440). to_fro_dt(vienna,warsaw,450).
... let's find paths that
start in Vienna
include at least 2 other cities
and have a cumulative travel time of 10 hours (or less)!
?- W #=< 600, Path = [_,_,_|_], start_end_path_weight(vienna, _, Path, W).
W = 530, Path = [vienna,budapest,zagreb] ;
W = 595, Path = [vienna,munich,berlin] ;
W = 440, Path = [vienna,munich,frankfurt] ;
W = 522, Path = [vienna,munich,frankfurt,cologne] ;
W = 600, Path = [vienna,munich,hamburg] ;
W = 545, Path = [vienna,prague,berlin] ;
W = 563, Path = [vienna,venice,florence] ;
W = 600, Path = [vienna,venice,florence,bologna] ;
W = 595, Path = [vienna,venice,milan] ;
false. % terminates universally fast