I would like to use dask.array.map_overlap to deal with the scipy interpolation function. However, I keep meeting errors that I cannot understand and hoping someone can answer this to me.
Here is the error message I have received if I want to run .compute().
ValueError: could not broadcast input array from shape (1070,0) into shape (1045,0)
To resolve the issue, I started to use .to_delayed() to check each partition outputs, and this is what I found.
Following is my python code.
Step 1. Load netCDF file through Xarray, and then output to dask.array with chunk size (400,400)
df = xr.open_dataset('./Brazil Sentinal2 Tile/' + data_file +'.nc')
lon, lat = df['lon'].data, df['lat'].data
slon = da.from_array(df['lon'], chunks=(400,400))
slat = da.from_array(df['lat'], chunks=(400,400))
data = da.from_array(df.isel(band=0).__xarray_dataarray_variable__.data, chunks=(400,400))
Step 2. declare a function for da.map_overlap use
def sumsum2(lon,lat,data, hex_res=10):
hex_col = 'hex' + str(hex_res)
lon_max, lon_min = lon.max(), lon.min()
lat_max, lat_min = lat.max(), lat.min()
b = box(lon_min, lat_min, lon_max, lat_max, ccw=True)
b = transform(lambda x, y: (y, x), b)
b = mapping(b)
target_df = pd.DataFrame(h3.polyfill( b, hex_res), columns=[hex_col])
target_df['lat'] = target_df[hex_col].apply(lambda x: h3.h3_to_geo(x)[0])
target_df['lon'] = target_df[hex_col].apply(lambda x: h3.h3_to_geo(x)[1])
tlon, tlat = target_df[['lon','lat']].values.T
abc = lNDI(points=(lon.ravel(), lat.ravel()),
values= data.ravel())(tlon,tlat)
target_df['out'] = abc
print(np.stack([tlon, tlat, abc],axis=1).shape)
return np.stack([tlon, tlat, abc],axis=1)
Step 3. Apply the da.map_overlap
b = da.map_overlap(sumsum2, slon[:1200,:1200], slat[:1200,:1200], data[:1200,:1200], depth=10, trim=True, boundary=None, align_arrays=False, dtype='float64',
)
Step 4. Using to_delayed() to test output shape
print(b.to_delayed().flatten()[0].compute().shape, )
print(b.to_delayed().flatten()[1].compute().shape)
(1065, 3)
(1045, 0)
(1090, 3)
(1070, 0)
which is saying that the output from da.map_overlap is only outputting 1-D dimension ( which is (1045,0) and (1070,0) ), while in the da.map_overlap, the output I am preparing is 2-D dimension ( which is (1065,3) and (1090,3) ).
In addition, if I turn off the trim argument, which is
c = da.map_overlap(sumsum2,
slon[:1200,:1200],
slat[:1200,:1200],
data[:1200,:1200],
depth=10,
trim=False,
boundary=None,
align_arrays=False,
dtype='float64',
)
print(c.to_delayed().flatten()[0].compute().shape, )
print(c.to_delayed().flatten()[1].compute().shape)
The output becomes
(1065, 3)
(1065, 3)
(1090, 3)
(1090, 3)
This is saying that when trim=True, I cut out everything?
because...
#-- print out the values
b.to_delayed().flatten()[0].compute()[:10,:]
(1065, 3)
array([], shape=(1045, 0), dtype=float64)
while...
#-- print out the values
c.to_delayed().flatten()[0].compute()[:10,:]
array([[ -47.83683837, -18.98359832, 1395.01848583],
[ -47.8482856 , -18.99038681, 2663.68391094],
[ -47.82800624, -18.99207069, 1465.56517187],
[ -47.81897323, -18.97919009, 2769.91556363],
[ -47.82066663, -19.00712956, 1607.85927095],
[ -47.82696896, -18.97167714, 2110.7516765 ],
[ -47.81562653, -18.98302933, 2662.72112163],
[ -47.82176881, -18.98594465, 2201.83205114],
[ -47.84567 , -18.97512514, 1283.20631652],
[ -47.84343568, -18.97270783, 1282.92117225]])
Any thoughts for this?
Thank You.
I guess I got the answer. Please let me if I am wrong.
I am not allowing to use trim=True is because I change the shape of output array (after surfing the internet, I notice that the shape of output array should be the same with the shape of input array). Since I change the shape, the dask has no idea how to deal with it so it returns the empty array to me (weird).
Instead of using trim=False, since I didn't ask cutting-out the buffer zone, it is now okay to output the return values. (although I still don't know why the dask cannot concat the chunked array, but believe is also related to shape)
The solution is using delayed function on da.concatenate, which is
delayed(da.concatenate)([e.to_delayed().flatten()[idx] for idx in range(len(e.to_delayed().flatten()))])
In this case, we are not relying on the concat function in map_overlap but use our own concat to combine the outputs we want.
This is NOT a Beta issue. I am on Xcode 6.0.1, production release. The issue I am having is that when I try to do a Build or Run the code I am working on, Xcode becomes unresponsive for large periods of time and the SourceKitService consumes upwards of 400% of the CPU (according to Activity Monitor). This issue is new as of the last few days, although, oddly, I had been on Xcode 6.0 since it was officially released on Sept 17. I upgraded to 6.0.1 hoping it would contain a fix for this issue.
Any idea as to what the problem could be?
Ran into this problem with Xcode 6.1.1 earlier this afternoon (not beta, official released version). I had been running some code on Playground and was suspecting that to be the cause. CPU was pegged to nearly 100%, and Xcode was unable to complete builds.
So here's what I did:
1. Opened "Activity Monitor", which showed SourceKitService as the main CPU hog.
2. Within "Activity Monitor", double-clicked on the SourceKitService and clicked on "Open Files and Ports" section, which showed it was working on files under the /Users/myname/Library/Developer/Xcode/DerivedData/ModuleCache/ directory for a specific folder.
3. Deleted the specified folder (from a command-line, using rm -rf). The cache is regenerated based on Can I safely delete contents of Xcode Derived data folder? .
4. Using Activity Monitor again, Force-Quit SourceKitServer. Saw the now-all-too-familiar sign within Xcode saying that SourceKitService had crashed (so that's why SourceKitService sounded familiar!).
5. Repeated step 3.
The Mac is peaceful, again. No data was lost and Xcode didn't even have to be restarted (which I had tried unsuccessfully). Bottom line is that ModuleCache seems to be getting SourceKitService in a loop and deleting the folder seems to fix it. Hope this works for you too.
Bootnote:By the way, the cause for SourceKitService issue was that I had too long an array declaration in my Swift class. I had over 200 entries in an array. Reduced it to 30 and the error went away. So the issue may have arisen due to some kind of stack overflow in apple code (pun intended).
I was seeing the problem because I was declaring an array with about 60 elements that looked like this:
let byteMap = [
["ECG" : (0,12)],
["PPG" : (12,3)],
["ECG" : (15,12)],
["PPG" : (27,3)],
["ECG" : (30,12)]
By explicitly annotating the type like this:
let byteMap : [String: (Int, Int)] = [
["ECG" : (0,12)],
["PPG" : (12,3)],
["ECG" : (15,12)],
["PPG" : (27,3)],
["ECG" : (30,12)],
I was able to make it stop. I think it must have something to do with Swift's type-inference and type-checking that makes it go into a loop when it encounters a longish array.
This was in Xcode 6.2. I also deleted the ModuleCache as described above and now everything is good.
This problem happened like 10 times, 8 times it happened when I connected an actual device and didn't run through simulator.
I am not so sure if my solution is a good one, but for me I believe the problem was due to switching between simulator and an actual device. It may sound weird but it was as if it was creating interference between cache files.
What solved my problem:
Clean Build Folder:( on Xcode) Alt + Shift + Command + K
Reset Content and Settings: (on
Simulator) Command + Shift + K.
Waited a bit longer than normal and overload Xcode with constant clicks
So basically before you try to run on any new device, just delete any cache.
EDIT
I just had the problem without any device connection. I just quit Xcode and opened it again and the problem was gone. Not sure my guess is it could be some re-indexing issue after you fetch/pull merge new code.
I resolved another issue that was causing SourceKitService use up to 13GB of memory...
I had String(format line with lots of arguments:
return String(format: "%d,%.3f,%.3f,%.3f,%.3f,%.3f,%.3f,%.3f,%.3f,%.3f,%.3f,%.3f,%.3f,%.3f,%.3f,%.3f,%.3f,%.3f,%.3f", samples.count,sum1.x,sum1.y,sum1.z,sum1.rx,sum1.ry,sum1.rz,sum2.x,sum2.y,sum2.z,sum2.rx,sum2.ry,sum2.rz,sum3.x,sum3.y,sum3.z,sum3.rx,sum3.ry,sum3.rz)
when replaced with this it worked fine (no memory build up and normal CPU consumption)
var output: String = ""
output += String(format: "%d,", samples.count)
output += String(format: "%.3f,%.3f,%.3f,", sum1.x, sum1.y, sum1.z)
output += String(format: "%.3f,%.3f,%.3f,", sum1.rx, sum1.ry, sum1.rz)
output += String(format: "%.3f,%.3f,%.3f,", sum2.x, sum2.y, sum2.z)
output += String(format: "%.3f,%.3f,%.3f,", sum2.rx, sum2.ry, sum2.rz)
output += String(format: "%.3f,%.3f,%.3f,", sum3.x, sum3.y, sum3.z)
output += String(format: "%.3f,%.3f,%.3f", sum3.rx, sum3.ry, sum3.rz)
return output
The problem still occurs in XCode 10.0. You can fix it by disabling "Show Source Control changes" in Source Control options.
I spend 4 hours to figure out problems in a long compilation of my project.
The first try takes 42 min to compile.
I clear all cache from /Users/myname/Library/Developer/Xcode/DerivedData/ModuleCache/ as was suggested by #LNI, after restart SourceKitService and apply few changes for code:
1)
To
var initDictionary:[String:AnyObject] = [
"details" : "",
"duration" : serviceDuration,
"name" : serviceName,
"price" : servicePrice,
"typeId" : typeID,
"typeName" : typeName,
"url" : "",
"serviceId" : serviceID,
"imageName" : ""
]
From
var initDictionary= [
"details" : "",
"duration" : serviceDuration,
"name" : serviceName,
"price" : servicePrice,
"typeId" : typeID,
"typeName" : typeName,
"url" : "",
"serviceId" : serviceID,
"imageName: "" ]
2) To
if let elem = obj.property,
let elem2 = obj.prop2,
etc
{
// do stuf here
}
From
let value1 = obj.property ?? defaultValue
3)
To
let serviceImages = images.filter { $0.serviceId == service.id }
let sorted = serviceImages.sort { $0.sort > $1.sort }
From
let serviceImages = images.filter { $0.serviceId == service.id }. sort { $0.sort > $1.sort }
As result compile time - 3 min, not so fast but better for 42 min.
As result, before SourceKitService - take ~5,2Gb of memory and after ~0.37Gb
I had the same problem with SourceKitService.
I solved. NEVER ADD SUBVIEWS WITH FOR LOOP.
To detect issue I use:
https://github.com/RobertGummesson/BuildTimeAnalyzer-for-Xcode
I've been running into this issue with Xcode 9, and explored several solutions. For me, disabling Source Control seemed to do the trick.
Xcode -> Preferences -> Source Control -> uncheck "Enable Source Control"
If this doesn't work, I would recommend using the renice command at the terminal. More on that here
disabling Source Control
Other steps that I attempted, but did not help:
Close Xcode -> Delete Derived Data
cycling machine
"clean" project
https://www.logcg.com/en/archives/2209.html
SourceKitService took charge of Swift's type inference work.
private lazy var emojiFace = ["?", "?", "?", "?"]
change to explicitly type
private lazy var emojiFace:[String] = ["?", "?", "?", "?"]
SourceKitService CPU usage immediately drop down。
For me it worked to delete the Derived Data. Select 'Product' from the menu and hold the Alt-key and select 'Clean Build Folder'. Shortkey: Alt + Shift + Command + K
Quit Xcode
Run in Terminal:
rm -rf ~/Library/Developer/Xcode/DerivedData/ModuleCache/*
Note the difference between LNI's accepted answer and this one:
It's always better not to crash than to crash. Especially, when it comes to Xcode processes/components.
I'm not an Apple developer, but partial deleting the cache can break its integrity. I didn't notice any significant delays after cleaning all the cache.
Don't create dictionary in swift without specifying data types or with [String:Any]
If we use 'Any' type the compiler might run into an infinite loop for checking the data type.
It won't create any compiling error, it will make our mac to freeze at 'compiling swift source files' with acquiring much memory for the tasks named 'swift' & 'SourceKitService'.
I ran into something similar combining multiple ?? operators to provide a default for optional string values.
I was experimenting with the debug code below when the fan on my trusty mid-2010 MacBook Pro began running hard. SourceKitService was sucking up every CPU cycle it could get. Commenting and uncommenting the offending line made it very clear what SourceKitService was choking on. It looks like using more than one ?? operator to provide a default is an issue on an old machine. The work around is just don't do it. Break it up into multiple assignments which makes some ugly debug code even uglier.
placeMark is an instance of CLPlacemark. The properties used here return optional strings.
I was using Xcode Version 8.3.2 (8E2002) running on OS 10.12.4 (16E195)
// one term is not an issue
let debugString1 = (placeMark.locality ?? "")
// two terms pushes SourceKitService CPU use to 107% for about 60 seconds then settles to 0%
let debugString1 = (placeMark.locality ?? "") + ", " + (placeMark.administrativeArea ?? "")
// three terms pushes SourceKitService CPU use to 187% indefinitely
let debugString1 = (placeMark.locality ?? "") + ", " + (placeMark.administrativeArea ?? "") + (placeMark.postalCode ?? "")
// ugly but it's safe to use
var debugString1 = placeMark.locality ?? ""
debugString1 = debugString1 + ", " + (placeMark.administrativeArea ?? "")
debugString1 = debugString1 + " " + (placeMark.postalCode ?? "")
Converting long Arrays to Functions seem to resolve the problem for me:
var color: [UIColor] {
return [
UIColor(...),
UIColor(...),
...
]
}
to:
func color() -> [UIColor] {
return [
UIColor(...),
UIColor(...),
...
]
}
I have faced such an issue. Source kit service was using 10 gb of usage. Swift process in activity monitor reaches over 6 GB usage. I was using following code:
var details : [String : Any] = ["1":1,
"2":2,
"3":3,
"4":4,
"5":5,
"6":6,
"7":7,
"8":8,
"9":9,
"10":10,
"11":11,
"12":12,
"13":13,
"14":14,
"15":15,
"16":16]
I have changed code to following to solve this issue:
var details : [String : Any] = [:]
details["1"] = 1
details["2"] = 2
details["3"] = 3
details["4"] = 4
details["5"] = 5
details["6"] = 6
details["7"] = 7
details["8"] = 8
details["9"] = 9
details["10"] = 10
details["11"] = 11
details["12"] = 12
details["13"] = 13
details["14"] = 14
details["15"] = 15
details["16"] = 16
Faced the same issue on Xcode 7.2 (7C68)
The solution was to implement a method of a protocol, which my class had in the definition.
This is still an issue in xcode Version 7.3.1 (7D1014)
the cause for me was, like LNI pointed it out, a too long array, not so long actually.
I fixed my problem by breaking the array into various arrays like this:
let firstLevel = [
[1, 0, 1, 0, 1],
[0, 0, 0, 0, 0],
[1, 0, 1, 0, 1],
[0, 0, 0, 0, 0],
[1, 0, 1, 0, 1],
[0, 0, 0, 0, 0]
]
let secondLevel = [
[0, 0, 0, 0, 0],
[0, 1, 0, 1, 0],
[0, 0, 0, 0, 0],
[0, 1, 0, 1, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]
]
let thirdLevel = [
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 1, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]
]
let map = [firstLevel, secondLevel, thirdLevel]
I had the same problem with XCode 8.2.1 (8C1002) and the following code:
import UIKit
import AVFoundation
import Photos
import CoreMotion
import Foundation
class TestViewController: UIViewController
{
let movieFileOutput = AVCaptureMovieFileOutput()
var anz_total_frames = 0, anz_total_miss = 0
#IBOutlet weak var tfStatistics: UITextView!
func showVideoStatistics()
{
let statisticText:String = "frames: \(self.anz_total_frames)" + String.newLine +
"frames/s: \(self.anz_total_frames / self.movieFileOutput.recordedDuration.seconds)" + String.newLine +
"miss: " + formatText4FramesPercent(self.anz_total_miss) + String.newLine +
"nicht erkannt: " + formatText4FramesPercent(self.anz_total_miss) + String.newLine +
"nicht erkannt: " + formatText4FramesPercent(self.anz_total_miss) + String.newLine +
"nicht erkannt: " + formatText4FramesPercent(self.anz_total_miss) + String.newLine +
"nicht erkannt: " + formatText4FramesPercent(self.anz_total_miss) + String.newLine +
"nicht erkannt: " + formatText4FramesPercent(self.anz_total_miss) + String.newLine +
"nicht erkannt: " + formatText4FramesPercent(self.anz_total_miss) + String.newLine +
"nicht erkannt: " + formatText4FramesPercent(self.anz_total_miss) + String.newLine +
"nicht erkannt: " + formatText4FramesPercent(self.anz_total_miss) + String.newLine +
"nicht erkannt: " + formatText4FramesPercent(self.anz_total_miss) + String.newLine +
"nicht erkannt: " + formatText4FramesPercent(self.anz_total_miss) + String.newLine
self.tfStatistics.text = statisticText
}
func formatText4FramesPercent(_ anz:Int) -> String
{
let perc = Double(anz)*100.0/Double(anz_total_frames)
return String(perc.format(".1") + "%")
}
}
and these extensions:
extension String {
var localized: String {
return NSLocalizedString(self, tableName: nil, bundle: Bundle.main, value: "", comment: "")
}
static var newLine: String {
return "\r\n"
}
}
extension Int {
func format(_ f: String) -> String {
return String(format: "%\(f)d", self)
}
}
extension Double {
func format(_ f: String) -> String {
return String(format: "%\(f)f", self)
}
}
I solved it by commenting this line in TestViewController:
"frames/s: \(self.anz_total_frames / self.movieFileOutput.recordedDuration.seconds)" + String.newLine +
Took me more than an hour to find it, I hope a can save some time of somebody else.
I filed a bug report to Apple with number 30103533
I was facing the same problem after migrating the project to swift 3, find out solution it was taking time because of dictionaries and array created without data type.
This behavior appeared in my project when I accidentally declared a class that inherited from itself. Xcode 8.2.1, using Swift 3.
I also had this issue, in my case I was declaring a big array like this:
var myArray: [(String, Bool?)]?
myArray = [("someString", someBool),
("someString", someBool),
("someString", someBool),
("someString", someBool),
("someString", someBool)
.
.
("someString", someBool)]
I solved the problem by adding the items 1 per line instead of all at the same time:
var myArray = [(String, Bool?)]()
myArray.append(("someString", someBool))
myArray.append(("someString", someBool))
myArray.append(("someString", someBool))
myArray.append(("someString", someBool))
myArray.append(("someString", someBool))
.
.
.
this fixed the problem.
For Objective-C projects:
I had the same problem, and there's zero Swift code in our project, so it wasn't the type inference checker.
I tried every other solution here and nothing worked - what FINALLY fixed it for me was rebooting the computer in recovery mode and running the disk repair. I can finally work in peace again!
I'm guessing that it happened because of some broken symlinks, probably pointing towards each other and making the service run around in an endless loop.
I'm having a similar issue with Xcode 8.2.1 - with a section of 1,000+ lines of code commented-out via /* */. Commenting-out the section caused the issue, and removing the commented-out code fixed it.
run in terminal:
killall Xcode
rm -rf ~/Library/Developer/Xcode/DerivedData/ModuleCache
open /Applications/Xcode.app
you could also create a terminal command using this alias:
echo alias xcodeFix='killall Xcode;rm -rf ~/Library/Developer/Xcode/DerivedData/ModuleCache;open /Applications/Xcode.app' >> ~/.profile
source ~/.profile
and then just run
xcodeFix
Happened to me on XCode 11.4.1 when calling #dynamicMemberLookup subscripts inside a SwiftUI #ViewBuilder block.
I had the same issue and it was caused by a programming error.
In my case I was implementing the comparable and equatable protocols and the lhs.param and rhs.param did not correspond with parameters of the lhs and rhs classes.