I’m playing with the new VNGeneratePersonSegmentationRequest Vision API to make a simple background removal filter
I made a small project to test it, works great, but I’m running into issues with memory. After executing the request the app’s memory consumption adds 300MBs that are never freed.
I’ve cycled through a bunch of images and requests in a test run and thankfully memory consumption remains constant even when filtering more images, but I worry about that initial memory that is never freed, even when inducing a memory warning. I suspect the Vision Framework needs that memory after being called, but my app doesn’t handle video frames or anything, so it’s memory going to waste
//Autorelease pool doesn't helps
autoreleasepool {
// Create request
let request = VNGeneratePersonSegmentationRequest()
request.revision = VNGeneratePersonSegmentationRequestRevision1
request.qualityLevel = .accurate
request.outputPixelFormat = kCVPixelFormatType_OneComponent8
let handler:VNImageRequestHandler = VNImageRequestHandler(ciImage: inputs.ciImage,
options: [:])
//A jump in memory usage after running the handler
//Subsequent calls don't add to memory usage
do{
try handler.perform([request])
}
catch{
return
}
//Even if I delete this chunk of code, memory consumption remains high
let mask = request.results?.first!
if let maskBuffer = mask?.pixelBuffer{
self.personMask = CIImage(cvPixelBuffer: maskBuffer)
let maskScaleX = inputs.ciImage.extent.width / personMask!.extent.width
let maskScaleY = inputs.ciImage.extent.height / personMask!.extent.height
self.personMask = personMask!.transformed(by: __CGAffineTransformMake(
maskScaleX, 0, 0, maskScaleY, 0, 0))
}
}
Related
I have a navigation and location tracking app. While a user is tracking his trip, coordinates, speed, timestamp (and a few more) are logged with each coordinate that comes in. I don't want to store this in memory as that would make the app memory grow as the user moves along, eventually leading to a didReceiveMemoryWarning and even an app crash. (At least that's been my experience so far)
What would be the most efficient way to do this? Thinking of battery consumption, CPU usage and also memory usage if that comes into play.
I can think of two options to do this:
Log in a file (what I'm currently doing, using this code snipped):
let newLine: String = "bla bla bla"
let url: URL = URL(string: "someLocalPath")
newLine.appendToURL(url)
extension String {
func appendToURL(_ fileURL: URL) throws {
let data = self.data(using: String.Encoding.utf8)!
try data.appendToURL(fileURL)
}
}
extension Data {
func appendToURL(_ fileURL: URL) throws {
if let fileHandle = try? FileHandle(forWritingTo: fileURL) {
defer {
fileHandle.closeFile()
}
fileHandle.seekToEndOfFile()
fileHandle.write(self)
fileHandle.closeFile()
}
else {
try write(to: fileURL, options: .atomic)
}
}
}
Using Core Data
I have not tried this yet, but it would be relatively easy I believe. Just creating an entity with all the fields required, and then adding a managedObject for each received coordinate.
So, to repeat my question: which of these two options would be more efficient from a battery, CPU and memory perspective?
Additional question: Is there perhaps another way to do this. One that I didn't think of?
I found another option that works best in my case. Should have thought of that in the first place 🤦🏻♂️.
So, I did not want to log to memory because it could make iOS kill my app due to excessive memory usage. But, iOS sends a didReceiveMemoryWarning first, upon which you can reduce the memory footprint of the app, preventing it from being killed.
So, what I do now:
I added a instance variable:
fileprivate var logString: String = ""
And log to it when a new coordinate is received:
logString += newLine
If / when I get a didReceiveMemoryWarning I write the entire logString to the file on disk and empty it afterwards:
override func didReceiveMemoryWarning() {
logString.appendToURL(url)
logString = ""
}
This way I only write to disk when necessary (and not with each coordinate received), which is a lot more energy efficient. (Writing to disk costs more energy than writing to memory. Source: https://developer.apple.com/videos/play/wwdc2016/719/)
Plus, my app is prevented from being killed by iOS due to high memory usage, because I clear up memory on time.
What is difference between URLSession vs DispatchQueue.global().async + Data(contentsOf: ) in terms of download images from image urls?
func loadImageWithUrlSession() {
guard let url = URL(string: IMAGE_URL) else { return }
URLSession.shared.dataTask(with: url) { (data, response, error) in
if let error = error {
print(error.localizedDescription)
return
}
guard let data = data else { return }
let image = UIImage(data: data)
DispatchQueue.main.async { [weak self] in
guard let self = self else { return }
self.urlSessionImageView.image = image
}
}.resume()
}
func loadImageWithGCD() {
DispatchQueue.global(qos: .background).async {
guard
let url = URL(string: self.IMAGE_URL),
let data = try? Data(contentsOf: url) else {
return
}
let image = UIImage(data: data)
DispatchQueue.main.async { [weak self] in
guard let self = self else { return }
self.gcdImageView.image = image
}
}
}
I know that URLSession can cancel or suspend task.
But if I use Rx instead, I can do the same thing as above as well.
I had an experiment that and it was depending on which QoS I'm using.
By the way, .userInitiated QoS was way faster than URLSession.
Which one are you guys use for something like downloading task with a background thread and why?
Is there any kind-teacher can help me specifically?
URLSession offers far greater configuration control, diagnostics of failures, cancelation, background sessions, ability to download directly to persistent storage to minimize peak memory usages, etc. URLSession and Data(contentsOf:) just are not comparable on feature set.
The synchronous Data(contentsOf:) unnecessarily blocks GCD worker threads and is also susceptible to misuse. It also is quite limiting and you will easily find yourself regretting the decision in the future (e.g. you later add some authentication process; you want to customize the cache behaviors, you want to parse and act upon status codes in the responses, you need cancelation capabilities because you are retrieving images for collection or table views, etc.).
It’s illuminating to look at the documentation for one of the init with a URL methods for Data, where it warns us:
Important
Don't use this synchronous initializer to request network-based URLs. For network-based URLs, this method can block the current thread for tens of seconds on a slow network, resulting in a poor user experience, and in iOS, may cause your app to be terminated.
Instead, for non-file URLs, consider using the dataTask(with:completionHandler:) method of the URLSession class. See Fetching Website Data into Memory for an example.
Yes, dispatching this to a background thread addresses many of the above concerns, but Apple didn’t just suggest “just dispatch this to some background queue,” but rather explicitly advised to use URLSession instead. While your use of GCD global queue avoids some of issues that Apple warns us of above, it also imposes many unnecessarily limitations. If you use Data(contentsOf:), this is a decision that you’ll likely regret/refactor in the future. You might as well use URLSession now.
Regarding Data(contentsOf:) being appreciably faster when using .userInitiated, vs .default or URLSession approach, usually the network latency and transmission time dwarfs any queue priority related factors, so I find that claim hard to believe. In fact, I just tested download of 50 images via GCD (using both .default and .userInitiated) and the speed was not appreciably different than URLSession approach.
I'm looping through all pages in a PDFDocument (200+ pages) but app crashes with
Message from debugger: Terminated due to memory issue
The pdf is approx 4mb in size yet each iteration of the loop jumps the memory up approx 30mb. Which doesn't seem right to me. I have managed to locate where in my code the memory is being used just not sure how to claim it back. Tried setting variables to nil but no effect. Tried code in the for loop in an autoreleaspool{} but no effect.
#objc func scrapePDF(){
let documentURL = self.documentDisplayWebView!.url!
let document = PDFDocument(url: documentURL)
let numberOfPages = document!.pageCount
DispatchQueue.global().async {
for pageNumber in 1...numberOfPages {
print(document?.page(at: pageNumber)!.string!)
}
}
}
UPDATE: solved ..... kind of
Playing around a bit I found that rather than passing a reference to the PDFDocument inside the loop, if instead I create a new instance for each loop this strangely solves the memory issue. I don't quite understand why though. PDFDocument is a Class not a Struct so is passed by reference. Meaning it is only created once and then referenced to inside my loop. So why would it cause a memory issue?
#objc func scrapePDF(){
let documentURL = self.documentDisplayWebView!.url!
let document = PDFDocument(url: documentURL)
let numberOfPages = document!.pageCount
DispatchQueue.global().async {
for pageNumber in 1...numberOfPages {
let doc = PDFDocument(url: documentURL)
print(doc?.page(at: pageNumber)!.string!)
}
}
}
Though the above code clears the memory issue the problem with it is that its too slow. Each loop takes 0.5 seconds and with 300+ pages I can't accept that. Any tips on speeding it up? Or why it doesn't give the memory back if referencing the PDFDocument from outside the loop
Further UPDATE.
It seems that it’s calling the .string method of the PDFPage that is increases the memory to the point of crashing.
I'm probably missing something. I'm trying to change filter to my GPUImageView.It's actually working the first two times(sometimes only one time), and than stop responding to changes. I couldn't find a way to remove the target from my GPUImageView.
Code
for x in filterOperations
{
x.filter.removeAllTargets()
}
let f = filterOperations[randomIntInRange].filter
let media = GPUImagePicture(image: self.largeImage)
media?.addTarget(f as! GPUImageInput)
f.addTarget(g_View)
media.processImage()
Any suggestions? * Processing still image from my library
UPDATE
Updated Code
//Global
var g_View: GPUImageView!
var media = GPUImagePicture()
override func viewDidLoad() {
super.viewDidLoad()
media = GPUImagePicture(image: largeImage)
}
func changeFilter(filterIndex : Int)
{
media.removeAllTargets()
let f = returnFilter(indexPath.row) //i.e GPUImageSepiaFilter()
media.addTarget(f as! GPUImageInput)
f.addTarget(g_View)
//second Part
f.useNextFrameForImageCapture()
let sema = dispatch_semaphore_create(0)
imageSource.processImageWithCompletionHandler({
dispatch_semaphore_signal(sema)
return
})
dispatch_semaphore_wait(sema, DISPATCH_TIME_FOREVER)
let img = f.imageFromCurrentFramebufferWithOrientation(img.imageOrientation)
if img != nil
{
//Useable - update UI
}
else
{
// Something Went wrong
}
}
My primary suggestion would be to not create a new GPUImagePicture every time you want to change the filter or its options that you're applying to an image. This is an expensive operation, because it requires a pass through Core Graphics and a texture upload to the GPU.
Also, since you're not maintaining a reference to your GPUImagePicture beyond the above code, it is being deallocated as soon as you pass out of scope. That tears down the render chain and will lead to a black image or even crashes. processImage() is an asynchronous operation, so it may still be in action at the time you exit your above scope.
Instead, create and maintain a reference to a single GPUImagePicture for your image, swap out filters (or change the options for existing filters) on that, and target the result to your GPUImageView. This will be much faster, churn less memory, and won't leave you open to premature deallocation.
Hi I am developing an app in which I require to chache 50 images (size of all images is 2.5 mb) ,It is chaching the images but also increases the memory by 10 mb in Apple Watch App due to which app crashes .
Xcode gives error in xCode “Message from debugger: Terminated due to memory error”
The code i am using is below :
for (var i : Int = 1; i<26; i++) {
let filenameHuman = NSString(format: "human_%d", i )
let filenameZombie = NSString(format: "zombie_%d", i )
var imageHuman : UIImage! = UIImage(named: filenameHuman as String)
var imageZombie : UIImage! = UIImage(named: filenameZombie as String)
WKInterfaceDevice.currentDevice().addCachedImage(imageZombie, name: filenameZombie as String)
WKInterfaceDevice.currentDevice().addCachedImage(imageHuman, name: filenameHuman as String)
}
NSLog("Currently cached images: %#",WKInterfaceDevice.currentDevice().cachedImages)
Also the screenshot of Memory allocation and memory leak is :
Please Help, Thanks in advance .
Are any of your images actually animations (that would use up more space)?
Collect the return value of each call to addCachedImage(). False means it could not be added -- you need to check to that and it might give clues as to a particular problem image.
Before calling anything, try to empty the cache, removeAllCachedImages. This means you will be clean from previous cache interactions using up the pool of memory.
I think your problem is not a leak, I think your problem is over-retained allocations. So use the Allocations tool (with retain count tracking) to see how much memory was allocated (VM allocations) and how many entities are holding on to such memory (retain counts).
Inside your loop try to autorelease the memory that is used by the images since you don't want to wait for autorelease to happen later when the method returns.
for (var i : Int = 1; i<26; i++) {
autoreleasepool {
/* code to cache images */
}
}