I have an app on the app store that uses AFNetworking 2.x to download large files in the background with NSURLSession-based downloads, because the user will often put the app in the background (it gets terminated after a while of course, but the downloads finish all the same. Wonderful). This app is working well. Usually users are only downloading a few files at a time.
Now I need to make another similar app, but this time instead of a few large files, it is very likely that the user will want to download a large number of smallish files: for example, 500 files that are 1-5mb each. Again, the app will often be put in the background, so I want to stay with NSURLSessionDownloadTask unless there's a really good reason not to.
My question is, can I simply create 500 NSURLSessionDownloadTasks all at once? Does AFNetworking do some clever throttling so as not to overload the system? Or does iOS do it? Or does nothing do it, and I have to painfully track & organize the state of transfers across restarts of my app (ie. because it gets put in the background eventually terminated) ?
If anyone knows the limits of how many NSURLSessionDownloadTasks you can create reliably simultaneously, that would be awesome...
thanks!
p.s. I greatly prefer obj-c to swift, thx :)
Last I checked (haven't looked at the iOS 9 betas), task creation was unexpectedly expensive and also superlinear. On my test runs:
50 tasks -> ~1.5s
200 tasks -> ~11.5s
500 tasks -> ~55s
Since my file count was often a 5-digit number, scheduling everything at once wasn't a solution for me. My approach (which isn't in production yet, I stopped working on the feature in favour of other things), combines persistence with NSURLSessionDownloadTask and uses the session identifiers to sort out which logical download a particular file belongs to. Further downloads are scheduled from one of the delegates depending on whether I'm on the normal lifecycle or coming from -application:handleEventsForBackgroundURLSession:completionHandler: (debugging this situation can get painful; NSUserDefaults is your friend). The theory seems sound, I can see that tasks do get scheduled, but I'm currently stuck getting the iOS downloader daemon to conform to my will.
If the server-side zip as suggested by Benjamin Jimenez is an option for you, do yourself a favour and use that instead.
The Apple staff member "eskimo" on apple developer forums helped me find the answer, which you can see in this forum post:
https://forums.developer.apple.com/thread/11621
Pasting here the relevant parts:
(me) I've read through this thread and the one you linked to here
(https://devforums.apple.com/message/938057#938057) and I have a
question about best practices to download 10,000-20,000 files via
NSURLSessionDownloadTasks. (Disclaimer, i'm using AFNetworking 2.x).
I'm targeting iOS 8 and newer, so answers do not have to work on iOS
7. How can we compute a reasonable batch (group) size ? I understand the resume-rate limiter means one wants the batch size to be higher,
but there's an unknown max limit of simultaneous task requests that
will crash the daemon.
(me) My assumption here is that when the user opens my app and it runs
for some time in the foreground, then the rate limiter is "reset" or
similar -- so now things will flow nicely again. Is this assumption
correct?
(eskimo) Yes. Also, starting with iOS 8, if the user brings your app
to the front then iOS will automatically give tasks a 'kick'. I've
forgotten the exact mechanics of this but I'm pretty sure it's covered
in WWDC 2014 Session 707 What's New in Foundation Networking.
Related
Currently what I want to achieve is download files from an array that download only one file at a time and it still performs download even the app goes to the background state.
I'm using Rob code as stated in here but he's using URLSessionConfiguration.default which I want to use URLSessionConfiguration.background(withIdentifier: "uniqueID") instead.
It did work in the first try but after It goes to background everything became chaos. operation starts to download more than one file at a time and not in order anymore.
Is there any solution to this or what should I use instead to achieve what I want. If in android we have service to handle that easily.
The whole idea of wrapping requests in operation is only applicable if the app is active/running. It’s great for things like constraining the degree of concurrency for foreground requests, managing dependencies, etc.
For background session that continues to proceed after the app has been suspended, though, none of that is relevant. You create your request, hand it to the background session to manage, and monitor the delegate methods called for your background session. No operations needed/desired. Remember, these requests will be handled by the background session daemon even if your app is suspended (or if it terminated in the course of its normal lifecycle, though not if you force quit it). So the whole idea of operations, operation queues, etc., just doesn’t make sense if the background URLSession daemon is handling the requests and your app isn’t active.
See https://stackoverflow.com/a/44140059/1271826 for example of background session.
By the way, true background sessions are really useful when download very large resources that might take a very long time. But it introduces all sorts of complexities (e.g., you often want to debug and diagnose when not connected to the Xcode debugger which changes your app lifecycle, so you have to resort to mechanisms like unified messaging; you need to figure out how to restore UI if the app was terminated between the time the requests were initiated and when they finished; etc.).
Because of this complexity, you might want to consider whether this is absolutely needed. Sometimes, if you only need less than 30 seconds to complete some requests, it’s easier to just ask the OS to keep your app running in the background for a little bit after the user leaves the app and just use standard URLSession. For more information, see Extending Your App's Background Execution Time. It’s a much easier solution, bypassing many background URLSession hassles. But it only works if you only need 30 seconds or less. For larger requests that might exceed this small window, a true background URLSession is needed.
Below, you asked:
There are some downside with [downloading multiple files in parallel] as I understanding.
No, it’s always better to allow downloads to progress asynchronously and in parallel. It’s much faster and is more efficient. The only time you want to do requests consecutively, one after another, is where you need the parse the response of one request in order to prepare the next request. But that is not the case here.
The exception here is with the default, foreground URLSession. In that case you have to worry about latter requests timing out waiting for earlier requests. In that scenario you might bump up the timeout interval. Or we might wrap our requests in Operation subclass, allowing us to constrain not only how many concurrent requests we will allow, but not start subsequent requests until earlier ones finish. But even in that case, we don’t usually do it serially, but rather use a maxConcurrentOperationCount of 4 or something like that.
But for background sessions, requests don’t time out just because the background daemon hasn’t gotten around to them yet. Just add your requests to the background URLSession and let the OS handle this for you. You definitely don’t want to download images one at a time, with the background daemon relaunching your app in the background when one download is done so you can initiate the next one. That would be very inefficient (both in terms of the user’s battery as well as speed).
You need to loop inside an array of files and then add to the session to make it download but It will be download asynchronously so it's hard to keeping track also since the files are a lot.
Sure, you can’t do a naive “add to the end of array” if the requests are running in parallel, because you’re not guaranteed the order that they will complete. But it’s not hard to capture these responses as they come in. Just use a dictionary for example, perhaps keyed by the URL of the original request. Then you can easily look up in that dictionary to find the response associated with a particular request URL.
It’s incredibly simple. And we now can perform requests in parallel, which is much faster and more efficient.
You go on to say:
[Downloading in parallel] could lead the battery to be high consumption with a lot of requests at the same time. that's why I tried to make it download each file one at a time.
No, you never need to perform downloads one at a time for the sake of power. If anything, downloading one at a time is slower, and will take more power.
Unrelated, if you’re downloading 800+ files, you might want to allow the user to not perform these requests when the user is in “low data mode”. In iOS 13, for example, you might set allowsExpensiveNetworkAccess and allowsConstrainedNetworkAccess.
Regardless (and especially if you are supporting older iOS versions), you might also want to consider the appropriate settings isDiscretionary and allowsCellularAccess.
Bottom line, you want to make sure that you are respectful of a user’s limited cellular data plan or if they’re on some expensive service (e.g. connecting on an airplane’s expensive data plan or tethered via some local hotspot).
For more information on these considerations, see WWDC 2019 Advances in Networking, Part 1.
My question involves keeping an app that monitors user interactions in the background, for example time spent in one or the app. The issue arises when you can not have a background process run for more than 10 min or violate Apple's sandbox restrictions. Since I am relatively new to the Apple API, and could not find a direct answer that didn't involve location services or VOIP (which are both interesting options, but my application will not be able to use either viably), I come to ask for options in which I can understand when another app opens, when it closes, when it fetches data, and when user holds phone in certain orientation (ie when people hold their phone at certain angles to read text and etc.) for certain amount of time.
The purpose of this analyzation is to predict an attention span for the user, so I need to be able to run in the background, as the user will not be using my app while it predicts attention span.
My thoughts on this are possibly accessing the system log, and somehow parse previous statements (I don't think sandbox will allow), but inevitably the iOS system will suspend my processes after some point unless I put a timer. There is also the option of having the system wake up my app via opportunistic fetching, but that won't be useful if I don't collect the data.
Keep in mind this is within IOS 11, so it is much more restrictive than previous iterations. I understand this may seem like a complex problem, but even a direction in which to head could be useful to me.
This solution might work, (not recommended since it drains the battery quicker).
Just update your current location, every 10 mins. It will reset the background thread timer.
Scenario:
As a user I am able to take (an unlimited amount of) photos and videos which are stored in the apps documents folder. Each of these media files gets a record within a Sqlite database with additional information (for exeample a caption). All this is possible to do completely offline.
Back online I get a dialog with a list of all the videos and photos I took and a button which starts an upload process.
Each file is uploaded after the other together with its metadata by making a multipart POST request to the server. The response of the server is stored together with the metadata in the Sqlite database (so there is no fire and forget).
Reliable solutions?
If I am reading and understanding this chart correctly, the most simple solution would be to wrap each of these uploads within a Task. Side effect: after 10 minutes every task would be cancelled, which becomes a problem by having a slow connection or very large files (for example a very long video).
The recommended way would be to use NSUrlSession/Background transfer service.
Which leads me to my question:
Is it possible to wrap multipart POSTs in NSURLSessionDataTasks and would this be reliable, even if the task is running longer than 10 minutes or the user is suspending the app?
As I am a Xamarin/C# guy I would really appreciate some sample snippets for a working multipart upload, even if it's in Objective-C ;-).
Almost and... yes.
Background Transfer service works with NSUrlSessionDownloadTasks and NSUrlSessionUploadTasks only. Not NSUrlSessionDataTasks, as described here.
Other than this "basic" limitation, it's safe to use background transfer service with upload tasks.
The 10-minute-freepass-in-the-background no longer applies on iOS 7 (basically, it's there, but different), however, with NSURLSession and background transfer service you do not need it.
I've a blog post here for background transfer service, based on download tasks.
An important thing to note is that, starting a task basically means that it will actually start sometime and actually finish some other time. This depends on whether the device is on cellular or Wi-Fi and other factors which are (probably) only known to iOS (and Apple).
I have an iPhone app which pretty much is a mobile app for a website. Pretty much everything it does is call API methods from our server. The app retrieves the user's information, and keeps updating the server using the API.
My colleague and I had a discussion whether to introduce GCD to the downloading aspect on the app. My colleague argues that since the UI needs to wait for the download to complete before it can display the pictures, text or whatever, there is absolutely no need for GCD. My argument is that we should keep the main thread busy with UI rendering (even if there is no data), and introduce GCD to the app to create other threads for download.
Which argument is right here? In my case, if the UI renders with no data, will there be some sort of lag? Which will yield a cleaner, sleeker and faster app?
One argument would be : what will happen when the download fails and times out because there is a problem at the server end ?
Without GCD the app will remain blocked and will crash after a time
out since the UI can not be blocked for longer than 20 seconds.
With GCD the application remains functional but there will be no data
being downloaded and the application will not crash.
Other aspects to take into account are :
the thread safety of the objects that you are using
how you handle downloads that are no longer necessary because the user navigates away from the page
I don't think doing time consuming operations in the main thread is a good idea.
Even if user have to wait for the data te be downloaded before he can do anything meaningful, still he will not hope UI is blocked.
Let's assume you have a navigator view, and after user tap some button, you push a new view to it and start download something. If user suddenly decides he don't want to wait anymore, he tap the "back" button. If your downloading operation blocks UI, user will have to wait it to end, it's really bad.
A more appropriate question would perhaps be if you should download asynchronously or on the main thread for your app, since there are several different methods to download asynchronously on iOS (e.g. using NSThread, NSOperation or indeed GCD). An easy approach to achieve your goals could be to use the AFNetworking library. It makes multithreaded networking / internet code very easy to implement and understand.
Personally I'm very fond of GCD and recommend you learn it someday soon, though in itself it is not as suitable for asynchronous downloading compared to a library like AFNetworking (that uses GCD under the hood, I believe).
Here is a good read on using NSOperationQueues (that uses GCD behind the scenes) to download images. There is also some Github code you can check out. They have an elegant solution to pause downloads and enqueue new downloads when the user moves to different parts of your app.
http://eng.alphonsolabs.com/concurrent-downloads-using-nsoperationqueues/?utm_medium=referral&utm_source=pulsenews
Use GCD / NSOperationQueues as opposed to using NSThreads. You will have a good learning on core fundamentals and at the same time create a well architectured app. :)
My app needs to encode a large amount of audio data to an M4A file. I am currently using AVAssetWriter, which works fine, except that it takes a few minutes to encode all the data.
Instead of asking the user to keep the app running until the process has finished, I would like to pause the encoding when the app terminates and continue on relaunch.
Unfortunately, AVAssetWriter doesn't seem to support this, as it always creates a new file when initializing.
Do you know any other APIs that I could use? Maybe a third-party library?
This is exactly what background processing is intended for. As long as you can complete within 10 minutes, you can use beginBackgroundTaskWithExpirationHandler: to ask the system to let you keep running after the user switches applications. See Completing a Finite-Length Task in the Background in the iOS Application Programming Guide. Not only is this the easiest to use, but it'll give the best user experience.
The only issue you'd face is if it's possible for an audio file to take longer than 10 minutes in a non-resumable way. If that's a real possibility, then you'll need another solution.