AWS Transfer Utility uploads are slow and inconsistent in iOS - ios

I have implemented the basic AWS transfer utility upload video (file) code in my app and this had been working for me flawlessly until recently the uploads got extremely slow and even stuck.
I tried changing many things in the AWS code like shifting from TransferUtilityUpload to TrasferUtility UploadUsing MultiPart, changing the AWSServiceConfiguration from AWSCognitoCredentialsProvider(using poolId & region) to AWSStaticCredentialsProvider (using Accesskey, secret key and region), enabling acceleration etc but nothing has helped to increase the upload speed. Apart from this, the uploads are very inconsistent. For example sometimes a 30sec video (size 180MB) gets uploaded in under 2 mins and then again same video takes more than 5 minutes/gets stuck in the same network (speed 150MBps or more)
Can someone please help me understand the issue and fix it?
Code snippets below.
Service Configuration
let credentialsProvider = AWSStaticCredentialsProvider(accessKey: "******", secretKey: "*****")
let configuration = AWSServiceConfiguration.init(region: AWSRegionType.USEast1, credentialsProvider: credentialsProvider)
AWSServiceManager.default().defaultServiceConfiguration = configuration
AWS Upload function
private func uploadfile(fileSize: Int, fileUrl: URL, fileName: String, contenType: String, progress: progressBlock?, completion: completionBlock?) {
// Upload progress block
var previousUploadedBytes: Double = 0.0
let expression = AWSS3TransferUtilityMultiPartUploadExpression()
expression.progressBlock = {(task, awsProgress) in
if task.status == AWSS3TransferUtilityTransferStatusType.waiting {
task.cancel()
}
guard let uploadProgress = progress else { return }
DispatchQueue.main.async {
uploadProgress(awsProgress.fractionCompleted)
//CODE FOR UI UPDATES
//DO SOMETHING WITH THE PROGRESS
}
}
// Completion block
var completionHandler: AWSS3TransferUtilityMultiPartUploadCompletionHandlerBlock?
completionHandler = { (task, error) -> Void in
DispatchQueue.main.async(execute: {
if error == nil {
let url = AWSS3.default().configuration.endpoint.url
let publicURL : URL = (url?.appendingPathComponent(self.bucketName).appendingPathComponent(fileName))!
if let completionBlock = completion {
completionBlock(publicURL.absoluteString, nil)
}
} else {
if let completionBlock = completion {
completionBlock(nil, error)
}
}
})
}
//acceleration mode enabled
let serviceConfiguration = AWSServiceConfiguration(
region: .USEast1,
credentialsProvider: AWSServiceManager.default().defaultServiceConfiguration.credentialsProvider
)
let transferUtilityConfiguration = AWSS3TransferUtilityConfiguration()
transferUtilityConfiguration.isAccelerateModeEnabled = true
AWSS3TransferUtility.register(
with: serviceConfiguration!,
transferUtilityConfiguration: transferUtilityConfiguration,
forKey: "transfer-acceleration"
)
// Start uploading using AWSS3TransferUtility
let awsTransferUtility = AWSS3TransferUtility.default()
awsTransferUtility.uploadUsingMultiPart(fileURL: fileUrl, bucket: bucketName, key: fileName, contentType: contenType, expression: expression, completionHandler: completionHandler).continueWith { (task) -> Any? in
if let error = task.error {
UploadHelper.sharedInstance.showSSLError = false
if (error as NSError).code == -1001 {
DispatchQueue.main.async {
UploadHelper.sharedInstance.noOfRetries = 0
UploadHelper.sharedInstance.changeToRetryUpload() // internal code to call for retry
}
} else if (error as NSError).code == -1009 {
DispatchQueue.main.async {
UploadHelper.sharedInstance.noOfRetries = 0
UploadHelper.sharedInstance.changeToRetryUpload() // internal code to call for retry
}
} else if (error as NSError).code == -1003 {
DispatchQueue.main.async {
UploadHelper.sharedInstance.noOfRetries = 0
UploadHelper.sharedInstance.changeToRetryUpload() // internal code to call for retry
}
} else if (error as NSError).code == -1200 {
DispatchQueue.main.async {
UploadHelper.sharedInstance.noOfRetries = 0
UploadHelper.sharedInstance.changeToRetryUpload() // internal code to call for retry
UploadHelper.sharedInstance.showSSLError = true
}
}
}
if let _ = task.result {
// your uploadTask
}
return nil
}
}

Related

How to download a file from a AWS S3 bucket in iOS?

In my iOS app, I try to download a file from an AWS S3 bucket. Here is what I tried:
I initialize AWSMobileClient:
import AWSMobileClient
import AWSS3
let configuration = AWSServiceConfiguration(
region: AWSRegionType.EUCentral1,
credentialsProvider: AWSMobileClient.default())
AWSServiceManager.default().defaultServiceConfiguration = configuration
AWSMobileClient.default().initialize { (userState: UserState?, error: Error?) in
if (userState != nil)
{
print("Initialize OK : \(userState.debugDescription)")
}
if (error != nil)
{
print("Initialize error: \(String(describing: error))")
}
}
I got:
"initialize OK : Optional(AWSMobileClient.UserState.guest)"
Now I try to download a file:
let expression = AWSS3TransferUtilityDownloadExpression()
expression.progressBlock = {(task, progress) in DispatchQueue.main.async(execute: {
print("Progress : \(progress)")
})
}
let completionHandler: AWSS3TransferUtilityDownloadCompletionHandlerBlock = {
(task: AWSS3TransferUtilityDownloadTask, url: URL?, data: Data?, error: Error?) -> Void in
DispatchQueue.main.async(execute: {
print("End download 1")
})
}
let fileManager = FileManager.default
let fileURL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent("100.ogg")
let transferUtility: AWSS3TransferUtility = AWSS3TransferUtility.default()
transferUtility.download(
to: fileURL,
bucket: "my-s3-bucket",
key: "100.ogg",
expression: expression,
completionHandler: completionHandler).continueWith { (task) -> AnyObject? in
if let error = task.error {
print("Error download : \(error)")
}
if let result = task.result {
print("Result : \(result.debugDescription)")
}
print("End download 2 : \(fileManager.fileExists(atPath: fileURL.absoluteString))")
return nil
}
I got:
"Result : <AWSS3TransferUtilityDownloadTask: 0x6000020bd4d0>
"End download 2 : false"
I don't get any progress, and I also don't get the "End download 1"
So basically, I dont get any error, but it does look like nothing has been downloaded. Also, on a side note, it works well with the Android version of my app, so it's very likely that there is an error in my code.
So what should I change to make it work?
Thanks.
My bad, the example above is actually working, But I had to:
change fileManager.fileExists(atPath: fileURL.absoluteString)) by fileManager.fileExists(atPath: fileURL.path))
check if file exists in the first completionHandler (where I wrote print("End download 1"))

Almofire multiple images Download and save them locally

I have more than 500 image links I want to download those images and store locally in app document directory when app starts. I am using Almofire for download but I am getting error like
"URLSession task was cancelled" and Request timeOut
func downloadAllImages(images:[String: String], retryCount: Int = 0,completion: #escaping((Bool)->Void)){
var success: Bool = true
var failedImages = [String: String]()
for (localName, severPath) in images {
self.dispatchGroup.enter()
let destination: DownloadRequest.Destination = { _, _ in
let documentsURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0]
let fileURL = documentsURL.appendingPathComponent(localName)
return (fileURL, [.removePreviousFile, .createIntermediateDirectories])
}
let path = severPath.replacingOccurrences(of: "\\", with: "/").addingPercentEncoding(withAllowedCharacters: .urlQueryAllowed) ?? ""
//AF.sessionConfiguration.httpShouldSetCookies = false
AF.download(path, to: destination).response { response in
switch response.result {
case .success(_):
break
case .failure(let error):
if response.response?.statusCode != 404 {
success = false
failedImages[localName] = path
print("Image Download Error = \(error.localizedDescription)")
}
break
}
debugPrint(response)
self.dispatchGroup.leave()
}
}
dispatchGroup.notify(queue: .main) {
//retry If some Images failed to download
if failedImages.isEmpty || retryCount >= self.maximumRetryCount {
completion(success)
}else {
self.downloadAllImages(images: failedImages, retryCount: retryCount + 1) { (success) in
completion(success)
}
}
}
}
images dictionary contains
localName as key
serverPath as value
AFImageDownloaders have limit of active downloads, and I believe changing maximumActiveDownloads or something similar in your API will fix that. The new downloads just cancel the previous ones. But it's better to download them in chunks.
For example this one is for ImageDownloader
public init(session: Session,
downloadPrioritization: DownloadPrioritization = .fifo,
maximumActiveDownloads: Int = 4,
imageCache: ImageRequestCache? = AutoPurgingImageCache()) {
precondition(!session.startRequestsImmediately, "Session must set `startRequestsImmediately` to `false`.")
self.session = session
self.downloadPrioritization = downloadPrioritization
self.maximumActiveDownloads = maximumActiveDownloads
self.imageCache = imageCache
}
UPD:
The limit is not on AF, but URLSession's. And one AF downloader uses one URLSession. You have to pass custom URLSessionConfigurations to handle more active downloads HTTPMaximumConnectionsPerHost. And pass it AF Session class

How to get download progress in the Google Drive API (Swift 5)

I use this function to download file from the Google Drive API and I want to get download progress. Maybe anybody knows how to do it?
func download(file: GTLRDrive_File) {
let url = "https://www.googleapis.com/drive/v3/files/\(file.identifier!)?alt=media"
let fetcher = drive.fetcherService.fetcher(withURLString: url)
fetcher.beginFetch(completionHandler: { data, error in
if let error = error {
print(error.localizedDescription)
}
//Here I save data to the Documents
})
}
I tried to get it from fetcher.receivedProgressBlock but it's always return nil
Solution. Actual for Swift 5:
func download(file: GTLRDrive_File) {
let fileSize = file.size?.doubleValue //You need to fetch file size in your request q.fields = "kind,nextPageToken,files(mimeType,id,name,size)"
let url = "https://www.googleapis.com/drive/v3/files/\(file.identifier!)?alt=media"
let fetcher = drive.fetcherService.fetcher(withURLString: url)
fetcher.beginFetch(completionHandler: { data, error in
if let error = error {
print(error.localizedDescription)
}
//Here I save data to the Documents
})
//Here you can get total bytes received (value is updated in real time)
fetcher.receivedProgressBlock = { _, totalBytesReceived in
guard let fileSize = fileSize else { return }
let progress = Double(totalBytesReceived) / fileSize
print(progress) //Here you can update UI (progress bar) or do something else
}
}
For some reason, file.size returns nil so I used fetcher.response?.expectedContentLength instead.
func download(file: GTLRDrive_File, service: GTLRDriveService) {
let url = "https://www.googleapis.com/drive/v3/files/\(file.identifier!)?alt=media"
let fetcher = service.fetcherService.fetcher(withURLString: url)
fetcher.beginFetch(completionHandler: { fileData, error in
if error == nil {
print("finished downloading file Data...")
print(fileData as Any)
// do anything with data here
} else {
print("Error: \(error?.localizedDescription)")
}
})
fetcher.receivedProgressBlock = { _, totalBytesReceived in
print("totalBytesReceived: \(totalBytesReceived)")
print("size: \(fetcher.response?.expectedContentLength)")
if let fileSize = fetcher.response?.expectedContentLength {
let progress: Double = Double(totalBytesReceived) / Double(fileSize)
print(progress)
// Update progress bar here
}
}
}

How to download A LOT of files from S3 using the transfer utility?

I have several thousand images I want to download from a S3 bucket to an iOS App.
But I'm getting memory issues I'm unable to track down.
Here is my sketchy code:
let client = HttpClient<[SomeImage]>()
client.get(fromURL: URL(string: endpoint)!) {
(result, error) in
if let error = error {
self.log(message: "\(error)", level: .error)
return
}
if let result = result {
let downloadGroup = DispatchGroup()
var count = 0
// just assembling a list of s3 keys to download here...
for item in result {
for image in (item.images ?? []) {
let prefix = "\(image.key)/"
for key in ["\(globalGetThumbnailS3Key(byImageKey: image.key))",
"\(globalGetPreviewS3Key(byImageKey: image.key))"] {
count = count + 1
let completionHandler: AWSS3TransferUtilityDownloadCompletionHandlerBlock = {
(task, URL, data, error) in
if let error = error {
self.log(message: "\(error)", level: .error)
return
}
if let data = data, let localDir = FileManager.default.applicationSupportURL {
do {
let imageURL = localDir.appendingPathComponent(key)
FileManager.default.directoryExistsOrCreate(localDir.appendingPathComponent(prefix))
try data.write(to: imageURL)
self.log(message: "downloaded \(prefix)\(key) to \(imageURL.absoluteString)", level: .verbose)
} catch let error {
self.log(message: "\(error)", level: .error)
return
}
}
}
bgSyncQueue.async(group: downloadGroup) {
self.transferUtility.downloadData(fromBucket: "\(globalDerivedImagesBucket)", key: key,
expression: nil,
completionHandler: completionHandler).continueWith {
(task) in
if let error = task.error {
// iirc, this error is caused, if the task couldnt be created due to being offline
self.log(message: "\(error)", level: .error)
return nil
}
if let result = task.result {
// do something with the task?
return nil
}
return nil
}
}
}
}
}
self.log(message: "\(count) images to download...", level: .debug)
bgSyncQueue.activate()
downloadGroup.notify(queue: DispatchQueue.main) {
self.log(message: "All items downloaded?!")
}
}
}
}
So I put all calls to the transfer utility in a serial dispatch queue, which is initially inactive. Then I activate the queue and downloading starts just fine. But after a while the app crashes with "Message from debugger: Terminated due to memory issue."
The app is only consuming about 100M of memory though. What am I overlooking?
Rob's suggestion to use the "downloadToUrl" method was the way to go, without using GCD on my part. Thanks again, Rob!
The transferUtility seems to be a fine tool, though very badly documented.
Here is the simple code used to download about 20k of images:
for key in keys {
let imageURL = localDir.appendingPathComponent(key.1)
let completionHandler: AWSS3TransferUtilityDownloadCompletionHandlerBlock = {
(task, URL, data, error) in
if let error = error {
self.log(message: "failed downloading \(key.1): \(error)", level: .error)
DispatchQueue.main.async {
countingDown()
}
return
}
DispatchQueue.main.async {
countingDown()
if let onProgress = self.onProgress {
onProgress(100.0 - ((100.0 / Double(total)) * Double(count)))
}
}
//self.log(message: "downloaded \(key.1)")
}
transferUtility.download(to: imageURL, bucket: "\(globalDerivedImagesBucket)", key: key.1, expression: nil, completionHandler: completionHandler).continueWith {
(task) in
if let error = error {
self.log(message: "\(error)", level: .error)
DispatchQueue.main.async {
countingDown()
}
return nil
}
return nil
}
}
You may need to consider using an autoreleasepool to better manage the memory used by the bridged data types as detailed here
Exert from article (in case of link changes)
Consider the code:
func run() {
guard let file = Bundle.main.path(forResource: "bigImage", ofType: "png") else {
return
}
for i in 0..<1000000 {
let url = URL(fileURLWithPath: file)
let imageData = try! Data(contentsOf: url)
}
}
Even though we’re in Swift, this will result in the same absurd memory spike shown in the Obj-C example! This is because the Data init is a bridge to the original Obj-C [NSDatadataWithContentsOfURL] -- which unfortunately still calls autorelease somewhere inside of it. Just like in Obj-C, you can solve this with the Swift version of #autoreleasepool; autoreleasepool without the #:
autoreleasepool {
let url = URL(fileURLWithPath: file)
let imageData = try! Data(contentsOf: url)
}
Disclaimer: I am no expert in Swift or Objective-C advanced memory management but I have used this in a similar scenario with good results.

AWS S3 iOS SDK: How to resume upload after connection is interrupted?

This is my code to accomplish the upload task:
let image = UIImage(named: "12.jpeg")
let fileManager = FileManager.default
let imageData = UIImageJPEGRepresentation(image!, 0.99)
let path = (NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true)[0] as NSString).appendingPathComponent("\(imageData!).jpeg")
fileManager.createFile(atPath: path as String, contents: imageData, attributes: nil)
let fileUrl = NSURL(fileURLWithPath: path)
uploadRequest?.bucket = "testrawdata"
uploadRequest?.key = "test/loodfd.jpeg"
uploadRequest?.contentType = "image/jpeg"
uploadRequest?.body = fileUrl as URL!
uploadRequest?.serverSideEncryption = AWSS3ServerSideEncryption.awsKms
uploadRequest?.uploadProgress = { (bytesSent, totalBytesSent, totalBytesExpectedToSend) -> Void in
DispatchQueue.main.async(execute: {
print("bytes sent \(bytesSent), total bytes sent \(totalBytesSent), of total \(totalBytesExpectedToSend)")
})
}
transferManager?.upload(uploadRequest).continue(with: AWSExecutor.mainThread(), withSuccessBlock: { (taskk: AWSTask) -> Any? in
if taskk.error != nil {
// Error.
} else {
// Do something with your result.
}
return nil
})
}
I know I don't need to apply it to image, but this is just an example, by default I'm going to send files like 100mb.
When I put my phone into airplane mode during the transfer then turn the network on again, it does not finish the upload task.
Docs are not saying explicitly what should I do to resume interrupted task.
Here is what I tried:
I put initialization of request and manager into viewDidLoad() to assure I'm not creating another request
class ViewController: UIViewController {
var uploadRequest:AWSS3TransferManagerUploadRequest!
var transferManager: AWSS3TransferManager!
override func viewDidLoad() {
super.viewDidLoad()
uploadRequest = AWSS3TransferManagerUploadRequest()
transferManager = AWSS3TransferManager.default()
}
and tried to call
func resumeTransfer() {
transferManager?.resumeAll(nil)
}
But it does not work.
Thanks in advance
It turns out that Transfer Utility is the right tool to accomplish this task
func uploadData(data: NSData) {
let expression = AWSS3TransferUtilityUploadExpression()
expression.progressBlock = progressBlock
let transferUtility = AWSS3TransferUtility.default()
transferUtility.uploadData(
data as Data,
bucket: "test",
key: "test/test.jpeg",
contentType: "image/jpeg",
expression: expression,
completionHander: completionHandler).continue(successBlock: { (task) -> AnyObject! in
if let error = task.error {
NSLog("Error: %#",error.localizedDescription);
}
if let exception = task.exception {
NSLog("Exception: %#",exception.description);
}
if let _ = task.result {
NSLog("Upload Starting!")
// Do something with uploadTask.
}
return nil;
})
}
This way all upload stuff happens in the background, I don't have to worry about app being killed by the iOS, about networks problem etc.
One can even specify
configuration?.allowsCellularAccess = false
in AWSServiceConfiguration
to resume the task only when wifi is available.

Resources