iOS Rich notifications: Sending MP4 files result in transparent thumbnails - ios

Question

I just added MP4/GIF attachments to my push notifications in my iOS app. Everything works fine with respect to playback. The issue I am facing is when MP4 videos are sent, the small thumbnail that is sent in the push looks transparent. However, when I expand it, it looks perfect and I can play it well too inside the push. When I send the same video converted to GIF the thumbnail also looks perfect.
Here is an example:
The example above shows two different apps, just to show how MP4 and GIF thumbnails show up for the same event. If I were to send GIF to the app on the top, the output of the thumbnail looks exactly like the Pushover app thumbnail.
And here is what happens when I slide and view the thumbnail (transparent one). This particular expanded thumbnail is for a different event (I lost that old event). But the point I wanted to make is the expanded view looks perfect. And plays perfectly too.
So in conclusion, in IOS, when I send MP4 files as attachments the small thumbnail looks transparent, but plays back well. expanded thumbnail looks perfect.
This is my client code:
//
// NotificationService.m
// NotificationService
//
//
//
//
// Credit https://github.com/Leanplum/Leanplum-iOS-Samples/blob/master/iOS_basicSetup/basicSetup/richPushExtension/NotificationService.m
#import "NotificationService.h"
#interface NotificationService ()
#property (nonatomic, strong) void (^contentHandler)(UNNotificationContent *contentToDeliver);
#property (nonatomic, strong) UNMutableNotificationContent *bestAttemptContent;
#end
#implementation NotificationService
- (void)didReceiveNotificationRequest:(UNNotificationRequest *)request withContentHandler:(void (^)(UNNotificationContent * _Nonnull))contentHandler {
self.contentHandler = contentHandler;
self.bestAttemptContent = [request.content mutableCopy];
NSDictionary *userInfo = request.content.userInfo;
// If there is no image in the payload than
// the code will still show the push notification.
if (userInfo == nil || userInfo[#"image_url_jpg"] == nil) {
NSLog(#"zmNinja Notification: Did not get a payload or image");
[self contentComplete];
return;
}
NSString *mediaUrl = userInfo[#"image_url_jpg"];
// if (mediaType == nil) {
// NSLog(#"zmNinja Notification: No media type specified, assuming .jpg");
// mediaType = #".jpg";
// }
// load the attachment
[self loadAttachmentForUrlString:mediaUrl
completionHandler:^(UNNotificationAttachment *attachment) {
if (attachment) {
self.bestAttemptContent.attachments = [NSArray arrayWithObject:attachment];
}
[self contentComplete];
}];
}
- (NSString*)determineType:(NSString *) fileType {
// Determines the file type of the attachment to append to NSURL.
//return #".gif";
// Determines the file type of the attachment to append to NSURL.
NSLog (#"zmNinja Notification: determineType got filetype=%#",fileType);
if ([fileType isEqualToString:#"image/jpeg"]){
NSLog (#"zmNinja Notification: returning JPG");
return #".jpg";
}
if ([fileType isEqualToString:#"video/mp4"]){
NSLog (#"zmNinja Notification: returning MP4");
return #".mp4";
}
if ([fileType isEqualToString:#"image/gif"]) {
NSLog (#"zmNinja Notification: returning GIF");
return #".gif";
}
if ([fileType isEqualToString:#"image/png"]) {
NSLog (#"zmNinja Notification: returning PNG");
return #".png";
}
NSLog (#"zmNinja Notification: unrecognized filetype, returning JPG");
return #".jpg";
}
- (void)loadAttachmentForUrlString:(NSString *)urlString
completionHandler:(void(^)(UNNotificationAttachment *))completionHandler {
__block UNNotificationAttachment *attachment = nil;
NSURL *attachmentURL = [NSURL URLWithString:urlString];
NSURLSession *session = [NSURLSession sessionWithConfiguration:[NSURLSessionConfiguration defaultSessionConfiguration]];
[[session downloadTaskWithURL:attachmentURL
completionHandler:^(NSURL *temporaryFileLocation, NSURLResponse *response, NSError *error) {
if (error != nil) {
NSLog(#"unable to add attachment: %#", error.localizedDescription);
} else {
NSString *fileType = [self determineType: [response MIMEType]];
NSFileManager *fileManager = [NSFileManager defaultManager];
NSURL *localURL = [NSURL fileURLWithPath:[temporaryFileLocation.path stringByAppendingString:fileType]];
[fileManager moveItemAtURL:temporaryFileLocation toURL:localURL error:&error];
NSError *attachmentError = nil;
attachment = [UNNotificationAttachment attachmentWithIdentifier:#"" URL:localURL options:nil error:&attachmentError];
if (attachmentError) {
NSLog(#"unable to add attchment: %#", attachmentError.localizedDescription);
}
}
completionHandler(attachment);
}] resume];
}
- (void)contentComplete {
self.contentHandler(self.bestAttemptContent);
}
- (void)serviceExtensionTimeWillExpire {
// Called just before the extension will be terminated by the system.
// Use this as an opportunity to deliver your "best attempt" at modified content, otherwise the original push payload will be used.
NSLog (#"zmNinja Notification: Time about to expire, handing off to best attempt");
self.contentHandler(self.bestAttemptContent);
}
#end
The server side uses FCM legacy APIs:
my $ios_message = {
to => $obj->{token},
notification => {
title => $title,
body => $body,
sound => "default",
badge => $badge,
},
data => {
myMessageId => $notId,
mid => $mid,
eid => $eid,
summaryText => $eid
}
};
$ios_message->{data}->{image_url_jpg} = $pic; # $pic is a URL for the mp4
# image_url_jpg is just a field name. It was originally meant for static images
# haven't changed it yet, as you see in client code above, it uses that field.
$json = encode_json($ios_message);
my $req = HTTP::Request->new( 'POST', $uri );
$req->header(
'Content-Type' => 'application/json',
'Authorization' => $key
);
$req->content($json);
my $lwp = LWP::UserAgent->new(%ssl_push_opts);
my $res = $lwp->request($req);
Finally, if you want to take a look at a sample MP4 to rule out any format issue, here is one that I've uploaded to google drive (link). I've extracted frame information with ffshow and it doesn't look out of place to me (plus it plays perfectly).
Can someone help me understand why the initial small thumbnail looks messed up in iOS? (If it helps, I am on iOS 13.x)
Thanks.

Answer 1

Had the same problem. Solved it by creating a notification content extension and sending a thumbnail image URL along with the video URL.
In the service extension, I add both the video and the thumbnail as attachments, with the thumbnail as the first element, which iOS will show in the notification preview:
// mediaAttachment and thumbnailAttachment are UNNotificationAttachments
// that have just been downloaded
if let mediaAttachment = mediaAttachment {
mutableContent.attachments = [mediaAttachment]
}
if let thumbnailAttachment = thumbnailAttachment {
mutableContent.attachments.insert(thumbnailAttachment, at: 0)
}
contentHandler(mutableContent)
The expanded notification UI is handled by the content extension which replaces the default UI, and there I ignore the thumbnail image and just show the video, which will be the last attachment.