I want to prioritise Frame Detail over Frame Rate.
There is a parameter named contentHint in video media stream for it, but it may not be able to set by twilio sdk
https://www.w3.org/TR/mst-content-hint/
detail
The track should be treated as if video details are extra important.
This is generally applicable to presentations or web pages with text content,
painting or line art.
This setting would normally optimize for detail in the resulting individual frames
rather than smooth playback. Artefacts from quantization or downscaling
that make small text or line art unintelligible should be avoided.
https://github.com/twilio/twilio-video-ios/issues/59
In this github issue, I found that isScreencast is for it.
however Network Bandwidth Profile should be the official way to achieve it.
But as far as I tried, network bandwidth profile cannot achieve "Frame Detail over Frame Rate". What is the best way?
Twilio developer evangelist here.
the contentHint attribute is set on a MediaStreamTrack so you could set this in a Twilio Video application. For example:
const stream = await navigator.mediaDevices.getUserMedia({ video: true, audio: true });
const videoTrack = stream.getVideoTracks()[0];
const audioTrack = stream.getAudioTracks()[0];
videoTrack.contentHint = "detail";
const room = await Twilio.Video.connect(token, { tracks: [videoTrack, audioTrack] });
If you set the contentHint on the video track and then pass the tracks directly to the connect method, they will be used.
If you want to set this directly via the Twilio Video SDK, I would recommend raising a ticket in the Video.js repo.
Related
I am using Google cloud speech through Python and finding many transcriptions are inaccurate and missing several words. This is a simple script I'm using to return a transcript of an audio file, in this case 'out307.wav':
client = speech.SpeechClient()
with io.open('out307.wav', 'rb') as audio_file:
content = audio_file.read()
audio = speech.types.RecognitionAudio(content=content)
config = speech.types.RecognitionConfig(
enable_word_time_offsets=True,
language_code='en-US',
audio_channel_count=1)
response = client.recognize(config, audio)
for result in response.results:
alternative = result.alternatives[0]
print(u'Transcript: {}'.format(alternative.transcript))
This returns the following transcript:
to do this the tensions and suspicions except
This is very far off what the actual audio says (I've uploaded it at https://vocaroo.com/i/s1zdZ0SOH1Ki). The audio is a .wav and very clear with no background noise. This is worse than average, as in some cases it will get the transcription fully correct on a 10 second audio file, or it may miss just a couple of words. Is there anything I can do to improve results?
This is weird, I tried your audio file with your code and I get the same result, but, if I change the language_code to "en-UK" I am able to get the full response.
I'm working for Google Cloud and I created for you a public issue here, you can track there the updates.
I'm working on a video app, we are changing form regular mp4 files to HLS, one of the many reasons we have to do the change is that we hace much more control over the bandwidth usage of videos (we load lots of other stuff in our player, so we need to optimize the experience the best way).
So, AVFoundation introduced in iOS10 the ability to control the bandwidth using:
AVPlayerItem *playerItem = [AVPlayerItem playerItemWithAsset:self.urlAsset];
playerItem.preferredForwardBufferDuration = 30.0;
playerItem.preferredPeakBitRate = 200000.0; // Remember this line
There's also a configuration introduced on iOS11 to set the maximum resolution of the item with preferredMaximumResolution, So we're using it, but we still need a solution for iOS10 devices.
Well, now we have control over the preferredPeakBitRate that's nice, but we have a problem, not all the HLS sources are generated by us, so, let's say we want to set a maximum resolution of 480p when you're not connected to a wifi network, today I don't have way to achieve that, not always I'm going to be able to know how much bandwidth needs the 480p source for the selected HLS playlist.
One thing I was thinking about is to read the information inside the m3u8 file, to at least know which are the different quality sources that my player can show and how much bandwidth needs everyone.
One way to do this, would download the m3u8 playlist as a plain text, use a regex to read the file and process this data, well, I'm trying to avoid that, I think that this should far less difficult.
I cannot read this information from the tracks, because a) I can't find the information, b) the tracks are replaced dynamically when changing the quality, yeah 1 track for every quality level.
So, I don't know how I can get this information, I've searched google, stackoverflow and I can't find this information, does any one can help me?
Here's an example for what I want to do, I have this example playlist:
#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=314000,RESOLUTION=228x128,CODECS="mp4a.40.2"
test-hls-1-16a709300abeb08713a5cada91ab864e_hls_duplex_192k.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=478000,RESOLUTION=400x224,CODECS="avc1.42001e,mp4a.40.2"
test-hls-1-16a709300abeb08713a5cada91ab864e_hls_duplex_400k.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=691000,RESOLUTION=480x270,CODECS="avc1.42001e,mp4a.40.2"
test-hls-1-16a709300abeb08713a5cada91ab864e_hls_duplex_600k.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1120000,RESOLUTION=640x360,CODECS="avc1.4d001f,mp4a.40.2"
test-hls-1-16a709300abeb08713a5cada91ab864e_hls_duplex_1000k.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1661000,RESOLUTION=960x540,CODECS="avc1.4d001f,mp4a.40.2"
test-hls-1-16a709300abeb08713a5cada91ab864e_hls_duplex_1500k.m3u8
And I just want to have that information available on an array inside my code, something like this:
NSArray<ZZMetadata *> *metadataArray = self.urlAsset.bandwidthMetadata;
NSLog(#"Metadata info: %#", metadataArray);
And print something like this:
<__NSArrayM 0x123456789> (
<ZZMetadata 0x234567890> {
trackId: 1
neededBandwidth: 314000
resolution: 228x128
codecs: ...
...
}
<ZZMetadata 0x345678901> {
trackId: 2
neededBandwidth: 478000
resolution: 400x224
}
...
}
Given an HLS manifest with multiple variants/renditions:
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1612430,CODECS="avc1.4d0020,mp4a.40.5",RESOLUTION=640x360
a.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=3541136,CODECS="avc1.4d0020,mp4a.40.5",RESOLUTION=960x540
b.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=5086455,CODECS="avc1.640029,mp4a.40.5",RESOLUTION=1280x720
c.m3u8
Is it possible to get an array of the three variants (with the attributes such as bandwidth and resolution) from either the AVAsset or AVPlayerItem?
I am able to get the currently playing AVPlayerItemTrack by using KVO on the AVPlayerItem, but again, it's only the track that's actively being played not the full list of variants.
I'm interested in knowing if the asset is being played at it's highest possible quality, so that I can make a decision on whether the user has enough bandwidth to start a simultaneous secondary video stream.
To know which variant you are currently playing, you can keep a KVO on AVPlayerItemNewAccessLogEntryNotification and by looking at AVPlayerItemAcessLogEvent in the access log, you can tell current bitrate and any change in bitrate.
AVPlayerItemAccessLog *accessLog = [((AVPlayerItem *)notif.object) accessLog];
AVPlayerItemAccessLogEvent *lastEvent = accessLog.events.lastObject;
if( lastEvent.indicatedBitrate != self.previousBitrate )
{
self.bitrate = lastEvent.indicatedBitrate
}
As far as knowing the entire list of available bitrates, you can simply make a GET request for the master m3u8 playlist and parse it. You will only need to do it once so not much of an overhead.
New in iOS 15, there’s AVAssetVariant
My understanding currently is that:
CameraUI
I can use the CameraUI to access the built in camera for MediaType.VIDEO and that delegates to the built-in video camera app and lets me record a video. My app does that now.
When I stop recording and click the "Use" button, I am returned to my app and theoretically I have a valid MediaPromise.
iOS does -not- provide a valid/usable url/filename to the recorded video (or to photos) and so I would have to use a Loader to bring-in/use/access the 'recorded' video... AND... iOS does not actually create a file anywhere on the device, most importantly, in the Camera Roll where one would expect by the normal behavior when uses the system native camera/video app.
The documentation says that the Loader can load various image types and SWFs but nothing about video data, so I conclude from that that I cannot actually use the CameraUI to generate a valid MediaPromise that I can then pass to a Loader class or similar to read in the information created by the system camera and then manipulate (upload, save to applicationStorageDirectory, and/or display in one of the two video player components available in the API).
CameraRoll
I can have video entities in the iOS Camera Roll but the AS3/Air3.5 CameraRoll class won't let me view/access/reference them in any way.
Normal File I/O
All my attempts to use the Air3.5 File classes to browse to the storage location of the iOS Camera Roll have been rebuffed.
------- Questions -------
Am I correct in believing that there is a way to take video but no way to use the video that's been captured. (No way to use the resulting MediaPromise successfully).
I believe you can take video and access it using Android, but there's nothing in the documentation that says that you cannot using iOS.
Am I correct in believing that iOS sandboxes apps so that they cannot browse to video/photo storage using standard File I/O, but only through the apparently non-workable means I've tried (CameraUI & CameraRoll)
Am I wrong to think that these should be rather obvious NEEDS that one can achieve using the XCode Objective C++ etc route but the AIR Mobile Framework does not allow either because of Apple blocking functionality or because Adobe has failed to meet reasonable expectations?
One item of ironic note to convey. If I use the iOS system camera app to record a video, a thumnail of that video then appears in the Gallery/Camera Roll, and of course, I can share it or view it, or whatever... If I use AIR's CameraRoll.browseForImage(), provided I haven't used the camera to take another image, when it shows me the folder where the pictures are stored, the folder icon uses the thumbnail of the last object added... in this case, the video I took, but if I then enter the folder, the video cannot be found. It's teasing us. It knows it's there, but it is apparently forbidden fruit.
I can't answer all your questions, so this entry may not be acceptable, but I found this page while searching a solution for some the problems you described and thought that someone else may find this answer (partially) useful.
To save the movie you just took you need to open and read the data from the promise.
The iOS won't save the file anywere, so the MediaPromise.file is always null.
This is my solution to the problem:
private var camera:CameraUI;
private var dataInput:IDataInput;
public function recordVideo():void
{
// Start the camera and ask for a video
camera = new CameraUI();
camera.addEventListener(MediaEvent.COMPLETE, onCameraComplete);
camera.launch(MediaType.VIDEO);
}
private function onCameraComplete(event:MediaEvent):void
{
// event.data is a MediaPromise and MediaPromise.open() returns a IDataInput
// Let's cast it to a dispatcher and check when it's complete
dataInput = event.data.open();
var dispatcher:IEventDispatcher = IEventDispatcher(dataInput);
dispatcher.addEventListener(Event.COMPLETE, onDataInputComplete);
}
private function onDataInputComplete(event:Event):void
{
// We can do whatever we want with the data, so we'll store it in a File
var file:File = new File();
var bytes:ByteArray = new ByteArray();
var stream:FileStream = new FileStream();
// Reading the data from the opened MediaPromise
dataInput.readBytes(bytes);
stream.open(file, FileMode.WRITE);
stream.writeBytes(bytes, 0, bytes.bytesAvailable);
stream.close();
}
Also, I'm still looking for a way to put the movie in the CameraRoll
Can anyone suggest how to handle a slow network when streaming video in a web view?
When the network strength is poor, a blank screen appears or video doesn't stream.
Is there a way to detect this condition so that we can alert the user? (Apart from using private API.)
Perhaps ifi_baudrate member of the if_data structure (declared in <net/if.h>) is what you need. If baudrate is less than some threshold value, then you can show an alert.
Please see the following answer to know how to obtain the if_data structure for a particular network interface:
https://stackoverflow.com/a/8014012/1310204
You can easily detect the state of the network connection via the HTML5 networking API
http://www.html5rocks.com/en/mobile/optimization-and-performance/#toc-network-detection
Also if you want to test the network speed, just set up some files on your server of a specific size, and do a ajax request for the file, while timing how long it takes to download.
You can use a simple:
var start = new Date();
$.get("someFile.jpg")
.done(function() {
var elapsed = (new Date() - start);
});
Or dig into the HTML5 performance API:
http://www.html5rocks.com/en/tutorials/webperformance/basics/
...if you not using javascript, the same applies. Just open a network connection with whatever is at your disposition, download a small file & do the math ;-)