I've implemented Apple's SpeechRecognizer to convert speech to text. I have multiple audio recordings so I'm creating mulitple SFSpeechRecognizer instance so that all of those are converted parallely and I've also used DispatchGroup so that I can get completion at last one's end. But I'm keep getting error kAFAssistantErrorDomain error 209.
private var dispatchGroup = DispatchGroup()
allURLs.forEach { (singleURL) in
DispatchQueue.main.async {
thisSelf.dispatchGroup.enter()
let request = SFSpeechURLRecognitionRequest(url: url)
guard let recognizer = SFSpeechRecognizer() else {
thisSelf.dispatchGroup.leave()
completion(.failure(thisSelf.speechReconInitError))
return
}
request.shouldReportPartialResults = false
if !recognizer.isAvailable {
thisSelf.dispatchGroup.leave()
return
}
recognizer.recognitionTask(with: request) { [weak thisSelf] (result, error) in
guard let reconSelf = thisSelf else { return }
if let error = error {
completion(.failure(error))
if let nsError = error as NSError? {
print("Error while transcripting audio: \(url.path), Code, Domain, Description: \(nsError.code), \(nsError.domain), \(nsError.localizedDescription)")
} else {
print("Error while transcripting audio: \(url.path), Error: \(error.localizedDescription)")
}
reconSelf.dispatchGroup.leave()
} else if let transcriptionResult = result, transcriptionResult.isFinal {
transcribedText += transcriptionResult.bestTranscription.formattedString
reconSelf.dispatchGroup.leave()
}
}
thisSelf.dispatchGroup.notify(queue: .main) {
if !transcribedText.isEmpty {
completion(transcribedText)
}
}
}
}
And If I transcribe only one audio to text at one time then I don't get any error.
TIA
Related
IOS 15 Swift 5.
I want to fully transcribe recorded audio from a file
func transcribeAudio(url: URL) {
let recognizer = SFSpeechRecognizer()
let request = SFSpeechURLRecognitionRequest(url: url)
request.requiresOnDeviceRecognition = true;
request.taskHint = .dictation;
recognizer?.recognitionTask(with: request) { [unowned self] (result, error) in
guard let result = result else {
print("There was an error: \(error!)")
return
}
if result.isFinal {
print(result.bestTranscription.formattedString)
}
}
}
Console output first phrase instead of full text
How can I rewrite a code to transcribe the whole audio file?
Thanks!
The solution was to
Put request and recognizer to class props
Exclude partial results
Print all results, not final one
func transcribeAudio(url: URL) {
request = SFSpeechURLRecognitionRequest(url: url)
request?.requiresOnDeviceRecognition = true;
request?.shouldReportPartialResults = false;
recognizer?.supportsOnDeviceRecognition = true;
recognizer?.recognitionTask(with: request!) { [unowned self] (result, error) in
guard let result = result else {
print("There was an error: \(error!)")
return
}
RNEventEmitter.emitter.sendEvent(withName: "OnRecognition", body: result.bestTranscription.formattedString)
print(result.bestTranscription.formattedString)
}
}
UPDATED WITH PROPOSED SOLUTION AND ADDITIONAL QUESTION
I'm officially stuck and also in callback hell. I have a call to Firebase retrieving all articles in the FireStore. Inside each article object is a an Image filename that translates into a storage reference location that needs to be passed to a function to get the absolute URL back. I'd store the URL in the data, but it could change. The problem is the ArticleListener function is prematurely returning the closure (returnArray) without all the data and I can't figure out what I'm missing. This was working fine before I added the self.getURL code, but now it's returning the array back empty and then doing all the work.
If anyone has some bonus tips here on chaining the methods together without resorting to PromiseKit or GCD that would be great, but open to all suggestions to get this to work as is
and/or refactoring for more efficiency / readability!
Proposed Solution with GCD and updated example
This is calling the Author init after the Article is being created. I am trying to transform the dataDict dictionary so it get's used during the Author init for key ["author"]. I think I'm close, but not 100% sure if my GCD enter/leave calls are happening in the right order
public func SetupArticleListener(completion: #escaping ([Article]) -> Void) {
var returnArray = [Article]()
let db = FIRdb.articles.reference()
let listener = db.addSnapshotListener() { (querySnapshot, error) in
returnArray = [] // nil this out every time
if let error = error {
print("Error in setting up snapshot listener - \(error)")
} else {
let fireStoreDispatchGrp = DispatchGroup() /// 1
querySnapshot?.documents.forEach {
var dataDict = $0.data() //mutable copy of the dictionary data
let id = $0.documentID
//NEW EXAMPLE WITH ADDITIONAL TASK HERE
if let author = $0.data()["author"] as? DocumentReference {
author.getDocument() {(authorSnapshot, error) in
fireStoreDispatchGrp.enter() //1
if let error = error {
print("Error getting Author from snapshot inside Article getDocumentFunction - leaving dispatch group and returning early")
fireStoreDispatchGrp.leave()
return
}
if let newAuthor = authorSnapshot.flatMap(Author.init) {
print("Able to build new author \(newAuthor)")
dataDict["author"] = newAuthor
dataDict["authorId"] = authorSnapshot?.documentID
print("Data Dict successfully mutated \(dataDict)")
}
fireStoreDispatchGrp.leave() //2
}
}
///END OF NEW EXAMPLE
if let imageURL = $0.data()["image"] as? String {
let reference = FIRStorage.articles.referenceForFile(filename: imageURL)
fireStoreDispatchGrp.enter() /// 2
self.getURL(reference: reference){ result in
switch result {
case .success(let url) :
dataDict["image"] = url.absoluteString
case .failure(let error):
print("Error getting URL for author: \n Error: \(error) \n forReference: \(reference) \n forArticleID: \(id)")
}
if let newArticle = Article(id: id, dictionary: dataDict) {
returnArray.append(newArticle)
}
fireStoreDispatchGrp.leave() ///3
}
}
}
//Completion block
print("Exiting dispatchGroup all data should be setup correctly")
fireStoreDispatchGrp.notify(queue: .main) { ///4
completion(returnArray)
}
}
}
updateListeners(for: listener)
}
Original Code
Calling Setup Code
self.manager.SetupArticleListener() { [weak self] articles in
print("๐๐๐๐๐๐๐In closure function to update articles๐๐๐๐๐๐๐")
self?.articles = articles
}
Article Listener
public func SetupArticleListener(completion: #escaping ([Article]) -> Void) {
var returnArray = [Article]()
let db = FIRdb.articles.reference()
let listener = db.addSnapshotListener() { (querySnapshot, error) in
returnArray = [] // nil this out every time
if let error = error {
printLog("Error retrieving documents while adding snapshotlistener, Error: \(error.localizedDescription)")
} else {
querySnapshot?.documents.forEach {
var dataDict = $0.data() //mutable copy of the dictionary data
let id = $0.documentID
if let imageURL = $0.data()["image"] as? String {
let reference = FIRStorage.articles.referenceForFile(filename: imageURL)
self.getURL(reference: reference){ result in
switch result {
case .success(let url) :
print("Success in getting url from reference \(url)")
dataDict["image"] = url.absoluteString
print("Dictionary XFORM")
case .failure(let error):
print("Error retrieving URL from reference \(error)")
}
if let newArticle = Article(id: id, dictionary: dataDict) {
printLog("Success in creating Article with xformed url")
returnArray.append(newArticle)
}
}
}
}
print("๐๐๐๐๐๐๐ sending back completion array \(returnArray)๐๐๐๐๐๐๐")
completion(returnArray)
}
}
updateListeners(for: listener)
}
GetURL
private func getURL(reference: StorageReference, _ result: #escaping (Result<URL, Error>) -> Void) {
reference.downloadURL() { (url, error) in
if let url = url {
result(.success(url))
} else {
if let error = error {
print("error")
result(.failure(error))
}
}
}
}
You need dispatch group as the for loop contains multiple asynchronous calls
public func SetupArticleListener(completion: #escaping ([Article]) -> Void) {
var returnArray = [Article]()
let db = FIRdb.articles.reference()
let listener = db.addSnapshotListener() { (querySnapshot, error) in
returnArray = [] // nil this out every time
if let error = error {
printLog("Error retrieving documents while adding snapshotlistener, Error: \(error.localizedDescription)")
} else {
let g = DispatchGroup() /// 1
querySnapshot?.documents.forEach {
var dataDict = $0.data() //mutable copy of the dictionary data
let id = $0.documentID
if let imageURL = $0.data()["image"] as? String {
let reference = FIRStorage.articles.referenceForFile(filename: imageURL)
g.enter() /// 2
self.getURL(reference: reference){ result in
switch result {
case .success(let url) :
print("Success in getting url from reference \(url)")
dataDict["image"] = url.absoluteString
print("Dictionary XFORM")
case .failure(let error):
print("Error retrieving URL from reference \(error)")
}
if let newArticle = Article(id: id, dictionary: dataDict) {
printLog("Success in creating Article with xformed url")
returnArray.append(newArticle)
}
g.leave() /// 3
}
}
}
g.notify(queue:.main) { /// 4
print("๐๐๐๐๐๐๐ sending back completion array \(returnArray)๐๐๐๐๐๐๐")
completion(returnArray)
}
}
}
updateListeners(for: listener)
}
Use Case
I'm using iOS 11 Replaykit framework to try to record frames from the screen, and audio from both the app and the microphone.
Problem
Randomly, when I call my .append(sampleBuffer) get AVAssetWriterStatus.failed with the AssetWriter.Error showing
Error Domain=AVFoundationErrorDomain Code=-11823 "Cannot Save" UserInfo={NSLocalizedRecoverySuggestion=Try saving again., NSLocalizedDescription=Cannot Save, NSUnderlyingError=0x1c044c360 {Error Domain=NSOSStatusErrorDomain Code=-12412 "(null)"}}
Side issue: I play a repeating sound when the app is recording to try to verify the audio is recorded, but the sound stops when I start recording, even where I the video and external audio mic is working.
If you require more info, I can upload the other code to GitHub too.
Ideas
Since sometimes the recording saves (I can export to Photos app and replay the video) I think it must be async issues where I'm loading things out of order. Please let me know if you see any!
One I idea I will be trying is saving to my own folder in /Documents instead of directly to /Documents in case of weird permissions errors. Although I believe this would be causing consistent errors, instead of only sometimes breaking.
My Code
func startRecording() {
guard let firstDocumentDirectoryPath = NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true).first else { return }
let directoryContents = try! FileManager.default.contentsOfDirectory(at: URL(fileURLWithPath: firstDocumentDirectoryPath), includingPropertiesForKeys: nil, options: [])
print(directoryContents)
videoURL = URL(fileURLWithPath: firstDocumentDirectoryPath.appending("/\(arc4random()).mp4"))
print(videoURL.absoluteString)
assetWriter = try! AVAssetWriter(url: videoURL, fileType: AVFileType.mp4)
let compressionProperties:[String:Any] = [...]
let videoSettings:[String:Any] = [...]
let audioSettings:[String:Any] = [...]
videoInput = AVAssetWriterInput(mediaType: .video, outputSettings: videoSettings)
audioMicInput = AVAssetWriterInput(mediaType: .audio, outputSettings: audioSettings)
audioAppInput = AVAssetWriterInput(mediaType: .audio, outputSettings: audioSettings)
guard let assetWriter = assetWriter else { return }
guard let videoInput = videoInput else { return }
guard let audioAppInput = audioAppInput else { return }
guard let audioMicInput = audioMicInput else { return }
videoInput.mediaTimeScale = 60
videoInput.expectsMediaDataInRealTime = true
audioMicInput.expectsMediaDataInRealTime = true
audioAppInput.expectsMediaDataInRealTime = true
if assetWriter.canAdd(videoInput) {
assetWriter.add(videoInput)
}
if assetWriter.canAdd(audioAppInput) {
assetWriter.add(audioAppInput)
}
if assetWriter.canAdd(audioMicInput) {
assetWriter.add(audioMicInput)
}
assetWriter.movieTimeScale = 60
RPScreenRecorder.shared().startCapture(handler: recordingHandler(sampleBuffer:sampleBufferType:error:)) { (error:Error?) in
if error != nil {
print("RPScreenRecorder.shared().startCapture: \(error.debugDescription)")
} else {
print("start capture complete")
}
}
}
func recordingHandler (sampleBuffer:CMSampleBuffer, sampleBufferType:RPSampleBufferType, error:Error?){
if error != nil {
print("recordingHandler: \(error.debugDescription)")
}
if CMSampleBufferDataIsReady(sampleBuffer) {
guard let assetWriter = assetWriter else { return }
guard let videoInput = videoInput else { return }
guard let audioAppInput = audioAppInput else { return }
guard let audioMicInput = audioMicInput else { return }
if assetWriter.status == AVAssetWriterStatus.unknown {
print("AVAssetWriterStatus.unknown")
if !assetWriter.startWriting() {
return
}
assetWriter.startSession(atSourceTime: CMSampleBufferGetPresentationTimeStamp(sampleBuffer))
}
if assetWriter.status == AVAssetWriterStatus.failed {
print("AVAssetWriterStatus.failed")
print("assetWriter.error: \(assetWriter.error.debugDescription)")
return
}
if sampleBufferType == RPSampleBufferType.video {
if videoInput.isReadyForMoreMediaData {
print("=appending video data")
videoInput.append(sampleBuffer)
}
}
if sampleBufferType == RPSampleBufferType.audioApp {
if audioAppInput.isReadyForMoreMediaData {
print("==appending app audio data")
audioAppInput.append(sampleBuffer)
}
}
if sampleBufferType == RPSampleBufferType.audioMic {
if audioMicInput.isReadyForMoreMediaData {
print("===appending mic audio data")
audioMicInput.append(sampleBuffer)
}
}
}
}
func stopRecording() {
RPScreenRecorder.shared().stopCapture { (error) in
guard let assetWriter = self.assetWriter else { return }
guard let videoInput = self.videoInput else { return }
guard let audioAppInput = self.audioAppInput else { return }
guard let audioMicInput = self.audioMicInput else { return }
if error != nil {
print("recordingHandler: \(error.debugDescription)")
} else {
videoInput.markAsFinished()
audioMicInput.markAsFinished()
audioAppInput.markAsFinished()
assetWriter.finishWriting(completionHandler: {
print(self.videoURL)
self.saveToCameraRoll(URL: self.videoURL)
})
}
}
}
I got it to work. I believe it was indeed an async issue. The problem, for some reason is you must make sure
assetWriter.startWriting()
assetWriter.startSession(atSourceTime: CMSampleBufferGetPresentationTimeStamp(sampleBuffer))
happen strictly serially.
Change your code from this:
if assetWriter.status == AVAssetWriterStatus.unknown {
print("AVAssetWriterStatus.unknown")
if !assetWriter.startWriting() {
return
}
assetWriter.startSession(atSourceTime: CMSampleBufferGetPresentationTimeStamp(sampleBuffer))
}
to this:
DispatchQueue.main.async { [weak self] in
if self?.assetWriter.status == AVAssetWriterStatus.unknown {
print("AVAssetWriterStatus.unknown")
if !self?.assetWriter.startWriting() {
return
}
self?.assetWriter.startSession(atSourceTime: CMSampleBufferGetPresentationTimeStamp(sampleBuffer))
}
}
Or even better, the whole block inside CMSampleBufferDataIsReady ie.
if CMSampleBufferDataIsReady(sampleBuffer) {
DispatchQueue.main.async { [weak self] in
...
...
}
}
Let me know if it works!
I had similar issue. I fixed it by first check whether videoURL file is already existed. If so, remove it first then the error will go away.
In one of my view controllers I have a function to establish a tcp connection , send and receive messages which is as follows.
func tcpConnection(host: NSString){
var client = TCPClient(addr: String(host) as String, port: Config.tcpPort)
let (success, err) = client.connect(timeout: Config.tcpConnectionTimeout)
log?.debug("Connected \(success)")
guard success else { log?.error("Cannot Connect \(err)"); return}
let (successMsg, failmsg) = client.send(str: self.jsonString)
let data = client.read(Config.expectedByteLength)
guard let d = data else { return }
guard let recievedMsg = String(bytes: d, encoding: NSUTF8StringEncoding) else { return }
self.recivedMsgFromServer = recievedMsg
log?.debug("Recieved msg\(recievedMsg)")
let (clientClosed, errormsg) = client.close()
guard clientClosed else { return }
}
I am using this piece of code twice in the same view controller. So I want to have a generic class for this function . Also I have many guards which I want to replace with a single try catch block.
Also after receiving a message I am doing different things in both the tcp connection functions.
Following is what I have tried until now.
class TcpService{
var jsonString : String = ""
func tcpConnection(host: NSString){
do {
var client = try TCPClient(addr: String(host) as String, port: Config.tcpPort)
let (success, err) = client.connect(timeout: Config.tcpConnectionTimeout)
log?.debug("Connected \(success)")
guard success else { log?.error("Cannot Connect \(err)"); return}
let (successMsg, failmsg) = client.send(str: self.jsonString)
let data = client.read(Config.expectedByteLength)
guard let d = data else { return }
guard let recievedMsg = String(bytes: d, encoding: NSUTF8StringEncoding) else { return }
log?.debug("Recieved msg\(recievedMsg)")
/*Do Something different in every viewController
//For Example
self.Info = Mapper<Info>().map(recievedMsg)
log?.debug("Status\(self.Info?.Status)")
*/
let (clientClosed, errormsg) = client.close()
guard clientClosed else { return }
} catch {
let fetchError = error as NSError
print(fetchError)
// Handle Error
}
}
}
When I try to call this function from the new class in my view controller it does not behave in the same way the function in the view controller did.
Is there anything I should change in the way I have created the class?
Any help will be appreciated as I am very new to swift. Thank you
This is a way your problem could be solved. Skip the custom error throwing part if you don't actually need it. The way to refactor it is using callback function, just send your function into this one, i.e.:
TCPService.connect("some_host", message: "your_message") { response in
Mapper<Info>().map(response)
}
Code:
import Foundation
extension NSError {
static func error(with localized: String) -> NSError {
return NSError(domain: "error", code: 2, userInfo: [NSLocalizedDescriptionKey: localized])
}
}
class TCPService {
static func connect(host: String, andSend message: String, onComplete: String -> ()) {
do {
let client = try TCPClient(addr: host, port: Config.tcpPort)
let (successfulConnection, error) = client.connect(timeout: Config.tcpConnectionTimeout)
log?.debug("Connected \(successfulConnection)")
guard successfulConnection else {
throw NSError.error(with: error)
log?.error("Cannot Connect \(error)")
return
}
let (successfullySent, failureMessage) = client.send(str: message)
let data = client.read(Config.expectedByteLength)
guard successfullySent,
let d = data,
receivedMessage = String(bytes: d, encoding: NSUTF8StringEncoding) else {
throw NSError.error(with: failureMessage)
return
}
log?.debug("Received msg\(receivedMessage)")
onComplete(receivedMessage)
let (clientClosed, errorMessage) = client.close()
guard clientClosed else {
throw NSError.error(with: errorMessage)
return
}
} catch let error as NSError {
let fetchError = error as NSError
print(fetchError)
// Handle errors
}
}
}
I am using nsurlsession on RxSwift.
I am facing two problems about nsurlsession on RxSwift.
I created Custom Observable.
This Observable has used nsurlsession.
nsurlsession.datataskwithrequst was canceled everytime on RxSwift.
My code is here
func getWorkInfo(request:NSURLRequest,type1:C.Type,type2:W.Type? = nil) -> Observable<(C?,[W]?, NSHTTPURLResponse)>{
return Observable.create { observer in
var d: NSDate?
if Logging.URLRequests(request) {
d = NSDate()
}
let session = NSURLSession.sharedSession()
let task = session.dataTaskWithRequest(request) { (data, response, error) in
guard let response = response, data = data else {
ColorLogger.defaultInstance?.error(error)
observer.on(.Error(error ?? RxCocoaURLError.Unknown))
return
}
guard let httpResponse = response as? NSHTTPURLResponse else {
observer.on(.Error(RxCocoaURLError.NonHTTPResponse(response: response)))
return
}
guard let jsonString: String = NSString(data:data, encoding:NSUTF8StringEncoding) as? String else{
observer.on(.Error(ApiError.FormatError))
return
}
//ใซใฆใณใ็ณป
let countObj = Mapper<C>().map(jsonString)
//ไธ่ฆง็ณป
//ไบใค็ฎใๆๅฎใใฆใชใๅ ดๅใฏnilใ่ฟใ
if type2 != nil{
let aryObj = Mapper<W>().mapArray(jsonString)
observer.on(.Next(countObj,aryObj, httpResponse))
}else{
observer.on(.Next(countObj,nil, httpResponse))
}
observer.on(.Completed)
}
let t = task
t.resume()
return AnonymousDisposable{task.cancel()}
}
}
Above method was called by here.
func getWorkCount(dicParam: NSDictionary) -> Observable<WorkCount?> {
// URLไฝๆ
let strParam = dicParam.urlEncodedString()
let strUrl = Const.ShiftApiBase.SFT_API_DOMAIN+Const.ApiUrl.WORK_COUNT+"?"+strParam
// ๆฑไบบใชในใใใผใฟใๅๅพ
let url = NSURL(string: strUrl)!
let request = NSURLRequest(URL: url)
let client = WorkClient<WorkCount,Work>()
ColorLogger.defaultInstance?.debug(strUrl)
return client.getWorkInfo(request, type1: WorkCount.self)
.observeOn(Dependencies.sharedDependencies.backgroundWorkScheduler)
.catchError{(error) -> Observable<(WorkCount?,[Work]?, NSHTTPURLResponse)> in
print("error")
ColorLogger.defaultInstance?.error("UnknownError")
return Observable.empty()
}
.map { countObj,workObj,httpResponse in
if httpResponse.statusCode != 200 {
throw ApiError.Bad
}
return countObj
}
.observeOn(Dependencies.sharedDependencies.mainScheduler)
}
And My subscribe is here.
/**
ๆค็ดขไปถๆฐใๅๅพ
- parameter param: <#param description#>
*/
func getSearchCount(param: [String:String]){
let dicParam = NSDictionary(dictionary: param)
api.getWorkCount(dicParam)
.catchError{
error -> Observable<WorkCount?> in
switch error{
case ApiError.Bad:
ColorLogger.defaultInstance?.error("status error")
break
case ApiError.FormatError:
ColorLogger.defaultInstance?.error("FormatError")
break
case ApiError.NoResponse:
ColorLogger.defaultInstance?.error("NoResponse")
break
default:
ColorLogger.defaultInstance?.error("UnKnownError")
break
}
return Observable.just(nil)
}
.subscribeNext { [weak self] countObj in
self?.count.value = countObj?.returned_count
}
.addDisposableTo(disposeBag)
}
I have two problems.
1:nsurlsession was canceled every time.I don know reason.
2:Even if I got error on NSURLSession,I could not catch error on "CatchError".
By the way,when i try to use the following code,But nsurlsession might be canceled.
It might be a base nsurlsession on RxSwift.
func getWorkCount4(dicParam: NSDictionary) -> Observable<WorkCount?> {
// URLไฝๆ
let strParam = dicParam.urlEncodedString()
let strUrl = Const.ShiftApiBase.SFT_API_DOMAIN+Const.ApiUrl.WORK_COUNT+"?"+strParam
// ๆฑไบบใชในใใใผใฟใๅๅพ
let url = NSURL(string: strUrl)!
let request = NSURLRequest(URL: url)
let session = ApiBase.sharedObj.createNSURLSession()
return NSURLSession.sharedSession().rx_response(request)
.observeOn(Dependencies.sharedDependencies.backgroundWorkScheduler)
.map { data,httpResponse in
if httpResponse.statusCode != Const.HTTP_RESPONSE.HTTP_STATUS_CODE_OK {
throw ApiError.Bad
}
guard let jsonString: String = NSString(data:data, encoding:NSUTF8StringEncoding) as? String else{
throw ApiError.FormatError
}
let countObj = Mapper<WorkCount>().map(jsonString)
return countObj
}
.observeOn(Dependencies.sharedDependencies.mainScheduler)
}
What is this problem?
I could resolve by myself.
Above method does not have any problem.
Below code has problem.
// MARK: - ๆค็ดขใซใฆใณใ
extension SearchCount{
/// ๆค็ดข็จใฎViewModel
var searchViewModel:SearchViewModel {
return SearchViewModel()
}
/**
ๆค็ดขใฎไปถๆฐใๅๅพ
*/
func getSearchCount(){
setApiParameter()
searchViewModel.getSearchCount(apiParam)
}
}
I defined searchViewModel on Class,the i could resolve.