Google Assistant SDK refusing authenticated channel as "UNAUTHENTICATED" - oauth

Question

I am trying to create a Google Assistant for my Raspberry Pi in Kotlin. I implemented a OAuth flow using the so called "device flow" proposed in this IETF draft, since my Raspberry shall later just expose a web interface and does not have any input devices or graphical interfaces.
Google does support this flow (of course) and I obtain a valid access token with user consent in the end. For testing purpose I also tried a default authorization flow that will just forward the user to localhost, as it is normally done but it did not solve the problem.
I tested the access token using this tool and it confirmed validity of scope and token. So the token itself should work.
Scope is: https://www.googleapis.com/auth/assistant-sdk-prototype as documented here
This actually does not point to any valid web resource but is referenced in every documentation.
Then I tried to stream audio data to the assistant SDK endpoint using the gRPC provided java stubs. As took a third party reference implementation as a guide how to authenticate the rpc stub. But neither the reference implementation nor my own one works. They both report
io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
The stub is authenticated this way:
embeddedAssistantStub.withCallCredentials(
MoreCallCredentials.from(OAuth2Credentials
.newBuilder()
.setAccessToken(
myAccessToken,
myAccessTokenExpirationDate))
.build()))
and the authenticated request is performed like this:
val observer = authenticatedEmbeddedAssistantStub.converse(myStreamObserverImplementation)
observer.onNext(myConfigConverseRequest)
while(more audio data frames available) {
observer.onNext(myAudioFrameConverseRequest)
}
observer.onCompleted()
(I prefixed pseudo variables with "my" for clarity, they can consist of more code in the actual implementation.)
I even contacted the author of this demo implementation. He told me, last time he checked (several months ago) it was working perfectly fine. So I finally ran out of options.
Since the client implementation I took as reference used to work and I do actually authenticate the stub (although the error message suggests the opposite) Probably, either my valid access token with correct scope is not suitable chosen for the assistant API (though I followed the suggestions of google) or the API servers had a change not properly documented in the getting started articles by google.
So: Did anyone ran in the same problem and know how to fix it? I have the project on github. So if anyone needs the broken source code, I can do a temporary commit that produces the error.
Note, to save some works for mods: This issue referres to this and this question, both unresolved and using different languages but describing a similar problem.

Answer 1

Well, seems I was right about my second assumption: The error is server side. Here is the github issue, let's just wait for the fix.
https://github.com/googlesamples/assistant-sdk-python/issues/138