Apache beam seems to be truncating pub sub message payload - google-cloud-dataflow

Question

We've created a pretty simple pipeline for pub sub event processing. The pub sub message payload itself is tab separated csv data.
After the message is read, the payload data is being truncated when inflating back into the event object. Using the direct runner and running locally the pipeline is working end to end.
Its only when running within the Google Cloud Dataflow runner where we are seeing this message data truncated?
// Create the pipeline
Pipeline pipeline = Pipeline.create(options);
LOG.info("Reading from subscription: " + options.getInputSubscription());
//Step #1: Read from a PubSub subscription.
PCollection<PubsubMessage> pubsubMessages = pipeline.apply(
"ReadPubSubSubscription",
PubsubIO.readMessagesWithMessageId()
.fromSubscription(options.getInputSubscription())
);
//Step #2: Transform the PubsubMessages into snowplow events.
PCollection<Event> rawEvents = pubsubMessages.apply(
"ConvertMessageToEvent",
ParDo.of(new PubsubMessageEventFn())
);
// other pipeline functions.....
Here the conversion function, where for every pub sub message were falling into the error case. Note that Event.parse() is actually a scala library but I don't see how that could affect this as the message data itself is what has been truncated between the two stages of the pipeline.
Perhaps there is an encoding issue?
public static class PubsubMessageEventFn extends DoFn<PubsubMessage, Event> {
#ProcessElement
public void processElement(ProcessContext context) {
PubsubMessage message = context.element();
Validated<ParsingError, Event> event = Event.parse(new String(message.getPayload()));
Either<ParsingError, Event> condition = event.toEither();
if (condition.isLeft()) {
ParsingError err = condition.left().get();
LOG.error("Event parsing error: " + err.toString() + " for message: " + new String(message.getPayload()));
} else {
Event e = condition.right().get();
context.output(e);
}
}
}
Here is a sample of the data that is emitted in the log message:
Event parsing error: FieldNumberMismatch(5) for message: 4f6ec25-67a7-4edf-972a-29e80320f67f web 2020-04-14 21:26:40.034 2020-04-14 21:26:39.884 2020-04-1

Answer 1

Note that the Pub/Sub implementation for DirectRunner is different from the implementation in Dataflow Runner as documented here - https://cloud.google.com/dataflow/docs/concepts/streaming-with-cloud-pubsub#integration-features.
I believe the issue is related to encoding because message.getPayload is of type bytes and the code might need to be modified as new String(message.getPayload(), StandardCharsets.UTF_8) in the below line
Validated<ParsingError, Event> event = Event.parse(new String(message.getPayload(), StandardCharsets.UTF_8));

Answer 2

Using readMessagesWithAttributesAndMessageId instead of readMessagesWithMessageId is the workaround according to this bug issue https://issues.apache.org/jira/browse/BEAM-9483.
It does not appear to have been fixed yet.