Reading BigTable and converting to Generic Records using GCP Cloud DataFlow - avro

Question

I am trying to convert BigTable table data to genric record using dataflow .After the conversion is done , i have to compare with another datasets in bucket .
Below is my pseudo code , for pipeline i have used
pipeline
.apply("Read from bigtable", BigTableIo.read)
.apply("Transform BigTable to Avro Genric Records ",
ParDo.of(new TransformAvro(out.toString())))
.apply("Compare to existing avro file ")
.apply("Write back the data to bigTable")
// Function code is below to convert genric record
public class BigTableToAvroFunction
extends DoFn<KV<ByteString, Iterable<Mutation>>, GenericRecord> {
#ProcessElement
public void processelement(ProcessContext context){
GenericRecord gen = null ;
ByteString key = context.element().getKey();
Iterable<Mutation> value = context.element().getValue();
KV<ByteString, Iterable<Mutation>> element = context.element();
}
I am stuck here .

Answer 1

It is unclear what do you mean by comparing to existing data in a bucket. It depends on how do you want to do the comparison, what the file size is, probably other things. Examples of input vs output would help.
For example, if what you're trying to do is similar to Join operation, you can try using CoGroupByKey (link to the doc) to join two PCollections, one reading from BigTable, another reading Avros from GCS.
Or alternatively, if the file has reasonable size (fits in memory), you can probably model it as a side input (link to the doc).
Or, ultimately you can always use raw GCS API to query the data in a ParDo and doing everything manually.