While parsing input as JSON and persisting, only firsts JSON object is persisted - google-cloud-dataflow

Question

I am trying to read array of JSON posted to a topic that my pipeline is subscribed to and persist the same to BigQuery. The problem I face while doing so is that it persists only the first object, can someone please provide me insight on what I am doing wrong here.
/** A DoFn that converts a table row from JSON into a BigQuery table row. */
static class FormatAsTableRowFn extends DoFn<TableRow, TableRow> {
private static final long serialVersionUID = 0;
static TableSchema getSchema() {
return new TableSchema().setFields(new ArrayList<TableFieldSchema>() {
// Compose the list of TableFieldSchema from tableSchema.
{
add(new TableFieldSchema().setName("PillBoxID").setType("STRING").setMode("NULLABLE"));
add(new TableFieldSchema().setName("Period").setType("STRING").setMode("NULLABLE"));
add(new TableFieldSchema().setName("Time").setType("TIMESTAMP").setMode("NULLABLE"));
add(new TableFieldSchema().setName("IsTaken").setType("STRING").setMode("NULLABLE"));
}
});
}
#Override
public void processElement(ProcessContext c) {
TableRow jsonRow = c.element();
// Setup a date formatter to parse the date appropriately
SimpleDateFormat ft = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss");
try {
TableRow bQueryRow = new TableRow()
.set("PillBoxID", (String) jsonRow.get("PillBoxID"))
.set("Period", (String) jsonRow.get("Period"))
.set("Time",ft.format(ft.parse((String) jsonRow.get("Time"))))
.set("IsTaken", (String) jsonRow.get("IsTaken"));
LOG.error("Inside try" + bQueryRow.getF());
c.output(bQueryRow);
} catch (ParseException pe) {
LOG.error("ParseException");
LOG.error(pe.getMessage());
}
}
}
and my pipleline code is as shown below,
bigQueryPipeLine
.apply(PubsubIO.Read.topic(options.getPubsubTopic()).withCoder(TableRowJsonCoder.of()))
.apply(ParDo.of(new FormatAsTableRowFn()))
.apply(BigQueryIO.Write.to(tableSpec)
.withSchema(FormatAsTableRowFn.getSchema()));

Answer 1

it is possible to process multiple records if you format the input JSON to include an array of items.
Example input:
{
"items":
[
{"PillBoxID":"ID5", "Period":"Morning", "Time":"2016-03-14T11:11:11", "IsTaken":"true"},
{"PillBoxID":"ID6", "Period":"Afternoon", "Time":"2016-03-14T15:11:11", "IsTaken":"false"}
]
}
The rough example processElement() code adds the 2 items to c.output() for later storage in BigQuery.
#Override
public void processElement(ProcessContext c) throws DatastoreException, IOException{
TableRow jsonRowObj = c.element();
LOG.info("Original input:" + c.element().toPrettyString());
ArrayList<Map> jsonRows = (ArrayList<Map>)jsonRowObj.get("items");
Iterator<Map> iterator = jsonRows.iterator();
while(iterator.hasNext()) {
Map jsonRow = (Map)iterator.next();
// Setup a date formatter to parse the date appropriately
SimpleDateFormat ft = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss");
try {
LOG.info("child input JSON: "+jsonRow.toString());
TableRow bQueryRow = new TableRow()
.set("PillboxID", (String) jsonRow.get("PillBoxID"))
.set("Period", (String) jsonRow.get("Period"))
.set("Time",ft.format(ft.parse((String) jsonRow.get("Time"))))
.set("IsTaken", (boolean) Boolean.parseBoolean((String)jsonRow.get("IsTaken")));
c.output(bQueryRow);
} catch (ParseException pe) {
LOG.error("");
LOG.error("ParseException " +pe.getMessage());
}
}
}