Read a csv file and fill the data in a BigQuery Table - google-cloud-dataflow

Question

The following is the code that is supposed to read from a csv file and write to another csv file and BigQuery:
import argparse
import logging
import re
import apache_beam as beam
from apache_beam.io import ReadFromText
from apache_beam.io import WriteToText
from apache_beam.metrics import Metrics
from apache_beam.metrics.metric import MetricsFilter
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import SetupOptions
parser = argparse.ArgumentParser()
parser.add_argument('--input',
dest='input',
default='gs://dataflow-samples/shakespeare/kinglear.txt',
help='Input file to process.')
parser.add_argument('--output',
dest='output',
required=True,
help='Output file to write results to.')
known_args, pipeline_args = parser.parse_known_args(None)
pipeline_options = PipelineOptions(pipeline_args)
pipeline_options.view_as(SetupOptions).save_main_session = True
p = beam.Pipeline(options=pipeline_options)
# Read the text file[pattern] into a PCollection.
lines = p | 'read' >> ReadFromText(known_args.input)
lines | beam.Map(lambda x: x.split(','))
lines | 'write' >> WriteToText(known_args.output)
lines | 'write2' >> beam.io.Write(beam.io.BigQuerySink('xxxx:yyyy.aaaa'))
# Actually run the pipeline (all operations above are deferred).
result = p.run()
It is able to write to the output file but it is not able to do so to the BigQuery Table (xxxx:yyyy.aaaa)
The following is the message that appears:
WARNING:root:A task failed with exception.
'unicode' object has no attribute 'iteritems'
The table contained in the csv file is not written into BigQuery even though the schema is the same and the BigQuery table is empty. I suspect the reason for this is because the data has to be converted to JSON format.
What are the corrections that have to be done to this code in order for it to work properly? Could you please give the lines of code that I have to add in order for this to work?

Answer 1

Looking at the following lines:
1: lines = p | 'read' >> ReadFromText(known_args.input)
2: lines | beam.Map(lambda x: x.split(','))
3: lines | 'write' >> WriteToText(known_args.output)
4: lines | 'write2' >> beam.io.Write(beam.io.BigQuerySink('xxxx:yyyy.aaaa'))
Defines lines to be a PCollection of the lines read from the text file.
Creates a new PCollection of words by splitting each line. But it doesn't actually retain that PCollection, so it effectively does nothing.
Writes the original lines to a text file (so you don't see one word per line, you see one original line on each output).
Writes the lines read from the input to the BigQuery file.
If you look at the BigQuery tornadoes example you can see that (1) you need to convert each line into a dictionary with fields for each column ad (2) you need to provide the schema matching that dictionary to the BigQuerySink. For example:
def to_table_row(x):
values = x.split(',')
return { 'field1': values[0], 'field2': values[1] }
lines = p | 'read' >> ReadFromText(known_args.input)
lines
| 'write' >> WriteToText(known_args.output)
lines
| 'ToTableRows' >> beam.Map(to_table_row)
| 'write2' >> beam.io.Write(beam.io.BigQuerySink(
'xxxx:yyyy.aaaa',
schema='field1:INTEGER, field2:INTEGER'))