Google Cloud Storage solution to join 2 CSV in a bucket file based on a common column - join

Question

I need help/suggestions to implement the below use case.
I have 2 csv files in Google Cloud Storage bucket, I need to join these 2 files, based on one common column, and I need to save the output file back into the Google Cloud Storage bucket.
I need to implement this using any Google Cloud solution (cloud data flow with beam python), Cloud function or any other Cloud solutions, since I am new to Google cloud platform, I request all to help me on implementing this use case.
Looking forward to hearing from you
Thanks in advance

Answer 1

You have several way to achieve this. If the result of the merge take less than 1Gb and you want only 1 output file, you can do like this
Query external CSV files from BigQuery (federated query) and save the result in a temporary table like this
CREATE OR REPLACE EXTERNAL TABLE mydataset.table1
OPTIONS (
format = 'CSV',
uris = ['gs://mybucket/file1.csv'],
skip_leading_rows = 1
)
CREATE OR REPLACE EXTERNAL TABLE mydataset.table2
OPTIONS (
format = 'CSV',
uris = ['gs://mybucket/file2.csv'],
skip_leading_rows = 1
)
CREATE TABLE mydataset.newtable
OPTIONS(
expiration_timestamp=TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)
) AS
SELECT *
FROM mydataset.table1 join mydataset.table2 ON ....
Then, export the temp table mydataset.newtable to GCS
Else, you can use the solution that I describe in this article (that I wrote)
EDIT 1
You can use this sample of workflow definition that do what you need.
- loadFile1:
call: http.post
args:
url: https://bigquery.googleapis.com/bigquery/v2/projects/<projectID>/jobs
auth:
type: OAuth2
body:
configuration:
query:
query: CREATE OR REPLACE EXTERNAL TABLE mydataset.table1 OPTIONS (format = 'CSV', uris = ['gs://mybucket/file1.csv'], skip_leading_rows = 1)
useLegacySql: false
- loadFile2:
call: http.post
args:
url: https://bigquery.googleapis.com/bigquery/v2/projects/<projectID>/jobs
auth:
type: OAuth2
body:
configuration:
query:
query: CREATE OR REPLACE EXTERNAL TABLE mydataset.table2 OPTIONS (format = 'CSV', uris = ['gs://mybucket/file2.csv'], skip_leading_rows = 1)
useLegacySql: false
- joinQuery:
call: http.post
args:
url: https://bigquery.googleapis.com/bigquery/v2/projects/<projectID>/jobs
auth:
type: OAuth2
body:
configuration:
query:
query: CREATE TABLE mydataset.newtable OPTIONS( expiration_timestamp=TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)) AS SELECT * ......
useLegacySql: false
result: queryResult
- getState:
call: http.get
args:
url: ${"https://bigquery.googleapis.com/bigquery/v2/projects/<projectID>/jobs/" + queryResult.body.jobReference.jobId}
auth:
type: OAuth2
result: jobState
next: testState
- testState:
switch:
- condition: ${jobState.body.status.state == "DONE"}
next: extractData
next: waitAndGetState
- waitAndGetState:
call: sys.sleep
args:
seconds: 1
next: getState
- extractData:
call: http.post
args:
url: https://bigquery.googleapis.com/bigquery/v2/projects/<projectID>/jobs
auth:
type: OAuth2
body:
configuration:
extract:
destinationUri: gs://<YourBucket>/bq-extract.csv
destinationFormat: CSV
sourceTable:
projectId: <projectID>
datasetId: mydataset
tableId: newtable
result: extractResult
- returnOutput:
return: ${extractResult}
Then, use Cloud Scheduler to call directly the Create Workflow execution API with an empty body {} and a OAuth2 authentication mode.