I have been trying to build a pipeline in Google Cloud Data Fusion where data source is a 3rd party API endpoint. I have been unable to successfully use the HTTP Plugin, but it has been suggested that I use Pub/Sub for the data ingest.
I've been trying to follow this tutorial as a starting point, but it doesn't help me out with the very first step of the process: ingesting data from API endpoint.
Can anyone provide examples of using Pub/Sub -- or any other viable method -- to ingest data from an API endpoint and send that data down to Data Fusion for transformation and ultimately to BigQuery?
I will also need to be able to dynamically modify the URI (e.g., date filter parameters) in the GET request in this pipeline.
In order to achieve the first step in the tutorial you are following
Ingest CSV (Comma-separated values) data to BigQuery using Cloud Data Fusion.
You need to set up a functioning pub/sub system. This can be done via the command line, the console, or in your case the best would be to use, one of the client libraries. If you follow this tutorial you should have a functioning pub/sub system.
At that point you should be able to follow the original tutorial
Related
I am new to the world of APIs. I am trying to utilize a REST API to GET and POST from a certain website and persist those values once a day into a SAP HANA table via SDI replication. I was able to successfully test the API in Postman using a JSON file for my parameters. Now I'm unsure of the next step of how to put it all together and get it into our HANA system.
I need to publish messages to GCP Pub/Sub with a POST request as the platform I'm using (Zoho) does not allow for any of the GCP libraries. I'm not sure how to make the request in a simple way, as the normal authentication system seems complex.
Is there an easy way to publish a message using, e.g., an API key?
Alternatively is there a simple way to create an API endpoint within GCP that I can then forward data on to the messaging system?
I have used the python client to publish to Pub/Sub, but cannot make POST requests because of the authentication issues.
Both of your questions will have the same answer, yes, and Google Cloud Endpoints is your way to go here.
With Google Cloud Endpoints you can create a custom endpoint and use API keys to authenticate the requests that are being done. There's a really good how-to guide from medium you can follow in order to set up your endpoint and your Pub/Sub push subscription.
More information about creating push subscriptions can be found in the public documentation.
I am trying to create BigQuery Data transfer config for Google Adwords through API using a programming language (Python, Java). I looked at the documentation about BigQuery data transfer API. But there is no proper process for that. Maybe I could not understand properly. Can anyone help me in understanding how to use API to get daily analytic data from YouTube instead of paying YouTube to use their BigQuery Data transfer?
You need to get started using Adwords SQL
https://developers.google.com/adwords/api/docs/guides/first-api-call
Refer to the Getting Started section of the Python client library README file to download and install the AdWords API client library for Python.
I am new to Google Cloud Platforms and not quite sure with the whole architecture but what I am trying to achieve is to save some data to Google Cloud from an iOS application and do some analytics work on this data using Google Cloud Products, such as: Dataproc and Datalab. From what I read so far I would need to create a dataset in Google Cloud BigQuery and create a table in it. I have done this using the Google Cloud Web UI but now I want to populate the table from my iOS app. I can't seem to find how to do that.
The most painless route would be to wire up Firebase Analytics and then turn on its daily log export to Big Query, as described by Google in the walkthrough Importing Firebase Analytics Data into BigQuery. Google maintains the entire analytic export stack for you then, seeing as they also maintain Firebase. The downside is that the analytics export happens only daily.
Alternatively, you'd be looking at using the Big Query REST API to upload data, as documented by Google in their Loading Data with a POST Request how-to guide. The iOS tooling for that would be your usual NSURLSession and NSURLDataTask APIs, or whatever abstraction you prefer that's built atop them.
Google does maintain a collection of iOS-native APIs, but unfortunately, Big Query is not included amongst the supported APIs as of May 2017. There are native Big Query clients for Go, C#, and Java, amongst others. So you could use your own API for upload to a server you control, and then use one of those client APIs serverside to implement the actual Big Query integration, if you wished.
I'm still new to GAE, and I would like to have more wisdom about couple of things.
I searched in documentation, but I think I'm just too stupid to understand some things from documentation.
How can I combine Google Cloud SQL with endpoints? Is there such possibility?
How can I use endpoints to upload videos to google platform?
You should be able to use anything you can do on a non cloud endpoints api to cloud endpoints like google cloud sql. But since it's on preview you might encounter bugs/changes when it goes out of preview. You should create a Cloud SQL tests models on regular app engine app then try to use it on cloud endpoints, so you can minimize debugging for errors.
https://developers.google.com/appengine/docs/python/cloud-sql/
You will need to use a blobstore api:
https://developers.google.com/appengine/docs/python/blobstore/
on your endpoints have a method that creates the upload url and use that to upload from your app then on the uploadHandler it will trigger once the whole file has been uploaded, process your blobInfo key store it appropriately.