I already read this question, but it didn't solve my problem.
I read from a PubSub topic in my Dataflow topology, but I am always getting the error of "resource setup failure":
Even if I have already enabled all of the Google Cloud APIs for the project.
Do you have any ideas? Could it be some issue with credentials?
Where can I get a more meaningful error message?
I needed to create the topics by hand.
Dataflow automatically creates the subscriptions, not topics.
Related
I have been trying to build a pipeline in Google Cloud Data Fusion where data source is a 3rd party API endpoint. I have been unable to successfully use the HTTP Plugin, but it has been suggested that I use Pub/Sub for the data ingest.
I've been trying to follow this tutorial as a starting point, but it doesn't help me out with the very first step of the process: ingesting data from API endpoint.
Can anyone provide examples of using Pub/Sub -- or any other viable method -- to ingest data from an API endpoint and send that data down to Data Fusion for transformation and ultimately to BigQuery?
I will also need to be able to dynamically modify the URI (e.g., date filter parameters) in the GET request in this pipeline.
In order to achieve the first step in the tutorial you are following
Ingest CSV (Comma-separated values) data to BigQuery using Cloud Data Fusion.
You need to set up a functioning pub/sub system. This can be done via the command line, the console, or in your case the best would be to use, one of the client libraries. If you follow this tutorial you should have a functioning pub/sub system.
At that point you should be able to follow the original tutorial
I need to publish messages to GCP Pub/Sub with a POST request as the platform I'm using (Zoho) does not allow for any of the GCP libraries. I'm not sure how to make the request in a simple way, as the normal authentication system seems complex.
Is there an easy way to publish a message using, e.g., an API key?
Alternatively is there a simple way to create an API endpoint within GCP that I can then forward data on to the messaging system?
I have used the python client to publish to Pub/Sub, but cannot make POST requests because of the authentication issues.
Both of your questions will have the same answer, yes, and Google Cloud Endpoints is your way to go here.
With Google Cloud Endpoints you can create a custom endpoint and use API keys to authenticate the requests that are being done. There's a really good how-to guide from medium you can follow in order to set up your endpoint and your Pub/Sub push subscription.
More information about creating push subscriptions can be found in the public documentation.
I am a beginner to spring cloud data flow and i am following their official doc. But when i deploy the stream from spring cloud data flow dashboard it just stuck on loading and the stream is never deployed.
The DSL for the stream i want to deploy is:
http | log
I changed the ports for skipper but nothing works
I expect that when i click on deploy the stream then it should show me the status 'deploying' but instead it just keeps on loading forever.
When reporting for issues like this, it'd be great if you could share the versions in use and the logs for review.
Depending on the platform (local, cf or k8s), I'd recommend reviewing the troubleshooting steps included in the SCDF Microsite.
If you followed those steps and if you still see issues, please update the description of the post with the relevant details, and we can review then.
According to what I read of DataFlow, the Pub/Sub datasource only gives the message body to work with in the pipeline. We have a use-case where we want to inspect the attributes of the message to make certain decisions. Is there any way of achieving this currently? I'm open to extending the Pub/Sub I/O to incorporate this if required.
Currently, there is no way to access the message attributes of your messages via the PubsubIO connector, but it would clearly be useful to do so. This is tracked in Apache Beam (incubating) as the issue BEAM-404.
I recommend following this issue to keep abreast of new developments.
I am developing RabbitMQ token auth plugin, where the token needs to be included in AMQP header so it can be validated upon every sent / consumed message.
I am wondering how can I achieve it? So far I am only familiar with RabbitMQ auth plugins and do not know much about other plugin mechanisms. After quick research I have found rabbit_channel_interceptor behavior which sounds like it could do the job.
I have read rabbitmq's source code about auth. In the source code tree, please pay attention to the files named as "rabbit_auth_machanism.erl", "rabbit_auth_backend", "rabbit_auth_backend_internal". In addition, there is another plugin named "xxx ldap".
After reading carefully and know how to integrate and build the rabbitmq project groups, you can start programming.