I created one table in BigQuery from Google Sheets, when I tried importing it in Cloud Data Prep it says that there are no tables in the dataset.
I'm not sure whether it's an issue with the Google sheet integration, because when I check the details of the table it says there are 0 rows even though there are 3 rows (it's test data).
I already tried giving Cloud Data Prep viewer permission for both the dataset and the project and nothing changes.
Print screens:
Data Prep
BigQuery table info
BigQuery entries
Well apparently when you create a table from google sheets it is not recognized as an actual table, so I made a query to replicate it to a second table and it works!
Related
I have a CRM board in Monday.com which has contact details for my suppliers. I would like to create a Zapp that pushes the data of this CRM to a Google Sheets file. After the initial push, I want it to always be in sync so if any new items are created in the CRM or existing ones updated, Google Sheets rows update to reflect this.
How can I do this? In the below, I am not sure which accomplishes first A) the initial push of all the data in the board (given that the Google Sheet is empty now) and then B) the continual update + sync.
.
I can find lots of material online on pushing data from Gsheets to Monday but not as much for the vice versa.
I want to create external table in BigQuery and the data source is from Google Sheet. Is it possible to do it using dbt? In the yml file, where should I put the URI?
The main problem is, I don’t have the access to create it directly in BigQuery.
One way to handle a Google Sheet as a source is by creating a new table out of it in BigQuery via Connected Sheets.
Then, you create a new source in dbt that relies on that table, and start building your downstream models from there.
As far as I know, you cannot create a source directly from dbt, unless it is a seed file, which I woul not recommend unless it is a rather static file (e.g. country names and ISO codes, which is not prone to change over time).
We have a similar situation where the data source is from Google Sheet.
The end user updates the Google sheet on a periodical basic and we replicate it using Fivetran to our Snowflake datastore.
DBT can then pick up the data seamlessly.
I would like to setup our organisation's Bigquery IAM so that some users would have access to views under our "Report" dataset, which are queried on Google Sheets. Users with edit rights on these Google Sheets should be able to hit the refresh button and access latest data.
We achieved this by granting some user access to the entire project, but that's very poor security. Instead, I want to limit their access to either the entire "Report" dataset or to specific views under that dataset.
We tried to define some custom role and share the dataset but there is no "refresh" button on the sheet for the user to refresh.
Sharing the dataset with a viewer roles showed the Refresh button but they were not allowed to refresh:
What did we miss?
For access to BigQuery data in Google Sheets, one should have:
An Enterprise Plus, G Suite Enterprise for Education, Enterprise
Standard, or Enterprise Essentials account.
Access to BigQuery.
A project with a billing setup in BigQuery.
A BigQuery Job Creator role on the selected billing project.
BigQuery Data Viewer role on the datasets containing the selected table.
According to this document:
If you share a sheet with someone who doesn't meet the criteria above,
they'll be able to see analysis created with Connected Sheets and
perform regular Sheets operations, but they won't be able to refresh
it or create their own connected sheet.
If you share a sheet with someone who doesn't have access to the correct data tables, they won't be able to create or refresh Connected Sheets analysis, and will need to contact the BigQuery admin.
I have several tables in BigQuery that are sourced from Google Sheets tables. When the Google Sheets table is updated then automatically the table in BigQuery is also updated. I am trying to understand how the log of this event looks like in the Operations Logging. My end idea is to create a sink of theses logs in order to create a Pub/Sub and run scheduled queries based on these events.
Thank you
When you use external Table (Google sheet or other) the data are never stored in BigQuery native storage. It's always external.
Therefore, when you update your Google Sheet, nothing happens in BigQuery. It's only when you query the data, you will read (again) the sheet document and get the latest data.
Therefore, there is no insert log that you can track when you update the data in Google Sheet. The only log that you have is when you perform a request in BigQuery to read the data (external or not), as mentioned by Sakshi.
When the external data source(Google Sheet or other) is updated and the BigQuery table associated with it is queried, BigQuery initiates an insert job which is visible in Cloud Logging.
You can find this log by applying filter resource type as BigQuery Project in Cloud Logging console, ie. you will see protoPayload.methodName set to google.cloud.bigquery.v2.JobService.InsertJob.
For more information on BigQuery Logs you can refer to this documentation.
I am attempting to fully automate a data pull from Jira into Google Sheets using the Jira cloud add-on for Google Sheets.
I am having trouble as when doing a Jira cloud pull, it creates a column in the sheet for each "value" you are pulling out of Jira.
I want to:
place a formula on a particular column that will always be the same data, but often includes data separated by semi-colons.
have the data in specific columns automatically split into 2 columns next to each other when the scheduled weekly job runs to pull the data.
I am unsure how to do this as the data pulled overwrites the entire sheet (including removal of an additional column since the import didn't have that extra value/column when pulling the data).
You can have conditional formatting in place and it is not overwritten by the data import, but if you were to have a formula typed into a specific cell it is overwritten by the data in the import.