I am trying to build a daily tracker for my work group and have been pulling the data from a weburl using IMPORTHTML function. But often the function fails saying Resource at url contents exceeded maximum size. What is the limit on the rows that can be imported using IMPORTHTML? It seems to be breaking somewhere around 10k rows.
If I have to import more rows than that, say 100k, what is the best way to do that?
You can use apps script to accomplish it.
With UrlFetchApp you can get all the data from your url.
Once you have the data you can use SpreadsheetApp to insert these data in the sheet.
Related
I have a spreadsheet that has a lot of sheets of data grouped by category. I have a main sheet that I want a user to be able to choose which categories of data to use, and it will pull that data into one long list.
Here is an example spreadsheet of what I want to do: LINK TO SHEET
I could combine all the data into one spreadsheet, but if I were to do that with the dataset I am working with, I would have tens of thousands of rows. So, having the data stored on separate sheets is preferred. I am open to having separate spreadsheets altogether if that will make it easier. Currently, I have tried Using INDIRECT and QUERY, but can only get the first sheet of data to show.
I would prefer to stick to normal functions, but could jump into appscript if I need to create a custom formula
I have tried using INDIRECT and CONCAT, and can get the first set of data, but not subsequent data. Also tried wrapping it in a query like this:
=QUERY({ARRAYFORMULA(INDIRECT(CONCAT(A2:A,"!a2:z100"))},"select * where Col1 is not null",1))
Splitting it up into separate spreadsheets, I was able to use the function:
=QUERY({ARRAYFORMULA(IMPORTRANGE(B2:B,"A1:Z10"))},"select * where Col1 is not null",1)
But it also would only pull the first set of data, not the subsequent rows.
All of this was me first attempting to get the information, not getting the information filtered by the sheets that were checked as well. I am pretty familiar with appscript as well and open to appscript solutions, but would prefer to stay away from it if possible.
You do not need additional column. Use REDUCE() with few other formula-
=REDUCE(HSTACK("Dataset Name","Data 1","Data 2","Data 3","Data 4"),FILTER(A2:A,B2:B=TRUE),
LAMBDA(x,y,VSTACK(x,QUERY(INDIRECT(y&"!A2:E"),"where A is not null"))))
I want to append a row of cells to a google sheet and also attach some developer metadata to that row.
In the Google Sheets v4 API, I know you can use batchUpdate to append a row with the appendCells request, and you can add developer metadata using the createDeveloperMetadata request.
My issue is that I wanna set some developer metadata to specifically the newly appended cells atomically. There's not really a way to specifically ensure the range of the newly added row in createDeveloperMetadata, and if I use two different requests, someone else may insert a row between those requests which could shift all the rows, causing the appended cell's range to be pointing to an incorrect row.
Is there a way to attach developer metadata to a newly added cell atomically?
Answer:
There is not currently way of ensuring that the sheet structure hasn't changed between requests.
More Information:
Your The best option, I think, is to make two sequential requests in the same batch. Though this isn't foolproof, in very unlucky circumstances. Even inserting the row directly using an UpdateCellsRequest isn't foolproof either, as simply knowing which row you inserted the data doesn't exclude the possibility that someone else may insert/delete a row before it between the two requests.
Feature Request:
You can however let Google know that this is a feature that is important for the Sheets API and that you would like to request they implement it. Google's Issue Tracker is a place for developers to report issues and make feature requests for their development services.
The page to file a Feature Request for the Google Sheets API is here.
References:
Requests - UpdateCellsRequest | Sheets API | Google Developers
Requests - AppendCellsRequest | Sheets API | Google Developers
The solution I figured out was essentially:
Fetch the dimensions of the sheet
Perform a batch update with insertDimension, updateCells, and createDeveloperMetadata all performed for the same sheet dimension index at the end of the sheet
This basically ensures that the dimension index will point to the same row for all 3 operations in the batch update, and if the index points out of bounds, all 3 operations will fail
Im having trouble importing data from my another sheet. Its quite big, has over 500 rows and to 100 columns. (I dont know if thats important but it has about 50 sheets, I want to import from first biggest one)
Importhtml and importdata take too long to load and then say that source is too big.
But what I want is importrange and that says internal error all the time.
Do you have any advice how can I import my data? Maybe even how to change source so it can be imported from?
Sheet I want import to
The function to use is IMPORTRANGE
=IMPORTRANGE("XXXXXXXXXX";"Prices!A1:AJ")
(where XXXXXXXXX is the sheet ID)
Pro tip
But what I want is importrange and that says internal error all the time.
A good practice is to first link the two sheets by using a very small range like:
=IMPORTRANGE("XXXXXXXXXX";"Prices!A1:A3")
and then change the range to meet your needs
=IMPORTRANGE("XXXXXXXXXX";"Prices!A1:AJ")
Try coupler.io.
It has a free plan and allows to COPY the data from one spreadsheet to another on a schedule. It means you can split your calculations, make heavy ones on one spreadsheet and then copy results to the one when you work with them.
It also allows to import from CRM, financial apps etc, but your case is covered with GSheet -> Gsheet import.
ImportRANGE returns a #NAME error occasionally as it updates itself every 30 mins.
I took a look through Google Sheet Script Returning #NAME? but there sadly wasn't any solutions. The best guess at a response was about how google sheets loads custom formulas, but since =IMPORTRANGE() is a vanilla formula it shouldn't have a problem with that.
My formula is
=IMPORTRANGE("URLHERE",(A5&"!"&"D1"))
First the URL, then the final section allowing it to request data from a different sheet in the file for every line. I have setup the sheets that contain the source data to be called 1,2,3,4, ect up to 100 to accomodate for this.
I am trying to import data from a different sheet that has a custom script running to import JSON data. The JSON is imported perfectly and once a day a macro will run that updates the JSON values. I have a separate sheet that I don't want to run the script on, so I am using IMPORTRANGE to get the data from JSON imports. Every 30 minutes Importrange updates itself and it researches for the values. It would work fine for me for this to be once a day, however, I can't find a way to change that.
Most of the time it has no problems, occasionally it has a problem with a couple of the imports but most are fine. A couple of times it has hit #NAME on every single IMPORTRANGE.
Problem:
In Google Sheets, I use the IMPORTDATA function with an API call to a 3rd party data provider that frequently produces a text string that is longer than the cell limit of 50,000 characters. I have no control over the length of the text string that is provided to me, and when the 50,000 character limit is exceeded, I end up with no data at all.
Question:
Is there any way to use scripting for the API call (to avoid the 50,000 character limit) - and then parse that data to two or more cells so that the cell limit is not exceeded?
My suggestion would be to use Google Apps Scripts to make the API call. That will return your data as JSON, a string, or some other data type depending on how the API returns data.
Once you have the data in your Google Apps Script you could split the data out into smaller chunks. There are lots of JavaScript string methods for that. You can convert JSON to a compact string using JSON.stringify(yourJSONObject). Once you have your data split out how you want, you can use the Google Sheets API (https://developers.google.com/sheets/api/quickstart/apps-script) to insert the cells you need.
On a side note, I know it says the character limit is 50K everywhere you look, but I have been adding cells that have close to 500K characters in them. Perhaps that is because it is my work account which has G Suite. I would like to read some documentation somewhere on why I can enter way more than 50K characters in a cell.
Documentation on API Calls in Google Apps Scripts:
https://developers.google.com/apps-script/reference/url-fetch/url-fetch-app
https://developers.google.com/apps-script/guides/services/external