This question is edited to make requirements clearer:
I am involved with a charity with multiple location. It provides low cost medicines. Current the transaction data - sale purchases is stored in Googlesheets in each location. As it grows it is becoming a pain to analyse one by one.
The data is structured. Each Table/Tab has defined fields. Each location has identical structure and I can easily have a unique ID.
I have also experimented with local Postgresql database. Data is in standard Table format - each location has identical format.
As each shop is selling each day, each day there are new transactions.
I need a simple way of collecting all this information in one place in the cloud so that I can analyse the data using SQL type queries without having to manually get it from each location each day. In total there are some 2000 new transactions in a day that need to be added to database.
One logical solution would be for each local postgresql database to send new data to a cloud master postgresql database - using WAL? Local databases dont have 24x7 internet access - net is patchy. Charity is in India. I am in UK.
Ideally I need a solution where when the local computer finds net, it transmits the new transactions to the cloud database.
"""Import these modules"""
import gspread
from oauth2client.service_account import ServiceAccountCredentials
"""Import the sheet"""
scope = [
'https://www.googleapis.com/auth/spreadsheets',
'https://www.googleapis.com/auth/drive'
]
creds = ServiceAccountCredentials.from_json_keyfile_name("Path to your
credentials.json", scope)
client = gspread.authorize(creds)
sheet = client.open("Your sheet").sheet1
"""The cell on Top of a column is for the title of the column """
sheet.updatecell(1, 1, "Sales")
"""Reading the file where the data is stored and writing it in the sheet"""
with open("Path to the file where the sales are stored", "r") as l:
data = l.readlines()
for e, i in enumerate(data):
i = i[:-1]
sheet.updatecell(len(e+2), 1, i)
I hope this helps, I can't do more since I don't know how the data is stored and organized.
Related
I have a dataset in bigquery containing dozens of tables. The business wants to access these tables through a connected gsheet
The gsheet UI only allows connecting tables one by one, this is a bit tedious. Is there an alternative way to connect all the tables in the dataset to individual sheets with the same gsheet document in one go?
The UI only allows one by one but you can also use the Sheets API to add data sources. You could build your own app and send the requests to the API as AddDataSourceRequest. An easier way would be to use Apps Script since it's already connected with Sheets and it also has a DataSourceTable class. A quick example of how to add a data source in Apps Script would look like this:
function addConnectedTable() {
SpreadsheetApp.enableBigQueryExecution();
var ss = SpreadsheetApp.getActive()
var spec = SpreadsheetApp.newDataSourceSpec()
.asBigQuery()
.setProjectId('your-billing-project') //set your own project ID here
.setRawQuery('select * from `bigquery-public-data.new_york_trees.tree_species`')
.build();
ss.insertDataSourceSheet(spec).asSheet().setName("Some Name"); //sets the name of the new sheet
}
As a sample I used a query from a public data set, but you can just write your own queries for each of your tables and run the script with them to quickly add them.
Sources:
Sheets API AddDataSourceRequest
Connected Sheets in Apps Script
Apps Script DataSourceTable
I need to pull data from a Google Sheet into BigQuery. So far I have created a Table in BQ using the CSV option and all of the data imported fine. But now I need to automatically update the BQ table with the data in the Google Sheet's tab. Can anyone point me in the right direction? Thanks in advance.
With BigQuery you can directly query your data from the Google Sheet by creating an external table from the console.
Given your data is properly formatted, you just have to "create table", from "Drive", provide your Google Sheet uri, fill-in some additional settings and that's it !
Any changes in the spreadsheet will be immediately accessible in BigQuery as well.
Documentation for reference.
I looked at BQ when they released and quickly dismissed it as it is free for a short period and then we have to pay. I have my own domain/website and it comes with MySQL, FTP, and so much more.
Anyway, I have many projects that go the opposite direction, pull the data down from the SQL DB to sheets, but which direction does not really matter.
Google sheets has Triggers, I have triggers set to launch at certain times, usually every day, you get to set up the trigger to fire off a function you specify.
https://developers.google.com/apps-script/reference/script/clock-trigger-builder
You can also set triggers via scripts
/**
* Creates time-driven triggers
*
* https://developers.google.com/apps-script/reference/script/clock-trigger-builder
*/
function createTimeDrivenTriggers() {
// Trigger every day at 04:00AM CT.
ScriptApp.newTrigger('csvDaily')
.timeBased()
.everyDays(1)
.atHour(4)
.create();
}
Is it possible to created a Stored Procedure in SAS Enterprise Guide and allow the user to enter in a list of values, without having to manually enter in the list?
I use more Base SAS than EG, so I'm not an expert on Stored Procedures. Currently, an analyst in my area may have to search for a list of values like so:
012345678
123456789
231456789
091236574
439857345
120129038
230918239
....
....
N
and is using a Stored Procedure that was built to enter in these values. However, this is not efficient as this last can be >40 values, and SAS will only allow you to enter in one at a time.
I've been messing around with the prompt manager for an hour or so and haven't had any luck. I've also tried 'User selects from a static list', using an excel doc that I imported. Which worked great ad-hoc, but, because the values will always be different, I can't figure out how to make EG first import this excel doc, then bring up the prompt for her to select all the (new) values, then run the rest of the program.
Also, it seems that I would have to change the 'Static Value List' in the prompt manager every time the doc was imported, even if the rest of the program was conditioned on the import of the excel doc. I'm going to continue playing around with this, but looking for ideas as to if anyone has done this previously.
Sounds like you want "select multiple values from a dynamic list". I suggest you read the excel file that holds all the response options into a SAS dataset. Then register that dataset in the SAS metadata server. When you create a dynamic prompt you point to the source SAS dataset that holds the response options. After creating the prompt, you can update the dataset any time you want (add/delete records), and then STP user will see those updated response options in the prompts.
It may also be possible to register an Excel file in metadata instead of reading it into a SAS dataset. But I always try to limit Excel usage as much as possible.
We need to get the daily data from the "Microsoft.VSTS.Scheduling.CompletedWork"field (which is detailed in Workload, scheduling and time tracking field references). However I get data from the Analysis database and found that it only records one last new data,and can't get the historical data.
For example the task of ID 3356, who's "CompletedWork" is 3 hours in 2016/8/4, and I get the exact 3 hours-data from the Analysis database in the second day, 2016/8/5, as the pictures in this post show.
Then on the 2016/8/5, I update the "CompletedWork" from 3 hours to 4 hours and I get the exact 4 hours-data from the Analysis database in the second day, 2016/8/6. However the 3 hours-data of 2016/8/4 is lost. Well, How can I get the historical data of "Microsoft.VSTS.Scheduling.CompletedWork"?
First of all, it's important to understand that the CompletedWork is a cumulatieve data field. So when one user enters 3 and another enters 4, the total number of hours worked on the field is 4 not 7.
The warehouse has a granularity of a day and keeps that data int he cube, though the relational warehouse tables will store all the changes to the reportable fields on a per-revision bases. You can't easily query this data using the qube or Excel Power Pivot and they're lost in the Dim* and fact* tables, but you can write a SQL query against tfs_warehouse and iterate through the tables containing the workitem data (tbl_workitems[are|were|latest]). This is much slower and much harder to build unfortunately.
Your other alternative is to use the TFS Client Object Model and query the WorkItemStore object directly. You'll be able to query all work items of interest and iterate through them and their revisions. The API for workitems is relatively easy to use and is well documented.
If you're on TFS 2015 you can also use the new REST api to query workitem data and revisions.
I have a very large excel spreadsheet that consists of a user name, a location, a date, and some fields of numbers, for example.
User,location,date,value1,value2,value3
Steve,NYC,2012,9,1,3
Steve,NYC,2011,3,3,2
Steve,CA,2011,1,2,0
Michael,CA,2012,10,3,2
Michael,CA,2011,10,2,0
How would I go about organizing a rails site such that one can view all the values for a certain user?
For example,
/users/steve/all
would display all the values in descending order of date where user=steve.
/users/steve/nyc
would display all the values in descending order of date where user=steve and location=nyc.
I think I would need to create a users model and import all the data from the excel into the database, but I'm lost about how to do that.
The application, in essence, would be a simple data visualizer. Maybe I have to separate the database and create a user has_many :locations and locations :belongs_to user, I'm not sure. I want the data to be viewed in all sorts of ways—maybe I want to display all the users from a certain location, or view all the locations of a certain user, etc...
I suggest setting up your model within your rails application first. Then, you can just write a rake task probably similar to this question or you can build it from scratch. There's also a railscast.
If you need to directly import from Excel (e.g. the excel sheets are uploade by a user). You can use one of the following gems to importing data from an Excel Sheet
roo Reads new and old excel format (+others)
spreadsheet Reads and writes old exel format
If you only have this one excel sheet it will be far easier to simply export the data to csv and follow the answers given in the stackoverflow question mentioned above.
As for the second part of your question on how to model your database you already had the right idea.
Easiest is to fully model what your domain looks like and transform the data accordingly.