I have a question regarding exporting a data from GEE. I am new to coding and GEE. What I would like to achieve is export pixel band data of points a have managed to upload using Fusion Table. Is is possible for GEE to export band data of those point in form of a table?
My code so far: https://code.earthengine.google.com/8a764b5d22a9f7108152fce1acc1fe16
The code:
// Load a FeatureCollection from a Fusion Table
var CRuHM_small_data = ee.FeatureCollection('ft:1ocXhbAqP_NbA0iE7tivKgKCfTFGseNdibklZj0NX');
// Print and display the FeatureCollection.
Map.addLayer(CRuHM_small_data,{},'CRuHM_small_data');
print(CRuHM_small_data);
//Navigate to area of interest
Map.setCenter(17.3834, -0.8929, 8);
// Select a specific Sentinel-2 image from the archive
var sent2a = ee.Image("COPERNICUS/S2/20170801T090021_20170801T091620_T33MYV");
// Add RGB composite to map, for water/land
Map.addLayer(sent2a,{bands:['B8','B11','B4'], min:0, max:3000}, "water/land");
However, the next step is more complicated for me.
I was trying this code, but something is missing:(
//exporting band data to table
//Export.table.toDrive(collection, description, folder,
//fileNamePrefix, fileFormat, selectors),
Export.table.toDrive({
collection: CRuHM_small_Data,
description: "CRuHM_small_Data",
folder: "GEE",
fileNamePrefix: "Table",
fileFormat: "CSV",
selectors: ["ID", "B3", "B2"]
});
So what you want is a function called sampledRegions for the ee.Image object.
In your case it would be something like this
var sampledData = sent2a.sampleRegions({
collection:CRuHM_small_data,
scale:10
});
Export.table.toDrive({
collection: sampledData,
description: "CRuHM_small_Data",
folder: "GEE",
fileNamePrefix: "Table",
fileFormat: "CSV",
selectors: ["ID", "B3", "B2"]
});
Since you want band information on the points, you have to sample the bands using the points and then export the sampled points.
Also, since it looks like you are trying to export only information from B2 and B3, it would be better if you did a select on your image before sampling.
Something like the following somewhere before your sampleRegions should do the trick.
sent2a = sent2a.select(['B2', 'B3']);
Related
I like to make a histogram of some data that is saved in a nested BigQuery table. In a simplified manner the table can be created in the following way:
CREATE TEMP TABLE Sessions (
id int,
hits
ARRAY<
STRUCT<
pagename STRING>>
);
INSERT INTO Sessions (id, hits)
VALUES
( 1,[STRUCT('A'),STRUCT('A'),STRUCT('A')]),
( 2,[STRUCT('A')]),
( 3,[STRUCT('A'),STRUCT('A')]),
( 4,[STRUCT('A'),STRUCT('A')]),
( 5,[]),
( 6,[STRUCT('A')]),
( 7,[]),
( 8,[STRUCT('A')]),
( 9,[STRUCT('A')]),
(10,[STRUCT('A'),STRUCT('A')]);
and it looks like
id
hits.pagename
1
A
A
A
2
A
3
A
A
and so on for the other ids. My goal is to obtain a histogram showing the distribution of A-occurences per id in data studio. The report for the MWE can be seen here: link
So far I created a calculated field called pageviews that evaluates the wanted occurences for each session via SUM(IF(hits.pagename="A",1,0)). Looking at a table showing id and pageviews I get the expected result:
table showing the number of occurences of page A for each id
However, the output of the calculated field is a metric, which might cause trouble for the following. In the next step I wanted to follow the procedure presented in this post. Therefore, I created another field bin to assign my sessions to bins according to the homepageviews as:
CASE
WHEN pageviews = 0 OR pageviews = 1 THEN "bin 1"
WHEN pageviews = 2 OR pageviews = 3 THEN "bin 2"
END
According to this bin-defintion I hope to obtain a histogram having 6 counts in bin 1 and 4 counts in bin 2. Well, in this particular example it will actually have 4 counts in bin one as ids 5 and 7 had "null" entries, but never mind. This won't happen in my real world table.
As you can see in the next image showing the same table as above, but now with a bin-column, this assignment works as well - each id is assigned the correct bin, but now the output field is a metric of type text. Therefore, the bar-chart won't let me use it (it needs it as dimension).
Assignment of each id to a bin
Somewhere I read the workaround to create a selfjoined blend, which outputs metrics as dimension. This works only by name: my field is now a dimension and i can use it as such for the bar-chart, but the bar chart won't load and shows a configuration error of the data source, which can be seen in this picture:
bar-chart of id-count over bin. In the configuration of the chart one can see that "bin" is now a dimension. The chart won't plot, however, as it shows a data configuration error (sorry for the German version of data studio).
I have the following formula: =ArrayFormula(INDEX(Items!F2:F,MATCH(C2,Items!E2:E,0)))
I would like to extend it such that the entire C column runs the same formula for values. Please help. If a script is necessary to achieve this, I'd like to explore that option too.
Use Apps Script!
Sheet functions (formulae) work great (especially if you are a master like player0), but I find it much easier to work within Apps Script for anything much more complicated than a simple INDEX MATCH. If you are willing to learn some JavaScript, I highly recommend learning some.
Custom Functions
You can write custom sheet functions in Apps Script that you can call with the traditional =FUNCTION() from a cell.
The way it works is that you write a function in Apps Script that returns a two dimensional array corresponding to the area that it needs to fill.
For example, if wanted a function to fill a 2 x 2 block with 1, you would need to make your function return:
[[1,1],[1,1]]
Or you can write it like this:
[
[1, 1],
[1, 1]
]
Implementing Index Match
There are many ways you can implement it, here is an example.
The example spreadsheet has 2 tabs, "Ledger" and "Items".
The goal of the function that follows is to get the costs of the items from the "Items" tab.
function ledgerIndexMatch(){
// Initializing the location of data
let ss = SpreadsheetApp.getActive();
let ledger = ss.getSheetByName("Ledger");
let source = ss.getSheetByName("Items");
let ledgerRange = ledger.getDataRange();
let sourceRange = source.getDataRange();
// Getting the values into a 2D array
let ledgerValues = ledgerRange.getValues();
let sourceValues = sourceRange.getValues();
// Discarding the first row (headers)
ledgerValues.shift();
sourceValues.shift();
// Initializing the output array
let output = [];
// This is where the INDEX MATCH happens
// For each row in ledger
ledgerValues.forEach(ledgerRow => {
// Get the second column (index 1)
let item = ledgerRow[1];
// Initialize the column
let value = [];
// For each row in the source
sourceValues.some(sourceRow => {
// Check if the item is there
if (item == sourceRow[0]) {
// if so, add to value
value.push(sourceRow[1]);
// stop looking for values
return true
// if not matched, keep looking
} else return false
})
// Add the found value (or blank if not found)
// to the output array.
output.push(value);
})
return output;
}
Which can be used like this:
Whats nice about Apps Script is that you can customize it to your heart's content. In this example, the function automatically detects the height of the respective tables, so you don't need to fiddle around with ranges.
You might want to extend this function with arguments so that its more flexible. Or you could just have a few versions of it for different operations, if you don't have too many. Or refactor it... its up to you.
References
Apps Script
Custom Functions
Tutorials
SpreadsheetApp
use:
=ARRAYFORMULA(IFNA(VLOOKUP(C2:C, Items!E2:F, 2, 0)))
Apologise in advance for the previous post. #xavdid did a great job helping me out. Due to my lack of expertise and knowledge in this field I failed to express accurately what I needed. I believe I have now enough information to express what I want to achieve. So I will do my best to express it here.
Here is my input information, Keys and Values. Each position of the keys corresponds to the position of the values.
I believed in order to solve this problem, I would need to know when a book starts and when it doesn't start. I was wrong.
All I need is to match the predefined keys with their values and group them together.
By predefined keys I mean returning only these 7 keys :"Project Details,Project Title,Addons,Upgrade,Word Count,Ebook Type,Upload your file here" with their values (ignore the other keys)
Example of 3 books:
Input Data Keys: Project Details,Project Title,Addons,Upgrade,Word
Count,Ebook Type,_builder_info,_builder_id,_master_builder,Upload your
file here,_builder_id,_master_builder,_builder_id,_builder_info,Ebook
Type,Word Count,Upgrade,Addons,Project Title,Project Details,Project
Details,Project Title,Addons,Upgrade,Word Count,Ebook
Type,_builder_info,_builder_id,_master_builder,Upload your file
here,_builder_id
Input Data Values: Book Description 3,Book Title 3,Book Cover Design - $59.00,No Package,Standard 10K - $270.00,Standard Ebook,Start~~//www.shappify-cdn.com/images/282516/127828454/001_Ebook Standard 325x325.png~~start,start1526659928051,1,https://cdn.shopify.com/s/files/1/0012/8814/2906/uploads/778dfc3dbdf278441776e9f5dd763910.png,start1526659928051,1,start1526659872230,Start~~//www.shappify-cdn.com/images/282516/127828455/001_Ebook Technical 325x325 (1).png~~start,Technical Ebook,Technical 15K - $450.00,No Package,No Addons,Book Title 2,Book Description 2,Book 1 Description,Book 1 Title,No Addons,Essential Package - $79.00,Standard 20K - $540.00,Standard Ebook,Start~~//www.shappify-cdn.com/images/282516/127828458/001_Ebook Standard 20k 325x325.png~~start,start1526659838425,1,https://cdn.shopify.com/s/files/1/0012/8814/2906/uploads/c09635c2e003fd8779a19651e36f4315.png,start1526659838425
Output desired:
[{'Ebook Type': 'Standard Ebook'},{'Ebook Type':'Technical Ebook'},{'Ebook Type':'Standard Ebook'}],
[{'Word Count': 'Standard 10K - $270.00'}, {'Word Count': 'Technical 15K - $450.00'},{'Word Count': 'Standard 20K - $540.00'}]
[{Upgrade: 'No Package'},{Upgrade: 'No Package'},{Upgrade: 'Essential Package - $79.00'}]
[{Project Title: 'Book Title 3'}, {Project Title: 'Book Title 2'}, {Project Title: 'Book Title 1'}]
[{'Project Details': 'Book Description 3'},{'Project Details': 'Book Description 2'},{'Project Details':'Book 1 Description'}],
[{'Addons: 'Book Cover Design - $59.00',{Addons:'No Addons'},{Addons:'No Addons'}],
[{'Upload your file here':'https://cdn.shopify.com/s/files/1/0012/8814/2906/uploads/778dfc3dbdf278441776e9f5dd763910.png'},{'Upload your file here':https://cdn.shopify.com/s/files/1/0012/8814/2906/uploads/c09635c2e003fd8779a19651e36f4315.png'}]
Thank you very much
Ok! So I think I have this how we want it. The big issues are the assumptions we're making about the structure of the data, namely:
there are never commas in values (A book titled "Murder, She Wrote" would bust this because of the comma splitting)
the keys can be in any order relative to each other (might do any number of titles before descriptions) but title 3 will aways come (somewhere) before title 2
Given that, we need to split up our inputs, separate into the 7 keys we want, and then build books out of those arrays. That is the following:
const keys = inputData.keys.split(',')
const values = inputData.values.split(',')
const keysWeWant = "Project Details,Project Title,Addons,Upgrade,Word Count,Ebook Type,Upload your file here".split(',')
let orderedValues = {}
// fill arrays so we can blindly push
keysWeWant.forEach(k => orderedValues[k] = [])
keys.forEach((key, index) => {
if (keysWeWant.includes(key)) {
orderedValues[key].push(values[index])
}
})
console.log(orderedValues)
// have to cut up books now
// don't know how many we have, hopefully our values have the right number of items
// .map(Object) is because of this: https://stackoverflow.com/questions/35578478/array-prototype-fill-with-object-passes-reference-and-not-new-instance
let result = Array(orderedValues['Project Title'].length)
.fill()
.map(Object)
for (const keyWeWant in orderedValues) {
orderedValues[keyWeWant].forEach((val, index) => {
result[index][keyWeWant] = val
})
}
// put book 1 first, this can be removed
result.reverse()
return result
you can see it in action here. Worth noting that in the sample data, there were only two "upload your file here", so the last book (book 1) is missing that key.
Also, when you test this in the zap editor, it'll look like only 1 item is returned. That's just for the test. It'll work normally when it's on an running for real (everything after the code step will run once for each item in the returned array).
Working on a grails application, wherein I am using dc-crossfilter to plot 5 bar graphs which are interconnected to each other.
Now, I want to do a simple thing here : In my first bar graph, there are 14 bars ( x number of departments and 1 "All" graph)
So the problem here is that the presence of "All" in the graph really messes up every other department's number as they are way too low (All is summation of all x number of departments) So each department number is so low that it's not even visible clearly.
But, it is very important to load the data for "All", as all the remaining 4 bar charts are filtered for "All" only.
So, is there any way how I can hide "All" in my first bar graph even though data is loaded? Is this even feasible?
There has to be some simple workaround to do this.
All approaches/suggestions are most welcome.
UPDATE :
Code:
This is the part where I define the groups and dimensions
var devValue = facts.dimension(function (d) {return d.c;});
var devValueGroupSum = devValue.group().reduceSum(function(d) { return +d.g;});
var mvValue = facts.dimension(function(d,i) {return d.b;});
var mvValueGroupSum = mvValue.group().reduceSum(function(d) {return +d.g;});
Now, where and how exactly do I define a fake group and prefilter the data. But will "ALL" be included in the actual dataset ( and not visualization part) even after prefiltering the data.
Code for fake group as in documentation :
var group = {
all:function () {
var cumulate = 0;
var g = [];
_group.all().forEach(function(d,i) {
cumulate += d.value;
g.push({key:d.key,value:cumulate})
});
return g;
}
};
I am really not able to understand the documentation as in how to implement this, can you help me in implementing this?
You can use a "fake group" to prefilter the data. Whenever the chart fetches the data it will go through your fake group, which then fetches the data from the real group and changes it.
The technique is described in the FAQ.
Actually i am using the Highstock library and I am generating a graph with line chart (Data Grouping) with using PHP and MySQL. I am using a JSON format to plot the chart.
I am able to plot the graph with my database. Now I want to open a popup form on click of any data point with ID (primary key in database) of that record and want to save the comment in database.
Data structure my table have ID as primary key, Time (x Value), Points (Y value), Comments (Wants to enter from chart form)
ID: 1,2,3,4,5,6
Time: 1345575960000, 1345576140000, 1345663860000, 1345690200000, 1345719780000, 1345741920000
Points: 1720, 2040, 2570, 1477, 978, 3007
Comments: (This i want to update from chart form)
In this Structure ID is primary key. So i want to save the comments on the basis of ID.
I want to save the comments when i click on data point in cha.
Can you please help me to solve my problem.
You can trap the clicking of points using
series.dataevents.click
Eg. http://jsfiddle.net/gh/get/jquery/1.7.2/highslide-software/highcharts.com/tree/master/samples/highcharts/plotoptions/series-point-events-click/
Once trapped, you could display a dialog with your form on it? The form can submit back to your server to save the comments and then hide itself.
Create your points follows:
{
x: 1142812800000,
Y: 55,
id: 12345
}