I have a golang app running in a kubernetes cluster which must push an image it builds periodically to gcr.io using REST endpoint (There are project constraints that does not allow me to use libraries such as go-containerregistry, docker cli for the push). I understand from documentation that I need to add "X-Registry-Auth" header to the requests to handle authorization. I am retrieving dockerconfigjson from imagePullSecret on the cluster (that works well for pod creation) and adding it as header value. However, I get an unauthorized error from gcr endpoint.
secret, err := secretsClient.Get(context.TODO(), "imagePullSecret", metaV1.GetOptions{})
secrets := []corev1.Secret{*secret}
keychain, _ := k8schain.NewFromPullSecrets(context.TODO(), secrets)
reg, _ := name.NewRegistry("gcr.io")
authorizer, _ := keychain.Resolve(reg)
auth, _ := authorizer.Authorization()
encodedJSON, err := json.Marshal(*auth)
if err != nil {
panic(err)
}
authStr := base64.URLEncoding.EncodeToString(encodedJSON) //being used as auth header value
Content of authStr created above has the following:
{
"username":"_json_key",
"password":
"{\n \"type\": \"service_account\",
\n \"project_id\": \"project\",
\n \"private_key_id\": \"porject-key-id\",
\n \"private_key\": \"-----BEGIN PRIVATE KEY-----
\n <validkeyhash>
\n-----END PRIVATE KEY-----\\n\",
\n \"client_email\": \"something#something.iam.gserviceaccount.com\",
\n \"client_id\": \"123344567678786\",
\n \"auth_uri\": \"https://accounts.google.com/o/oauth2/auth\",
\n \"token_uri\": \"https://oauth2.googleapis.com/token\",
\n \"auth_provider_x509_cert_url\": \"https://www.googleapis.com/oauth2/v1/certs\",
\n \"client_x509_cert_url\": \"https://www.googleapis.com/robot/v1/metadata/x509/manager%40something.iam.gserviceaccount.com\"
\n}\n",
"auth":"<base64-encoded-imagePullSecret>"}
Related
I am trying to do some testing using keycloak and go in a local docker setup. Currently, I have a keycloak container (with a postgres backend) running on 8080. I set this up with a client that has service account access, and have my ClientID, client secret, realm etc all in a config file.
I then have a go API running in docker as well. In this go API, at startup I create a new gocloak client and hold reference to it:
client := gocloak.NewClient(host)
log.Infof("clientid: %s pw: %s // realm: %s", clientId, clientSecret, realm)
ctx := context.Background()
tkn, err := client.LoginClient(ctx, clientId, clientSecret, realm)
if err != nil {
log.Errorf("error occurred connecting to keycloack: %s", err)
}
The value of host above is:
http://keycloak:8080
as the docker container running keycloak is just called keycloak. Obviosuly if i Try and use localhost:8080 here, it fails because we are inside a container.
And then as a middleware on any calls coming in to this service's routes, I use that client and check:
authHeader := c.GetHeader("Authorization")
parts := strings.Split(authHeader, " ")
if len(parts) != 2 {
c.AbortWithStatus(401)
return
}
ctx := context.Background()
kc.Logger.Infof("Incoming auth header: %s", parts[1])
token, err := kc.Auth.RetrospectToken(ctx, parts[1], kc.ClientId, kc.ClientSecret, kc.Realm)
if err != nil {
kc.Logger.Errorf("error occurred parsing token: %s", err)
}
kc.Logger.Infof("token: %v", token)
In order to test this, what I have been doing is starting up all the services, and then navigating to:
http://localhost:8080/auth/realms/dev/account/#/
and then I sign in with google (an account I have already set up on keycloak earlier).
Then, using the developer tools in the browser, im finding a network call that contains
KEYCLOAK_IDENTITY=eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiaSldUIiwia2lkIiA6ICIwMTg3MzU0MS03YmE3LTQ0MmQtYjMxOC0zMjBhYmIzYjNmMmIi<AND_SO_ON>; Version=1; Path=/auth/realms/dev/; SameSite=None; Secure; HttpOnly
From this i extract just the token, and then use that as a Bearer token in a postman call to my go back end.
The problem I am having is that, in step 1 when I log in my service account, if i retrospect the token returned there, I see the struct I'd expect with the oauth2 token values.
This middleware, when it retrospects the incoming tokens I give it, i just get
{
active: false
}
every time.
I am not entirely sure where I have gone wrong here. I have read a number of posts that say the host has to be the EXACT SAME between the acquire token, and the endpoint used to inspect it, which leads me to beleive that since I get a token from:
http://localhost:8080/auth/realms/dev/account/#/
but am decoding it on a host inside the container thats called:
http://keycloack:8080
That this may be my problem? Is this accurate? or have I gone wrong somewhere else?
If this IS in fact the problem, how can I go about solving this? I can't get a token from the internal docker name from the outside in order to test calls going in to my API, and I cannot use "localhost" inside the go container to match where i get it from as this wont resolve.
How can I either get a hostname i can access from both inside and outside the container?
Or, as mentioned, have I gone wrong somewhere else entirely?
Any help will be appreciated!
We wanted to list images and tags which names start with certain string. So far, we explored a few java lib (docker-java and spotify ones) and did quite amount of research, but still couldn't find a way out...
docker-java: 'com.github.docker-java', name: 'docker-java', version: '3.2.5'
The follow code lists images from public docker hub, not really the specified GCR. What's the right way to list image from our specified GCR?
DefaultDockerClientConfig config = DefaultDockerClientConfig
.createDefaultConfigBuilder()
.withRegistryUrl("http://eu.gcr.io/data-infrastructure-test-env")
.withDockerConfig("/home/me/.docker/config.json")
.build();
DockerClient dockerClient = DockerClientBuilder.getInstance(config).build();
List<SearchItem> items = dockerClient.searchImagesCmd("daas").exec();
List<String> images = new ArrayList<>();
for (SearchItem searchItem : items){
images.add(searchItem.getName());
}
Update - some progress
Inspired by this post: How to list images and tags from the gcr.io Docker Registry using the HTTP API?
I tried the following steps with my own google account, which has project owner (w/o firewall) permission:
gcloud auth login
gcloud auth print-access-token
define a function to get string for basic auth:
private String basicAuth(String username, String password) { return "Basic " + Base64.getEncoder().encodeToString((username + ":" + password).getBytes()); }
4, try the following code:
HttpRequest request = HttpRequest.newBuilder().uri(URI.create("https://gcr.io/v2/token?service=eu.gcr.io&scope=registry:my_gcp_project:*"))
.headers("Accept", "application/json"
, "Authorization",basicAuth("_token"
,"the_token_got_from_step_2"))
.GET()
.build(); UncheckedObjectMapper objectMapper = new UncheckedObjectMapper(); Map<String, String> response = HttpClient.newHttpClient()
.sendAsync(request, HttpResponse.BodyHandlers.ofString())
.thenApply(HttpResponse::body).thenApply(objectMapper::readValue)
.get();
String token = response.get("token");
request = HttpRequest.newBuilder().uri(URI.create("https://eu.gcr.io/v2/my_gcp_project/my_image/tags/list"))
.header("Authorization","Bearer " + token)
.GET().build(); String response2 = HttpClient.newHttpClient()
.sendAsync(request, HttpResponse.BodyHandlers.ofString())
.thenApply(HttpResponse::body)
.get();
However, the response2 I got was:
{"errors":[{"code":"UNAUTHORIZED","message":"Requested repository does not match bearer token resource: data-infrastructure-test-env/daas-master"}]}
Could you help to check what went wrong?
Docker engine API documentation clearly states that the ImageSearch command returns images from the Docker Hub registry: https://docs.docker.com/engine/api/v1.40/#operation/ImageSearch
For searching a GCR registry, you should rather use the Docker registry API.
Finally got it work!
I only need to change the second request uri to be: "https://eu.gcr.io/v2/my_gcp_project/tags/list" instead of "https://eu.gcr.io/v2/my_gcp_project/my_image/tags/list"
and I got some meaningful response back
This is an issue I've been trying to figure out for a few days now.
Service-to-service communication always results in 401 for my Cloud Run applications.
For token retrieval in service I've been using this code snippet:
tokenURL := fmt.Sprintf("/instance/service-accounts/default/identity?audience=%s", c.serviceURL)
var err error
token, err = metadata.Get(tokenURL)
if err != nil {
return nil, fmt.Errorf("metadata.Get: failed to query id_token: %+v", err)
}
I have also tried to supply a Service account key to my Cloud Run service and using the token that is returned from google.JWTAccessTokenSourceFromJSON to no avail.
The problem is that the JWT that is returned does work when used from cURL, but does not work when used in a service that runs on Cloud Run.
Both the Cloud Run service account and the external service account (for the key) has the roles/run.invoker IAM binding. And the audience is the Cloud Run issued service URL.
I have tried the following:
Using the metadata service for token retrieval (401 inside Cloud Run, but works when the token is used in cURL and Postman)
Using the JWTAccessTokenSourceFromJSON for token retrieval (401 inside Cloud Run, but works when the token is used in cURL and Postman)
Tried adding a delay before using issued token (in case the token is not propagated)
Tried manually pinging the Cloud Run service a few times with given token after the delay (using http.Client)
Nothing seems to work. Every request when run inside Cloud Run is returning 401 and the logs show The request was not authorized to invoke this service. Read more at https://cloud.google.com/run/docs/securing/authenticating. While using cURL or Postman I am able to access the service.
I've tried these methods to invoke the service:
tokenURL := fmt.Sprintf("/instance/service-accounts/default/identity?audience=%s", c.serviceURL)
var err error
token, err := metadata.Get(tokenURL)
if err != nil {
return nil, fmt.Errorf("metadata.Get: failed to query id_token: %+v", err)
}
// First one (ping)
req, _ := http.NewRequest("GET", "{{serviceURL}}", nil)
req.Header.Set("Authorization", "Bearer: "+token)
l, _ := cr.Do(req)
m, _ := ioutil.ReadAll(l.Body)
logrus.Println(l.Header)
logrus.Println(string(m))
// RoundTripper
r.Header.Set("Authorization", "Bearer: "+token)
return c.r.RoundTrip(r)
I appreciate any answer.
Thanks!
While posting code examples I just realized I added an extra colon(:) after Bearer which makes this request fail. Can't believe I spent days to solve this!
I am writing some code to try to get a token to use from Google in OAuth2. This is for a service account, so the instructions are here:
https://developers.google.com/identity/protocols/OAuth2ServiceAccount
I keep getting this error when I post the JWT to Google:
{ "error": "invalid_grant", "error_description": "Invalid JWT Signature." }
Here is the code:
try{
var nowInSeconds : Number = (Date.now() / 1000);
nowInSeconds = Math.round(nowInSeconds);
var fiftyNineMinutesFromNowInSeconds : Number = nowInSeconds + (59 * 60);
var claimSet : Object = {};
claimSet.iss = "{{RemovedForPrivacy}}";
claimSet.scope = "https://www.googleapis.com/auth/plus.business.manage";
claimSet.aud = "https://www.googleapis.com/oauth2/v4/token";
claimSet.iat = nowInSeconds;
claimSet.exp = fiftyNineMinutesFromNowInSeconds;
var header : Object = {};
header.alg = "RS256";
header.typ = "JWT";
/* Stringify These */
var claimSetString = JSON.stringify(claimSet);
var headerString = JSON.stringify(header);
/* Base64 Encode These */
var claimSetBaseSixtyFour = StringUtils.encodeBase64(claimSetString);
var headerBaseSixtyFour = StringUtils.encodeBase64(headerString);
var privateKey = "{{RemovedForPrivacy}}";
/* Create the signature */
var signature : Signature = Signature();
signature = signature.sign(headerBaseSixtyFour + "." + claimSetBaseSixtyFour, privateKey , "SHA256withRSA");
/* Concatenate the whole JWT */
var JWT = headerBaseSixtyFour + "." + claimSetBaseSixtyFour + "." + signature;
/* Set Grant Type */
var grantType = "urn:ietf:params:oauth:grant-type:jwt-bearer"
/* Create and encode the body of the token post request */
var assertions : String = "grant_type=" + dw.crypto.Encoding.toURI(grantType) + "&assertion=" + dw.crypto.Encoding.toURI(JWT);
/* Connect to Google And Ask for Token */
/* TODO Upload Certs? */
var httpClient : HTTPClient = new HTTPClient();
httpClient.setRequestHeader("content-type", "application/x-www-form-urlencoded; charset=utf-8");
httpClient.timeout = 30000;
httpClient.open('POST', "https://www.googleapis.com/oauth2/v4/token");
httpClient.send(assertions);
if (httpClient.statusCode == 200) {
//nothing
} else {
pdict.errorMessage = httpClient.errorText;
}
}
catch(e){
Logger.error("The error with the OAuth Token Generator is --> " + e);
}
Does anyone know why the JWT is failing?
Thanks so much!
Brad
The problem might be related to the fact that your StringUtils.encodeBase64() method is likely to perform a standard base64 encoding.
According to the JWT spec, however, it's not the standard base64 encoding that needs to be used, but the the URL- and filename-safe Base64 encoding, with the = padding characters omitted.
If you don't have a utility method handy for base64URL encoding, you can verify by
replacing all + with -;
replacing all / with _;
removing all =
in your base64-encoded strings.
Also, is your signature also base64-encoded? It needs to be, following the same rules as described above.
I had the same problem before and this is what was wrong:
wrong application name (project ID)
wrong service account ID (email)
The another reason for this error could be "Your service account is not activated", With gsutil installed from the Cloud SDK, you should authenticate with service account credentials.
1- Use an existing service account or create a new one, and download the associated private key.
2- Use gcloud auth activate-service-account to authenticate with the service account:
gcloud auth activate-service-account --key-file [KEY_FILE]
Where [KEY_FILE] is the name of the file that contains your service account credentials.
Link for more detail: Activate service account
This could also happen if a developer mistakenly copies, edits, and uses a service account key file for a purpose other than the one for which the file was originally intended. For example:
Developer A creates SA 1
Developer A uses gcloud iam service-accounts keys create ... to create the secret file for SA 1, encrypts it, and checks it in to source control
Developer B creates SA 2
Developer B (mistakenly) decrypts and copies the secret file from step 2, modifies some of its fields with data from SA 2, then attempts to use it in an application
The resolution in this scenario obviously is for Developer B to get rid of the copied/edited file and create a new secret file with gcloud like Developer A did in step 2.
I had this same error occur when using a service account. I couldn't figure out what was wrong so I came back to it the next day and it worked. So maybe Google Cloud takes some time to propagate every once in a while.
Here's the story:
I use Pentaho Kettle for querying web stats from Google Analytics via Simple API Access. For this request I need the API key. Now this API key turns invalid every now and then (I am not sure what the rythm is) and then the requests fail of course. So I want to generate a new one, receive it and make it available to the ETL-job and its GA steps.
My plan is to do this via Java embedded in one or more "User Defined Java Class" steps via google-api-java-client and a Service Account. The Kettle job generates a new API key, receives it and provides the API key via a field or directly as a paramter.
But primarily I am interested in a general Java solution for the described use case. If somebody has a Kettle-solution that would be even better but I mention those details chiefly to put the question into a context.
Question: How do I generate a new API key for Google Analytics Simple API Access and receive it using google-api-java-client via OAuth2 and without user-interaction (fully automated)?
Bounty: I would appreciate an answer that either provides a Java program or a detailed sequence of API calls. The output of the program is a functioning API key appliable for Simple API Access. Given the expected work involved I chose the highest possible bounty.
I registered a Service Account so I have the following IDs & Co. available:
Client ID
123.apps.googleusercontent.com
E-Mail address
123#developer.gserviceaccount.com
Public key fingerprints
abcxxx
client_secrets.json
{...}
private key
abcxxx-privatekey.p12
client_secrets.json:
{"web":{
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"client_email": "123#developer.gserviceaccount.com",
"client_x509_cert_url":"https://www.../123#developer.gserviceaccount.com",
"client_id": "123.apps.googleusercontent.com",
"auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs"
}}
A couple of things before the code:
Technically if you're using OAuth 2.0 for authentication then Simple API access doesn't apply at all. So you don't need to worry about the API Key. The instructions from Pentaho are old and don't apply unless you're using a non-OAuth 2.0 authentication mechanism (Which isn't what you should be doing)
What you mostly like want to use is a Service Account for authentication which does not require any human interaction.
Register the App
If you haven't already registered your application with the Google Cloud Console, then set up a project and application in the Cloud Console. The system guides you through the process of choosing or creating a project and registering a new application, and it automatically activates the API for you.
If you've already registered your application with the Cloud Console, then follow this procedure instead:
Go to the Google Cloud Console.
Select a project.
In the sidebar on the left, select APIs & auth. In the displayed list of APIs, make sure the Analytics API status is set to ON.
In the sidebar on the left, select Registered apps.
Select an application.
In either case, you end up on the application's credentials page.
To set up a service account, expand the Certificate section. Then select Generate Certificate. (If you already have a certificate, you can add a new key by selecting Generate New Key.) A dialog box appears; select Download private key to proceed. When it's finished downloading, select View public key.
After downloading the file and closing the dialog, you will be able to get the Service Account's email address.
Download the Java Client Library for Analytics
Visit https://developers.google.com/api-client-library/java/apis/analytics/v3
Go to the Add Library to Your Project section at the bottom and download the Google Analytics API v3 Client Library for Java.
Add Service Account Email to GA Account
Add the service account email to the Google Analytics account(s) you want to access through the using the API.
Sample Java App to Get # of Visits
The following code snippet is for the Core Reporting API. It authenticates using the service account. It then retrieves the visits for a Analytics profile and outputs the number.
To use this then you should obviously replace the Service account email and the path to the private key file. Also you need to import the Java client for analytics and the relevant JARS from the libs folder.
Also refer to the Google Analytics Core Reporting API docs to learn to query the API with dimensions and metrics, segments, filters, etc.
You should be able to customize this for use with your application.
import java.io.IOException;
import java.security.GeneralSecurityException;
import java.util.HashSet;
import java.util.Set;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.analytics.Analytics;
import com.google.api.services.analytics.AnalyticsScopes;
import com.google.api.services.analytics.model.GaData;
public class AnalyticsSample {
private static final String APPLICATION_NAME = "APP_NAME";
private static final String SERVICE_ACCOUNT_EMAIL = "<YOUR_SERVICE_ACCOUNT_EMAIL>#developer.gserviceaccount.com";
/** Path to the Service Account's Private Key file */
private static final String SERVICE_ACCOUNT_PKCS12_FILE_PATH = "/path/to/-privatekey.p12";
private static Analytics service;
public static Analytics getAnalyticsService() throws GeneralSecurityException, IOException {
Set<String> scopes = new HashSet<String>();
scopes.add(AnalyticsScopes.ANALYTICS_READONLY); // You can set other scopes if needed
HttpTransport httpTransport = new NetHttpTransport();
JacksonFactory jsonFactory = new JacksonFactory();
GoogleCredential credential = new GoogleCredential.Builder()
.setTransport(httpTransport)
.setJsonFactory(jsonFactory)
.setServiceAccountId(SERVICE_ACCOUNT_EMAIL)
.setServiceAccountScopes(scopes)
.setServiceAccountPrivateKeyFromP12File(
new java.io.File(SERVICE_ACCOUNT_PKCS12_FILE_PATH))
.build();
Analytics service = new Analytics.Builder(httpTransport, jsonFactory, null)
.setHttpRequestInitializer(credential)
.setApplicationName(APPLICATION_NAME)
.build();
return service;
}
public static void main(String[] args) {
try {
service = getAnalyticsService();
GaData result = service.data().ga().get(
"ga:<YOUR PROFILE ID>",
"2013-10-16", // Start date
"2013-10-17", // End Date
"ga:visits").execute(); // Add Dimensions and metrics
System.out.println(result.getRows().get(0).get(0)); // Just an example of output
} catch (IOException e) {
System.err.println(e.getMessage());
} catch (GeneralSecurityException e) {
System.err.println(e.getMessage());
} catch (Throwable t) {
t.printStackTrace();
}
System.exit(1);
}
}
i'm not a Java guy, but i am doing pretty much exactly what you're asking for using Go and the Google API. i can give you an overview of the process, hopefully this helps/sends you in the right direction? :)
overview:
download the certificate from your registered app in the google cloud console
convert the certificate from PCS12 encoding into PEM
craft a JWT (JSON web token) using that key, along with the email address (the foobarbazxxxxetc#developer.gserviceaccount.com one) and the API scope that you want to use)
add the same email address with appropriate permissions to the analytics account you want to use
use the JWT to get an oauth token
create an analytics service, authenticating using the oauth token
now you can talk to core reporting API
EDIT: no key regeneration neeeded. the token lasts 1 hour, and cannot be refreshed. but that doesn't matter because the next time you authenticate using the key, you'll get a new token lasting an hour.
EDIT 2: i'm including clean compiling Go code with docs/pre-reqs/comments. hope this helps a bit more!
package main
// overview of crqs.go:
// this example app and supporting documentation enables you to query analytics data via the Core Reporting API in a completely headless manner, with
// no "popup" or user intervention required. it doesn't "handle" the returned data very well, as that will depend on what metrics/dimensions your
// query has. better handling of this is left as an exercise for the reader. :) follow the pre-reqs, change the ga* consts specific to your own app,
// and you should be good to go. uses google-api-go-client, analytics/v3, oauth, and oauth/jwt which you can download and install from
// https://code.google.com/p/goauth2/ and http://code.google.com/p/google-api-go-client/
// docs/pre-reqs:
// 1. create a Project within the Google Cloud Console (https://cloud.google.com/console#/project)
// 2. within that Project, ensure you've turned on the Analytics API (APIs & Auth --> Analytics --> On)
// 3. within that Project, create an application of type "Web Application" (APIs & Auth --> Registered Apps)
// 4. click on the application name. expand the "Certificate" section. you will see that the email/public keys
// have not been generated. click "Generate". download the private key (.p12 file). also, note the private key password, and
// the email address, you will need these later. you will set the string const gaServiceAcctEmail to this email address.
// 5. download the JSON file, too. you will set the const gaServiceAcctSecretsFile to the location of this file.
// 6. now we need to convert the .p12 key to a PEM key. in a unix shell:
// $ openssl pkcs12 -in privatekeyfilename.p12 -nodes -nocerts > privatekeyfilename.pem
// it will ask you for the Import password. enter the private key password you were given in step 4. you should see
// "MAC verified OK", and you will have the PEM key. you will set the const gaServiceAcctPEMKey to the location of this file.
// 7. goto Google Analytics. in Admin section specific to the GA account/property you wish to query, give Read & Analyze permissions to the
// email address you were given in step 4.
// 8. now we need the Table ID of the GA account/property mentioned in step 7. login to the Google Analytics Query Explorer 2 at
// http://ga-dev-tools.appspot.com/explorer/ then select the appropriate account/property/profile in the dropdown menus. you will see
// the Table ID in the "*ids" box. it looks like "ga:1234556". you will set the string const gaTableID to this value.
// 9. that should be all you need to do! compile and "go".
// example runs:
// 1. shows total pageview count for all URLs (starting at default date october 1st through today).
// $ ./crqs
// gaStartDate=2013-10-01, gaEndDate=2013-10-18
// gaMetrics=ga:pageviews
// gaFilter=ga:pagePath=~^/
// len(gaData.Rows)=1, TotalResults=1
// row=0 [25020179]
//
// 2. shows unique pageview count per URL for top 5 URLs starting in "/movies" (starting at default date october 1st through today),
// in descending order.
// $ ./crqs -m=ga:uniquePageviews -d=ga:pagePath -s=-ga:uniquePageviews -f=ga:pagePath=~^/movies -x=5
// gaStartDate=2013-10-01, gaEndDate=2013-10-18
// gaMetrics=ga:uniquePageviews
// gaDimensions=ga:dimensions=ga:pagePath
// gaSortOrder=ga:sort=-ga:uniquePageviews
// gaFilter=ga:pagePath=~^/movie
// len(gaData.Rows)=5, TotalResults=10553
// row=0 [/movies/foo 1005697]
// row=1 [/movies/bar 372121]
// row=2 [/movies/baz 312901]
// row=3 [/movies/qux 248761]
// row=4 [/movies/xyzzy 227286]
//
// 3. shows unique pageview count per URL for top 5 URLs from Aug 1 --> Sep 1, in descending order.
// $ ./crqs -sd=2013-08-01 -ed=2013-09-01 -m=ga:uniquePageviews -d=ga:pagePath -s=-ga:uniquePageviews -x=5
// gaStartDate=2013-08-01, gaEndDate=2013-09-01
// gaMetrics=ga:uniquePageviews
// gaDimensions=ga:dimensions=ga:pagePath
// gaSortOrder=ga:sort=-ga:uniquePageviews
// len(gaData.Rows)=5, TotalResults=159023
// row=0 [/ 4280168]
// row=1 [/foo 2504679]
// row=2 [/bar 1177822]
// row=3 [/baz 755705]
// row=4 [/xyzzy 739513]
import (
"code.google.com/p/goauth2/oauth"
"code.google.com/p/goauth2/oauth/jwt"
"code.google.com/p/google-api-go-client/analytics/v3"
"encoding/json"
"flag"
"fmt"
"io/ioutil"
"log"
"net/http"
"os"
"time"
)
// constants
const (
// don't change these
dateLayout string = "2006-01-02" // date format that Core Reporting API requires
// change these! add in your own app and analytics-property-specific values
gaServiceAcctEmail string = "your-applications-email-address#developer.gserviceaccount.com" // email address of registered application
gaServiceAcctSecretsFile string = "/path/to/client_secret_your-application.apps.googleusercontent.com.json" // registered application JSON file downloaded from Google Cloud Console
gaServiceAcctPEMKey string = "/path/to/your-applications-converted-privatekey.pem" // private key (PEM format) of registered application
gaScope string = "https://www.googleapis.com/auth/analytics.readonly" // oauth 2 scope information for Core Reporting API
gaTableID string = "ga:12345678" // namespaced profile ID of the analytics account/property/profile from which to request data
)
// globals
var (
// vars for the runtime flags
gaDimensions string
gaEndDate string
gaFilter string
gaMetrics string
gaMaxResults int64
gaSortOrder string
help bool
gaStartDate string
)
// types
// struct to read the registered application's JSON secretsfile into
type GAServiceAcctSecretsConfig struct {
ClientEmail string `json:"client_email"`
ClientId string `json:"client_id"`
ClientSecret string `json:"client_secret"`
RedirectURIs []string `json:"redirect_uris"`
Scope string
AuthURI string `json:"auth_uri"`
TokenURI string `json:"token_uri"`
}
// func: init()
// initialisation function for the command line flags/options.
func init() {
flag.BoolVar(&help, "h", false, "Show all command line arguments.")
flag.StringVar(&gaDimensions, "d", "", "GA query dimensions")
flag.StringVar(&gaEndDate, "ed", "", "GA query end date")
flag.StringVar(&gaFilter, "f", "", "GA filter to apply")
flag.Int64Var(&gaMaxResults, "x", 10000, "GA maximum # of results to return (default: 10000)")
flag.StringVar(&gaMetrics, "m", "ga:pageviews", "GA metrics you want to query (default: ga:pageviews)")
flag.StringVar(&gaSortOrder, "s", "", "GA query reply sort order")
flag.StringVar(&gaStartDate, "sd", "2013-10-01", "GA query start date (default: 2013-10-01)")
}
// func: readConfig()
// reads in the registered application's JSON secretsfile and crafts an oauth.Config which
// it returns to the calling func. exits hard if either the JSON secretsfile load fails,
// or if the unmarshal fails.
func readGAServiceAcctSecretsConfig() *oauth.Config {
configfile := new(GAServiceAcctSecretsConfig)
data, err := ioutil.ReadFile(gaServiceAcctSecretsFile)
if err != nil {
log.Fatal("error reading GA Service Account secret JSON file -", err)
}
err = json.Unmarshal(data, &configfile)
if err != nil {
log.Fatal("error unmarshalling GA Service Account secret JSON file -", err)
}
return &oauth.Config{
ClientId: configfile.ClientId,
ClientSecret: configfile.ClientSecret,
Scope: gaScope,
AuthURL: configfile.AuthURI,
TokenURL: configfile.TokenURI,
}
} //
// func: main()
// the main function. duh.
func main() {
// grok the command line args
flag.Usage = usage
flag.Parse()
if help {
flag.Usage()
}
// read in the registered application's JSON secretsfile and get an oauth.Config in return.
oauthConfig := readGAServiceAcctSecretsConfig()
// read in the registered applications private PEM key.
pemKey, err := ioutil.ReadFile(gaServiceAcctPEMKey)
if err != nil {
log.Fatal("error reading GA Service Account PEM key -", err)
}
// create a JWT (JSON Web Token).
jsonWebToken := jwt.NewToken(gaServiceAcctEmail, oauthConfig.Scope, pemKey)
// form the JWT claim set.
jsonWebToken.ClaimSet.Aud = oauthConfig.TokenURL
// create a basic httpclient. we will use this to send the JWT.
httpClient := &http.Client{}
// build the request, encode it, and then send the JWT. we get a nifty oauth.Token in return.
oauthToken, err := jsonWebToken.Assert(httpClient)
if err != nil {
log.Fatal("error asserting JSON Web Token -", err)
}
// build the oauth transport
oauthTransport := oauth.Transport{Config: oauthConfig}
// set the oauthTransport's token to be the oauthToken
oauthTransport.Token = oauthToken
// create a new analytics service by passing in the httpclient returned from Client() that can now make oauth-authenticated requests
analyticsService, err := analytics.New(oauthTransport.Client())
if err != nil {
log.Fatal("error creating Analytics Service -", err)
}
// create a new analytics data service by passing in the analytics service we just created
dataGaService := analytics.NewDataGaService(analyticsService)
// w00t! now we're talking to the core reporting API. hard stuff over, let's do a simple query.
// if no gaEndDate specified via command line args, set it to today's date.
if gaEndDate == "" {
t := time.Now()
gaEndDate = t.Format(dateLayout)
}
// print the query start & end dates.
fmt.Printf("gaStartDate=%s, gaEndDate=%s\n", gaStartDate, gaEndDate)
fmt.Printf("gaMetrics=%s\n", gaMetrics)
// setup the query
dataGaGetCall := dataGaService.Get(gaTableID, gaStartDate, gaEndDate, gaMetrics)
// setup the dimensions (if any).
if gaDimensions != "" {
dimensions := fmt.Sprintf("ga:dimensions=%s", gaDimensions)
fmt.Printf("gaDimensions=%s\n", dimensions)
dataGaGetCall.Dimensions(gaDimensions)
}
// setup the sort order (if any).
if gaSortOrder != "" {
sortorder := fmt.Sprintf("ga:sort=%s", gaSortOrder)
fmt.Printf("gaSortOrder=%s\n", sortorder)
dataGaGetCall.Sort(gaSortOrder)
}
// setup the filter (if any).
if gaFilter != "" {
filter := fmt.Sprintf("%s", gaFilter)
fmt.Printf("gaFilter=%s\n", filter)
dataGaGetCall.Filters(filter)
}
// setup the max results we want returned.
dataGaGetCall.MaxResults(gaMaxResults)
// send the query to the API, get gaData back.
gaData, err := dataGaGetCall.Do()
if err != nil {
log.Fatal("API error -", err)
}
// how much data did we get back?
fmt.Printf("len(gaData.Rows)=%d, TotalResults=%d\n", len(gaData.Rows), gaData.TotalResults)
// a lazy loop through the returned rows
for row := 0; row <= len(gaData.Rows)-1; row++ {
fmt.Printf("row=%d %v\n", row, gaData.Rows[row])
}
}
// func: usage()
// prints out all possible flags/options when "-h" or an unknown option is used at runtime.
// exits back to shell when complete.
func usage() {
fmt.Printf("usage: %s [OPTION] \n", os.Args[0])
flag.PrintDefaults()
os.Exit(2)
}