We wanted to list images and tags which names start with certain string. So far, we explored a few java lib (docker-java and spotify ones) and did quite amount of research, but still couldn't find a way out...
docker-java: 'com.github.docker-java', name: 'docker-java', version: '3.2.5'
The follow code lists images from public docker hub, not really the specified GCR. What's the right way to list image from our specified GCR?
DefaultDockerClientConfig config = DefaultDockerClientConfig
.createDefaultConfigBuilder()
.withRegistryUrl("http://eu.gcr.io/data-infrastructure-test-env")
.withDockerConfig("/home/me/.docker/config.json")
.build();
DockerClient dockerClient = DockerClientBuilder.getInstance(config).build();
List<SearchItem> items = dockerClient.searchImagesCmd("daas").exec();
List<String> images = new ArrayList<>();
for (SearchItem searchItem : items){
images.add(searchItem.getName());
}
Update - some progress
Inspired by this post: How to list images and tags from the gcr.io Docker Registry using the HTTP API?
I tried the following steps with my own google account, which has project owner (w/o firewall) permission:
gcloud auth login
gcloud auth print-access-token
define a function to get string for basic auth:
private String basicAuth(String username, String password) { return "Basic " + Base64.getEncoder().encodeToString((username + ":" + password).getBytes()); }
4, try the following code:
HttpRequest request = HttpRequest.newBuilder().uri(URI.create("https://gcr.io/v2/token?service=eu.gcr.io&scope=registry:my_gcp_project:*"))
.headers("Accept", "application/json"
, "Authorization",basicAuth("_token"
,"the_token_got_from_step_2"))
.GET()
.build(); UncheckedObjectMapper objectMapper = new UncheckedObjectMapper(); Map<String, String> response = HttpClient.newHttpClient()
.sendAsync(request, HttpResponse.BodyHandlers.ofString())
.thenApply(HttpResponse::body).thenApply(objectMapper::readValue)
.get();
String token = response.get("token");
request = HttpRequest.newBuilder().uri(URI.create("https://eu.gcr.io/v2/my_gcp_project/my_image/tags/list"))
.header("Authorization","Bearer " + token)
.GET().build(); String response2 = HttpClient.newHttpClient()
.sendAsync(request, HttpResponse.BodyHandlers.ofString())
.thenApply(HttpResponse::body)
.get();
However, the response2 I got was:
{"errors":[{"code":"UNAUTHORIZED","message":"Requested repository does not match bearer token resource: data-infrastructure-test-env/daas-master"}]}
Could you help to check what went wrong?
Docker engine API documentation clearly states that the ImageSearch command returns images from the Docker Hub registry: https://docs.docker.com/engine/api/v1.40/#operation/ImageSearch
For searching a GCR registry, you should rather use the Docker registry API.
Finally got it work!
I only need to change the second request uri to be: "https://eu.gcr.io/v2/my_gcp_project/tags/list" instead of "https://eu.gcr.io/v2/my_gcp_project/my_image/tags/list"
and I got some meaningful response back
Related
I have a running cloud run service user-service. For test purposes I passed client secrets via environment variables as plain text. Now since everything is working fine I'd like to use a secret instead.
In the "Variables" tab of the "Edit Revision" option I can declare environment variables but I have no idea how to pass in a secret? Do I just need to pass the secret name like ${my-secret-id} in the value field of the variable? There is not documentation on how to use secrets in this tab only a hint at the top:
Store and consume secrets using Secret Manager
Which is not very helpful in this case.
You can now read secrets from Secret Manager as environment variables in Cloud Run. This means you can audit your secrets, set permissions per secret, version secrets, etc, and your code doesn't have to change.
You can point to the secrets through the Cloud Console GUI (console.cloud.google.com) or make the configuration when you deploy your Cloud Run service from the command-line:
gcloud beta run deploy SERVICE --image IMAGE_URL --update-secrets=ENV_VAR_NAME=SECRET_NAME:VERSION
Six-minute video overview: https://youtu.be/JIE89dneaGo
Detailed docs: https://cloud.google.com/run/docs/configuring/secrets
UPDATE 2021: There is now a Cloud Run preview for loading secrets to an environment variable or a volume. https://cloud.google.com/run/docs/configuring/secrets
The question is now answered however I have been experiencing a similar problem using Cloud Run with Java & Quarkus and a native image created using GraalVM.
While Cloud Run is a really interesting technology at the time of writing it lacks the ability to load secrets through the Cloud Run configuration. This has certainly added complexity in my app when doing local development.
Additionally Google's documentation is really quite poor. The quick-start lacks a clear Java example for getting a secret[1] without it being set in the same method - I'd expect this to have been the most common use case!
The javadoc itself seems to be largely autogenerated with protobuf language everywhere. There are various similarly named methods like getSecret, getSecretVersion and accessSecretVersion
I'd really like to see some improvment from Google around this. I don't think it is asking too much for dedicated teams to make libraries for common languages with proper documentation.
Here is a snippet that I'm using to load this information. It requires the GCP Secret library and also the GCP Cloud Core library for loading the project ID.
public String getSecret(final String secretName) {
LOGGER.info("Going to load secret {}", secretName);
// SecretManagerServiceClient should be closed after request
try (SecretManagerServiceClient client = buildClient()) {
// Latest is an alias to the latest version of a secret
final SecretVersionName name = SecretVersionName.of(getProjectId(), secretName, "latest");
return client.accessSecretVersion(name).getPayload().getData().toStringUtf8();
}
}
private String getProjectId() {
if (projectId == null) {
projectId = ServiceOptions.getDefaultProjectId();
}
return projectId;
}
private SecretManagerServiceClient buildClient() {
try {
return SecretManagerServiceClient.create();
} catch(final IOException e) {
throw new RuntimeException(e);
}
}
[1] - https://cloud.google.com/secret-manager/docs/reference/libraries
Google have documentation for the Secret manager client libraries that you can use in your api.
This should help you do what you want
https://cloud.google.com/secret-manager/docs/reference/libraries
Since you haven't specified a language I have a nodejs example of how to access the latest version of your secret using your project id and secret name. The reason I add this is because the documentation is not clear on the string you need to provide as the name.
const [version] = await this.secretClient.accessSecretVersion({
name: `projects/${process.env.project_id}/secrets/${secretName}/versions/latest`,
});
return version.payload.data.toString()
Be sure to allow secret manager access in your IAM settings for the service account that your api uses within GCP.
I kinda found a way to use secrets as environment variables.
The following doc (https://cloud.google.com/sdk/gcloud/reference/run/deploy) states:
Specify secrets to mount or provide as environment variables. Keys
starting with a forward slash '/' are mount paths. All other keys
correspond to environment variables. The values associated with each
of these should be in the form SECRET_NAME:KEY_IN_SECRET; you may omit
the key within the secret to specify a mount of all keys within the
secret. For example:
'--update-secrets=/my/path=mysecret,ENV=othersecret:key.json' will
create a volume with secret 'mysecret' and mount that volume at
'/my/path'. Because no secret key was specified, all keys in
'mysecret' will be included. An environment variable named ENV will
also be created whose value is the value of 'key.json' in
'othersecret'. At most one of these may be specified
Here is a snippet of Java code to get all secrets of your Cloud Run project. It requires the com.google.cloud/google-cloud-secretmanager artifact.
Map<String, String> secrets = new HashMap<>();
String projectId;
String url = "http://metadata.google.internal/computeMetadata/v1/project/project-id";
HttpURLConnection conn = (HttpURLConnection)(new URL(url).openConnection());
conn.setRequestProperty("Metadata-Flavor", "Google");
try {
InputStream in = conn.getInputStream();
projectId = new String(in.readAllBytes(), StandardCharsets.UTF_8);
} finally {
conn.disconnect();
}
Set<String> names = new HashSet<>();
try (SecretManagerServiceClient client = SecretManagerServiceClient.create()) {
ProjectName projectName = ProjectName.of(projectId);
ListSecretsPagedResponse pagedResponse = client.listSecrets(projectName);
pagedResponse
.iterateAll()
.forEach(secret -> { names.add(secret.getName()); });
for (String secretName : names) {
String name = secretName.substring(secretName.lastIndexOf("/") + 1);
SecretVersionName nameParam = SecretVersionName.of(projectId, name, "latest");
String secretValue = client.accessSecretVersion(nameParam).getPayload().getData().toStringUtf8();
secrets.put(secretName, secretValue);
}
}
Cloud Run support for referencing Secret Manager Secrets is now at general availability (GA).
https://cloud.google.com/run/docs/release-notes#November_09_2021
I've added all my configuration details in the Vault. The detail you can see in the attached image below. This follows a specific path i.e kv/unistad/dev/workflow/camunda/1.0
However, when I try to read this information using Vault.NET with the following nuget package
Install-Package Vault
My code looks something like this:
var endpoint = "http://openblue-bridge.com:32270";
var token = "s.inklpUdNxet1ZJtaCLMpEIPA";
var vaultClient = new VaultClient(new Uri(endpoint), token);
string project = "unistad";
string environment = "dev";
string appVersion = "1.0";
var secretPath = $"kv/{project}/{environment}/workflow/camunda/{appVersion}";
// Use client to read a key-value secret.
var secrets = await vaultClient.Secret.Read< Dictionary<string, string>> (secretPath);
When I run the above code I get the following error:
Invalid path for a versioned K/V secrets engine. See the API docs for
the appropriate API endpoints to use. If using the Vault CLI, use
'vault kv get' for this operation.
I'm not sure how can I fix this error. Any help would be really appreciated.
You are using v2 of the kv engine. For that engine, you need to have /data/ in the path, as shown in the API docs. The requirement for this prefix is also described in the engine docs.
So the solution to your problem is specifically to change your path from
var secretPath = $"kv/{project}/{environment}/workflow/camunda/{appVersion}";
to
var secretPath = $"kv/data/{project}/{environment}/workflow/camunda/{appVersion}";
I create a Cloud Run client, however, couldn't find a way to list a service that is deployed with Cloud Run on GKE (for Anthos).
Create the client:
HttpTransport httpTransport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
GoogleCredentials credential = GoogleCredentials.getApplicationDefault();
credential.createScoped("https://www.googleapis.com/auth/cloud-platform");
HttpRequestInitializer requestInitializer = new HttpCredentialsAdapter(credential);
CloudRun.Builder builder = new CloudRun.Builder(httpTransport, jsonFactory, requestInitializer);
return builder.setApplicationName(applicationName)
.setRootUrl(cloudRunRootUrl)
.build();
} catch (IOException e) {
e.printStackTrace();
}
try to list services:
services = cloudRun.namespaces().services()
.list("namespaces/default")
.execute()
.getItems();
My "hello" service is deploy on a GKE cluster under the namespace default. The above code doesn't work because the client always see "default" as project_id and complains about permission stuff. If I put the project_id rather than "default", permission errors are gone, but no services will be found.
I tried another project that does have Google fully-managed cloud run services, the same code returns result (with .list("namespaces/")).
How to access the service on GKE?
And my next question would be, how to programmatically create Cloud Run services on GKE?
Edit - for creating a service
As I couldn't figure out how to interact with Cloud Run on GKE, I took a step back to try fully managed one. The following code to create a service fails, and the error message just doesn't provide much useful insight, how to make it work?
Service deployedService = null;
// Map<String,String> annotations = new HashMap<>();
// annotations.put("client.knative.dev/user-image","gcr.io/cloudrun/hello");
ServiceSpec spec = new ServiceSpec();
List<Container> containers = new ArrayList<>();
containers.add(new Container().setImage("gcr.io/cloudrun/hello"));
spec.setTemplate(new RevisionTemplate().setMetadata(new ObjectMeta().setName("hello-fully-managed-v0.1.0"))
.setSpec(new RevisionSpec().setContainerConcurrency(20)
.setContainers(containers)
.setTimeoutSeconds(100)
)
);
helloService.setApiVersion("serving.knative.dev/v1")
.setMetadata(new ObjectMeta().setName("hello-fully-managed")
.setNamespace("data-infrastructure-test-env")
// .setAnnotations(annotations)
)
.setSpec(spec)
.setKind("Service");
try {
deployedService = cloudRun.namespaces().services()
.create("namespaces/data-infrastructure-test-env",helloService)
.execute();
} catch (IOException e) {
e.printStackTrace();
response.add(e.toString());
return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body(response);
}
Error message I got:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "The request has errors",
"reason" : "badRequest"
} ],
"message" : "The request has errors",
"status" : "INVALID_ARGUMENT"
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
And the base_url is: https://europe-west1-run.googleapis.com
Your question is quite detailed (and is about Java which I am no expert in) and there are actually too many questions in there (ideally, please ask only 1 question here). However, I'll try to answer a few things you asked:
First, Cloud Run (managed, and on GKE) both implement the Knative Serving API. I've explained this at https://ahmet.im/blog/cloud-run-is-a-knative/ In fact, Cloud Run on GKE is just the open source Knative components installed to your cluster.
And my next question would be, how to programmatically create Cloud Run services on GKE?
You will have a very hard time (if possible at all) using the Cloud Run API client libraries (e.g. new CloudRun above) because these are designed for *.googleapis.com endpoints.
The Knative API part of "Cloud Run on GKE" is actually just your Kubernetes (GKE) master API endpoint (which runs on an IP address, with a TLS certificate that isn't trusted by root CAs, but you can find the CA cert in GKE GetCluster API call to verify the cert.) The TLS is part is why it's so hard to use the API Client libraries.
Knative APIs are just Kubernetes objects. So your best bet is one of these:
See Kubernetes java client (https://github.com/kubernetes-client/java) actually allows dynamic objects. (Go implementation does) and try to use that to create Knative CRDs.
Use kubectl apply.
Ask Knative Serving open source repository for help (they should be providing client libraries, maybe they're already there I'm not sure)
To program Cloud Run (managed) with the API Client Libraries, you need to explicitly override the API endpoint to the region e.g. us-central1-run.googleapis.com. (This is documented on each API call's REST API reference documentation.)
I have written a blog post in detail (with sample code in Go) on how to create/update services on Cloud Run (managed) using the Knative Serving API here: https://ahmet.im/blog/gcloud-run-deploy/
If you want to see how gcloud run deploy works, and which APIs it calls, you can pass --log-http option to observe the request/response traffic.
As for the error you got, it seems like the error message isn't helpful, but it might be coming from anywhere (as you're trying to imitate Knative API in GCP client libraries). I recommend reading my blog posts and sample code in depth.
UPDATES: Our engineering team's looking at the issue, it appears that there's currently a bug not adding the "details" field to the error. That's being worked on.
In your case, we see the following errors from requests:
field: "spec.template.spec"
description: "Missing template spec."
Means you are not properly filling up the spec field as I shown in my blog post and sample code.
field: "metadata.name"
description: "The revision name must be prefixed by the name of the enclosing Service or Configuration with a trailing -"
Make sure the name you are specifying adheres the patterns specified in API docs. Try to create that name manually perhaps in the UI or gcloud CLI.
field: "api_version"
description: "Unsupported API version \'serving.knative.dev/v1\'. Expected \'serving.knative.dev/v1alpha1\'"
Do not use v1alpha1 API, use v1 directly.
We'll try to get the details to the error message, however it appears that you need to study the sample code I linked in my blog post more in detail:
https://github.com/GoogleCloudPlatform/cloud-run-button/blob/a52c7fbaae33a3e06c112206c7227a0ef9649647/cmd/cloudshell_open/deploy.go#L26-L112
The Java SDK is automatically generated from the fact that the Cloud Run (fully managed) API is public. It does not support Cloud Run for Anthos.
(gcloud.run.deploy) The revision name must be prefixed by the name of the enclosing Service or Configuration with a trailing -revision name
revision name name should be 65 character then problem will be resolved in Automation pipeline with GCP revision suffix should be less revision name is the combination of (service name +revision suffix) will automatically created by GCP.
I'm working on a simple REST client for Docker Registry. For private registries, name resolution is pretty simple; if the image name is myregistry.io/myimage:latest, I look for https://myregistry.io/v2 and query the API there.
However, I notice that for docker hub, it doesn't quite work that way. If I'm looking for ubuntu, I can expand that to docker.io/ubuntu:latest, but https://docker.io/v2 returns a 307 redirect to https://www.docker.com/v2, which just returns HTML. The actual registry endpoint is at https://registry-1.docker.io/v2.
Is this just a hardcoded special case in the docker client, or is there some extra logic to looking up registry endpoints that I'm unaware of? If it is just a special case, is there more to it than always going to registry-1.docker.io instead of docker.io?
The central Docker registry is a well-known special case, similar to Maven central. You can see the defaults e.g. at https://github.com/docker/docker-ce/blob/ea449e9b10cebb259e1a43325587cd9a0e98d0ff/components/engine/registry/config.go#L42:
var (
// DefaultNamespace is the default namespace
DefaultNamespace = "docker.io"
// DefaultRegistryVersionHeader is the name of the default HTTP header
// that carries Registry version info
DefaultRegistryVersionHeader = "Docker-Distribution-Api-Version"
// IndexHostname is the index hostname
IndexHostname = "index.docker.io"
// IndexServer is used for user auth and image search
IndexServer = "https://" + IndexHostname + "/v1/"
// IndexName is the name of the index
IndexName = "docker.io"
// DefaultV2Registry is the URI of the default v2 registry
DefaultV2Registry = &url.URL{
Scheme: "https",
Host: "registry-1.docker.io",
}
)
I have found bitbucket api like:
https://bitbucket.org/api/2.0/repositories/{teamname}
But this link return 301 status (moved permanently to !api/2.0/repositories/{teamname}).
Ok, but this one returns status 200 with zero repositories.
I provide two parameters as user and password, but nothing seems changed.
So, can anybody answer how to get full list of private repositories that allowed to specific user?
Atlassian Documentation - Repositories Endpoint provides a detail documentation on how to access the repositories.
The URL mentioned in bitbucket to GET a list of repositories for an account is:
GET https://api.bitbucket.org/2.0/repositories/{owner}
If you use the above URL it always retrieves the repositories where you are the owner. In order to retrieve full list of repositories that the user is member of, you should call:
GET https://api.bitbucket.org/2.0/repositories?role=member
You can apply following set of filters for role based on your needs.
To limit the set of returned repositories, apply the
role=[owner|admin|contributor|member] parameter where the roles are:
owner: returns all repositories owned by the current user.
admin: returns repositories to which the user has explicit
administrator access.
contributor: returns repositories to which the user has explicit write access.
member: returns repositories to which the user has explicit read
access.
Edit-1:
You can make use of Bitbucket REST browser for testing the request/response.(discontinued)
You should not use the API from the https://bitbucket.org/api domain.
Instead, you should always use https://api.bitbucket.org.
Now one reason you might be getting an empty result after following the redirect could be because some http clients will only send Basic Auth credentials if the server explicitly asks for them by returning a 401 response with the WWW-Authenticate response header.
The repositories endpoint does not require authentication. It will simply return the repos that are visible to anonymous users (which might well be an empty set in your case) and so clients that insist on a WWW-Authenticate challenge (there are many, including Microsoft Powershell) will not work as expected (note, curl always sends Basic Auth credentials eagerly, which makes it a good tool for testing).
Unfortunately, from what I see in the documentation, there is no way to list all private repositories which the user has access to.
GET https://api.bitbucket.org/2.0/repositories
"Returns a paginated list of all public repositories." according to the doco.
GET https://api.bitbucket.org/2.0/repositories/{owner}
"Returns a paginated list of all repositories owned by the specified account or UUID." according to the doco.
So, getting all private repositories not necessarily owned by the user is either not possible, or I haven't found the right endpoint, or the documentation is inacurate.
None of the answers above worked for me, so this is what I did. We'll use the Bitbucket REST API.
Authentication
You can't use your normal credentials. I created an API Password. I'm not sure how to get to this page via your browser, but go here: https://bitbucket.org/account/settings/app-passwords/
Create an App Password, then cut and save the password that Atlassian generates for you.
Curl
curl --user your_username:your_app_password https://api.bitbucket.org/2.0/repositories/your_workspace?pagelen=100
I piped that to jq and saved it to a file.
your_workspace you get from looking at the URL of any of your repositories.
Paging
The maximum pagelen appears to be 100. If you have more than 100 repos, then you might have to do this:
curl --user your_username:your_app_password https://api.bitbucket.org/2.0/repositories/your_workspace?pagelen=100&page=2
The JSON
The JSON isn't too bad. You want the "values" array. From there, look at links.clone, which might have two entries like this:
"clone": [
{
"href": "https://user#bitbucket.org/WORKSPACE/REPO.git",
"name": "https"
},
{
"href": "git#bitbucket.org:WORKSPACE/REPO.git",
"name": "ssh"
}
],
That's a cut & paste from my results with personal info changed. Also useful are two other fields:
"full_name": "WORKSPACE/repo",
"name": "Repo",
Expanding on blizzard's answer, here's a little node.js script I just wrote:
import axios from 'axios';
import fs from 'fs';
async function main() {
const bitbucket = axios.create({
baseURL: 'https://api.bitbucket.org/2.0',
auth: {
username: process.env.BITBUCKET_USERNAME!,
password: process.env.BITBUCKET_PASSWORD!,
}
});
const repos = [];
let next = 'repositories?role=member';
for(;;) {
console.log(`Fetching ${next}`)
const res = await bitbucket.get(next);
if(res.status < 200 || res.status >= 300) {
console.error(res);
return 1;
}
repos.push(...res.data.values);
if(!res.data.next) break;
next = res.data.next;
}
console.log(`Done; writing file`);
await fs.promises.writeFile(`${__dirname}/../data/repos.json`,JSON.stringify(repos,null,2),{encoding:'utf8'});
}
main().catch(err => {
console.error(err);
});