I've added all my configuration details in the Vault. The detail you can see in the attached image below. This follows a specific path i.e kv/unistad/dev/workflow/camunda/1.0
However, when I try to read this information using Vault.NET with the following nuget package
Install-Package Vault
My code looks something like this:
var endpoint = "http://openblue-bridge.com:32270";
var token = "s.inklpUdNxet1ZJtaCLMpEIPA";
var vaultClient = new VaultClient(new Uri(endpoint), token);
string project = "unistad";
string environment = "dev";
string appVersion = "1.0";
var secretPath = $"kv/{project}/{environment}/workflow/camunda/{appVersion}";
// Use client to read a key-value secret.
var secrets = await vaultClient.Secret.Read< Dictionary<string, string>> (secretPath);
When I run the above code I get the following error:
Invalid path for a versioned K/V secrets engine. See the API docs for
the appropriate API endpoints to use. If using the Vault CLI, use
'vault kv get' for this operation.
I'm not sure how can I fix this error. Any help would be really appreciated.
You are using v2 of the kv engine. For that engine, you need to have /data/ in the path, as shown in the API docs. The requirement for this prefix is also described in the engine docs.
So the solution to your problem is specifically to change your path from
var secretPath = $"kv/{project}/{environment}/workflow/camunda/{appVersion}";
to
var secretPath = $"kv/data/{project}/{environment}/workflow/camunda/{appVersion}";
Related
I have a running cloud run service user-service. For test purposes I passed client secrets via environment variables as plain text. Now since everything is working fine I'd like to use a secret instead.
In the "Variables" tab of the "Edit Revision" option I can declare environment variables but I have no idea how to pass in a secret? Do I just need to pass the secret name like ${my-secret-id} in the value field of the variable? There is not documentation on how to use secrets in this tab only a hint at the top:
Store and consume secrets using Secret Manager
Which is not very helpful in this case.
You can now read secrets from Secret Manager as environment variables in Cloud Run. This means you can audit your secrets, set permissions per secret, version secrets, etc, and your code doesn't have to change.
You can point to the secrets through the Cloud Console GUI (console.cloud.google.com) or make the configuration when you deploy your Cloud Run service from the command-line:
gcloud beta run deploy SERVICE --image IMAGE_URL --update-secrets=ENV_VAR_NAME=SECRET_NAME:VERSION
Six-minute video overview: https://youtu.be/JIE89dneaGo
Detailed docs: https://cloud.google.com/run/docs/configuring/secrets
UPDATE 2021: There is now a Cloud Run preview for loading secrets to an environment variable or a volume. https://cloud.google.com/run/docs/configuring/secrets
The question is now answered however I have been experiencing a similar problem using Cloud Run with Java & Quarkus and a native image created using GraalVM.
While Cloud Run is a really interesting technology at the time of writing it lacks the ability to load secrets through the Cloud Run configuration. This has certainly added complexity in my app when doing local development.
Additionally Google's documentation is really quite poor. The quick-start lacks a clear Java example for getting a secret[1] without it being set in the same method - I'd expect this to have been the most common use case!
The javadoc itself seems to be largely autogenerated with protobuf language everywhere. There are various similarly named methods like getSecret, getSecretVersion and accessSecretVersion
I'd really like to see some improvment from Google around this. I don't think it is asking too much for dedicated teams to make libraries for common languages with proper documentation.
Here is a snippet that I'm using to load this information. It requires the GCP Secret library and also the GCP Cloud Core library for loading the project ID.
public String getSecret(final String secretName) {
LOGGER.info("Going to load secret {}", secretName);
// SecretManagerServiceClient should be closed after request
try (SecretManagerServiceClient client = buildClient()) {
// Latest is an alias to the latest version of a secret
final SecretVersionName name = SecretVersionName.of(getProjectId(), secretName, "latest");
return client.accessSecretVersion(name).getPayload().getData().toStringUtf8();
}
}
private String getProjectId() {
if (projectId == null) {
projectId = ServiceOptions.getDefaultProjectId();
}
return projectId;
}
private SecretManagerServiceClient buildClient() {
try {
return SecretManagerServiceClient.create();
} catch(final IOException e) {
throw new RuntimeException(e);
}
}
[1] - https://cloud.google.com/secret-manager/docs/reference/libraries
Google have documentation for the Secret manager client libraries that you can use in your api.
This should help you do what you want
https://cloud.google.com/secret-manager/docs/reference/libraries
Since you haven't specified a language I have a nodejs example of how to access the latest version of your secret using your project id and secret name. The reason I add this is because the documentation is not clear on the string you need to provide as the name.
const [version] = await this.secretClient.accessSecretVersion({
name: `projects/${process.env.project_id}/secrets/${secretName}/versions/latest`,
});
return version.payload.data.toString()
Be sure to allow secret manager access in your IAM settings for the service account that your api uses within GCP.
I kinda found a way to use secrets as environment variables.
The following doc (https://cloud.google.com/sdk/gcloud/reference/run/deploy) states:
Specify secrets to mount or provide as environment variables. Keys
starting with a forward slash '/' are mount paths. All other keys
correspond to environment variables. The values associated with each
of these should be in the form SECRET_NAME:KEY_IN_SECRET; you may omit
the key within the secret to specify a mount of all keys within the
secret. For example:
'--update-secrets=/my/path=mysecret,ENV=othersecret:key.json' will
create a volume with secret 'mysecret' and mount that volume at
'/my/path'. Because no secret key was specified, all keys in
'mysecret' will be included. An environment variable named ENV will
also be created whose value is the value of 'key.json' in
'othersecret'. At most one of these may be specified
Here is a snippet of Java code to get all secrets of your Cloud Run project. It requires the com.google.cloud/google-cloud-secretmanager artifact.
Map<String, String> secrets = new HashMap<>();
String projectId;
String url = "http://metadata.google.internal/computeMetadata/v1/project/project-id";
HttpURLConnection conn = (HttpURLConnection)(new URL(url).openConnection());
conn.setRequestProperty("Metadata-Flavor", "Google");
try {
InputStream in = conn.getInputStream();
projectId = new String(in.readAllBytes(), StandardCharsets.UTF_8);
} finally {
conn.disconnect();
}
Set<String> names = new HashSet<>();
try (SecretManagerServiceClient client = SecretManagerServiceClient.create()) {
ProjectName projectName = ProjectName.of(projectId);
ListSecretsPagedResponse pagedResponse = client.listSecrets(projectName);
pagedResponse
.iterateAll()
.forEach(secret -> { names.add(secret.getName()); });
for (String secretName : names) {
String name = secretName.substring(secretName.lastIndexOf("/") + 1);
SecretVersionName nameParam = SecretVersionName.of(projectId, name, "latest");
String secretValue = client.accessSecretVersion(nameParam).getPayload().getData().toStringUtf8();
secrets.put(secretName, secretValue);
}
}
Cloud Run support for referencing Secret Manager Secrets is now at general availability (GA).
https://cloud.google.com/run/docs/release-notes#November_09_2021
We wanted to list images and tags which names start with certain string. So far, we explored a few java lib (docker-java and spotify ones) and did quite amount of research, but still couldn't find a way out...
docker-java: 'com.github.docker-java', name: 'docker-java', version: '3.2.5'
The follow code lists images from public docker hub, not really the specified GCR. What's the right way to list image from our specified GCR?
DefaultDockerClientConfig config = DefaultDockerClientConfig
.createDefaultConfigBuilder()
.withRegistryUrl("http://eu.gcr.io/data-infrastructure-test-env")
.withDockerConfig("/home/me/.docker/config.json")
.build();
DockerClient dockerClient = DockerClientBuilder.getInstance(config).build();
List<SearchItem> items = dockerClient.searchImagesCmd("daas").exec();
List<String> images = new ArrayList<>();
for (SearchItem searchItem : items){
images.add(searchItem.getName());
}
Update - some progress
Inspired by this post: How to list images and tags from the gcr.io Docker Registry using the HTTP API?
I tried the following steps with my own google account, which has project owner (w/o firewall) permission:
gcloud auth login
gcloud auth print-access-token
define a function to get string for basic auth:
private String basicAuth(String username, String password) { return "Basic " + Base64.getEncoder().encodeToString((username + ":" + password).getBytes()); }
4, try the following code:
HttpRequest request = HttpRequest.newBuilder().uri(URI.create("https://gcr.io/v2/token?service=eu.gcr.io&scope=registry:my_gcp_project:*"))
.headers("Accept", "application/json"
, "Authorization",basicAuth("_token"
,"the_token_got_from_step_2"))
.GET()
.build(); UncheckedObjectMapper objectMapper = new UncheckedObjectMapper(); Map<String, String> response = HttpClient.newHttpClient()
.sendAsync(request, HttpResponse.BodyHandlers.ofString())
.thenApply(HttpResponse::body).thenApply(objectMapper::readValue)
.get();
String token = response.get("token");
request = HttpRequest.newBuilder().uri(URI.create("https://eu.gcr.io/v2/my_gcp_project/my_image/tags/list"))
.header("Authorization","Bearer " + token)
.GET().build(); String response2 = HttpClient.newHttpClient()
.sendAsync(request, HttpResponse.BodyHandlers.ofString())
.thenApply(HttpResponse::body)
.get();
However, the response2 I got was:
{"errors":[{"code":"UNAUTHORIZED","message":"Requested repository does not match bearer token resource: data-infrastructure-test-env/daas-master"}]}
Could you help to check what went wrong?
Docker engine API documentation clearly states that the ImageSearch command returns images from the Docker Hub registry: https://docs.docker.com/engine/api/v1.40/#operation/ImageSearch
For searching a GCR registry, you should rather use the Docker registry API.
Finally got it work!
I only need to change the second request uri to be: "https://eu.gcr.io/v2/my_gcp_project/tags/list" instead of "https://eu.gcr.io/v2/my_gcp_project/my_image/tags/list"
and I got some meaningful response back
Title says it all pretty much.
I tried uploading the json file to azure storage and referenced it's url when setting the GOOGLE_APPLICATION_CREDENTIALS environment variable under app settings, but when remotely debugging the site, apparently the url/directory was not in an acceptable format. I can’t store the json file locally either because the website doesn’t have any idea about my C drive directories.
Where should I store this file so that I can set the GOOGLE_APPLICATION_CREDENTIALS environment variable for my azure site to the directory of the json file?
The ToChannelCredentials() approach does not seem to work anymore, so I come up with an other solution that works on Azure. I create a text file in the /bin folder of my Azure server with the credentials and then I point the environment variable to this file. Google Cloud API will use this for the default credentials.
string json = #"{
'type': 'service_account',
'project_id': 'xxx',
'private_key_id': 'xx',
'private_key': 'xxx',
...
}"; // this is the content of the json-credentials file from Google
// Create text file in projects bin-folder
var binDirectory = Path.GetDirectoryName(Assembly.GetCallingAssembly().CodeBase);
string fullPath = Path.Combine(binDirectory, "credentials.json").Replace("file:\\","");
using (StreamWriter outputFile = new StreamWriter(fullPath, false)) {
outputFile.WriteLine(json);
}
// Set environment variabel to the full file path
Environment.SetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS", fullPath);
// Now you can call the service and it will pick up your credentials
TranslationServiceClient client = TranslationServiceClient.Create();
If anyone is wondering how to handle the Google's credentials smoothly in .Net applications instead of strange way of using the file on server, this is how I solved it for Translation Service. Other services must follow same principle:
store the content of the Google credentials json file as an environment variable in settings.json/azure configuration for your app (using ' ' instead of " " for inner text):
"GOOGLE_APPLICATION_CREDENTIALS": "{'type': 'service_account','project_id': ...}"
create and return the client:
var credential = GoogleCredential.FromJson(Environment.GetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS"));
var channelCredentials = credential.ToChannelCredentials();
var channel = new Channel(TranslationServiceClient.DefaultEndpoint.ToString(), channelCredentials);
return TranslationServiceClient.Create(channel);
Took a while for me to figure it our. Hope it helps.
I use the .json file in my local environment (because of environment variable length limit in Windows) and on Azure I use an "Application setting" to set an environment variable. This code handles both cases:
string? json;
var filename = Environment.GetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS");
if (filename != null)
{
json = System.IO.File.ReadAllText(filename);
}
else
{
json = Environment.GetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS_STRING");
if (json == null)
{
throw new Exception(
"GOOGLE_APPLICATION_CREDENTIALS_STRING environment variable with JSON is not set");
}
}
var credential = GoogleCredential.FromJson(json).ToChannelCredentials();
var grpcChannel = new Channel("firestore.googleapis.com", credential);
var grcpClient = new Firestore.FirestoreClient(grpcChannel);
var firestoreClient = new FirestoreClientImpl(grcpClient, FirestoreSettings.GetDefault());
return await FirestoreDb.CreateAsync(FirebaseProjectId, firestoreClient);
Was looking how to set the "GOOGLE_APPLICATION_CREDENTIALS" in Azure App Service. The answers here didn't help me. My solution is very simple without any code change.
In the configuration of the app service go to the Path Mappings
Add a New Azure Storage Mount. eg /mounts/config
Add the credentials.json file to the file share
In the application settings, add the GOOGLE_APPLICATION_CREDENTIALS and set the value to: /mounts/config/credentials.json
That is all.
In the azure app on the azure portal go to application settings and add the credentials under application settings tab
Then you can reference them in your code as they were in your web.config file.
I used Swagger Yaml to describe an endpoint and generate the mock server. The existing endpoint (that I'm mocking) doesn't follow RESTful principles 100%, so I simply want to overwrite the response that is returned by the mock server. The simple server code is shown below:
var swagger = require('swagger-server');
var server = swagger('map-cache.yaml');
var port = 7072;
server.post('/map-qa_trunk/v2/getData', function(req, res, next) {
var foo = {
err : 123,
msg : "error message"
};
res.json(foo);
});
server.listen(port, function() {
console.log('Map Cache Mock Server is now running at http://localhost:' + port);
});
In the Yaml definition, there is an object defined called MapResponseData, how do I create an instance of this object so that I can populate it as needed and return in the res.json()? Something similar to below:
var response = getMapResponseData(); // don't know what this call should be
response.fieldA = 123;
res.json(response);
I am guessing this should be possible, since Swagger parsed the YAML file and is aware of all definitions that were specified.
Try outputting the request object to console.log to see if you can find reference to the swagger definition. Another option would be to pull the parsed swagger definition from the yaml file (using js-yaml for example) and extracting from there.
However, my best advice is to use swagger-tools instead of swagger-server. The swagger-server package is alpha version and has fewer downloads, revisions, and users than swagger-tools. Benefit of swagger-tools is that it will be actively maintained and there is a larger community that can support you. To convert your project to swagger-tools, use swagger.io > Swagger Editor > Online Editor > Paste yaml in left pane > Generate Server > Node.js
In swagger-tools the entire Swagger Yaml definition is contained in each request object:
req.swagger.swaggerObject
and you can pull the response object definitions from that as needed.
I'm trying to get the followers for a user that has authenticated through my meteorjs app. I have used the {{loginButtons}} template and have found where the users tokens are. However I now have to create my authorized request by hand and I was hoping this'd be easy. But it's really hard and I feel like I'm wasting time with trying to figure out a way to create the oauth_signature..
Any help is welcome!
Supposing it is Twitter you're talking about I might be able to help you out.
I just managed to do the same thing as you want to do.
This nice piece of code provides a client to the Twitter API: https://github.com/mynetx/codebird-js
Personally I have placed it in the server-folder in my app to avoid exposure of keys etc.
As the codebird-js code take use of XMLHttpRequests and node.js do not come with such functionality by default - at least in a meteor.js context - you have to add the XHR-functionality yourself.
This NPM did it for me: https://npmjs.org/package/xmlhttprequest
However, as you can not deploy your meteor app with additional npm packages I found this solution How can I deploy node modules in a Meteor app on meteor.com? that suggests placing it in the public folder.
Finally I added those lines of code in the codebird-js just below the line that says
var Codebird = function () {
var require = __meteor_bootstrap__.require;
var path = require('path');
var fs = require('fs');
var base = path.resolve('.');
var isBundle = fs.existsSync(base + '/bundle');
var modulePath = base + (isBundle ? '/bundle/static' : '/public') + '/node_modules';
var XMLHttpRequest = require(modulePath + '/xmlhttprequest').XMLHttpRequest;
Finally you have to provide your tokens generated at dev.twitter.com and find your user's tokens stored in the Users collection.
EDIT:
Whenenver you have the above you make a new Codebird object: var bird = new Codebird();
Then you set tokens:
bird.setToken(USER_ACCESS_TOKEN, USER_ACCESS_TOKEN_SECRET);
And makes the call:
bird.__call('friends/ids', {
screen_name': SCREEN_NAME,
user_id: TWITTER_ID
},
function(reply){
console.log(reply);
});
Note that USER_ACCESS_TOKEN, USER_ACCESS_TOKEN_SECRET, USER_NAME & TWITTER_ID in the above example are placeholders. They are all found in the Meteor Users collection.