Exporting Airflow connections from the CLI formatting - connection

I'm trying to set up a connection.json file in my local directory, so every time I spin up airflow on my docker container, it loads these connections through entrypoint.sh, therefore mitigating me having to establish connections via UI or programmatically in DAG.
As per the airflow docs, this is the proper format.
{
"airflow_db": {
"conn_type": "mysql",
"host": "mysql",
"login": "root",
"password": "plainpassword",
"schema": "airflow",
"port": null,
"extra": null
},
"druid_broker_default": {
"conn_type": "druid",
"host": "druid-broker",
"login": null,
"password": null,
"schema": null,
"port": 8082,
"extra": "{\"endpoint\": \"druid/v2/sql\"}"
}
}
Is this default format for every conn type?
For example, I'm trying to set up an s3 connection, and a google cloud connection. Here are the parameters on the UI:
So would I set up this connection in the json like
{
"google_cloud_default": {
"conn_type": "google_cloud_platform",
"project_id": "sample-proj-id123",
"keyfile_path": null,
"keyfile_json": {sample_json},
"scopes": "sample_scope",
"number_of_retries": 5,
}
or like the format pertaining to the airflow docs, however I'd just nullify unnecessary values, and add in data like proj_id, key_file_json, scopes in the extra field?
Hope this makes sense?

Related

chronograf: Not able to add default influxDB connection when using OAuth 2.0

I configured Chronograph to use generic OAuth 2.0 (using cloud foundry UAA). Users authentication works fine but the problem is that the default influxdb connection is not taken into consideration. In fact this configuration works:
chronograf --log-level="debug" --resources-path="/usr/share/chronograf/resources" --influxdb-url="http://influxDB.log.database:8086" --influxdb-username="usename" --influxdb-password="pass"
here is the content of /usr/share/chronograf/resources folder:
influxdb.src:
{
"id": "9999",
"name": "MyInfluxDB",
"username": "user1,
"password": "password1",
"url": "http://influxDB.log.database:8086",
"type": "influx",
"insecureSkipVerify": true,
"default": true,
"telegraf": "telegraf.autogen",
"organization": "Default"
}
Both connections are automatically created when chronoraf starts :
MyInfluxDB
http://influxDB.log.database:8086
but When I run chronograf with the following options (To use OAuth 2.0 and create an influxdb connection) :
export TOKEN_SECRET="token_secret"; export JWKS_URL="https://uaa/token_keys"; export PUBLIC_URL="http://chronograf:8888"; chronograf --log-level="debug" --resources-path="/usr/share/chronograf/resources" --generic-name="generic" --generic-client-id="id" --generic-client-secret="secret" --generic-scopes="openid" --generic-auth-url="https://uaa/oauth/authorize" --generic-token-url="https://uaa/oauth/token" --generic-api-url="https://uaa/userinfo"
The OAuth 2.0 works fine but once redirected to the chronograf dashboard I cannot see the connections and even when I created a connection manually and I log in I cannot found any connection that is created automatically on startup as wanted.
the field organization needs an id. The id for the Default orginization uses a lower case d. If you change your src file to,
{
"id": "9999",
"name": "MyInfluxDB",
"username": "user1,
"password": "password1",
"url": "http://influxDB.log.database:8086",
"type": "influx",
"insecureSkipVerify": true,
"default": true,
"telegraf": "telegraf.autogen",
"organization": "default"
}
It should now work.
you can see where the id is defined in their source here https://github.com/influxdata/chronograf/blob/9d8a49ba0ef8131cdce22d73718859f55f434db2/bolt/organizations.go#L20

Azure DevOps Extension custom service endopint for ID/KEY

I am developing Azure DevOps extension which contain service endpoint to hold secret ID/KEY. My requirement is to have endpoint just consist of Connection name, ID ,and Key in it.I have gone trough list of provided endpoints in Microsoft but I couldn't find suitable option to satisfy my requirement.
https://learn.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=vsts#sep-ssh
closest solution I found is as below . But it contains input box for server URL(Which I need to omit (in this example though I don't define server URL it displays in popup dialog)). Please refer below image.
Is it possible to remove Server URL from above dialog box Or it there better endpoint type I can use for this requirement? please be kind enough to share some light with me.
You'll need to create a custom service type, that will allow you to show/hide the individual text boxes. You can find an example in the Azure DevOps Extension Tasks which I maintain.
You define custom service endpoint types in the vss-extension.json along with your other extension points:
{
"id": "vsts-marketplace-endpoint-type",
"type": "ms.vss-endpoint.service-endpoint-type",
"targets": [
"ms.vss-endpoint.endpoint-types"
],
"properties": {
"name": "VstsMarketplacePublishing",
"displayName": "Visual Studio Marketplace",
"url": {
"displayName": "Marketplace URL",
"value": "https://marketplace.visualstudio.com",
"isVisible": "false"
},
"helpMarkDown": "Required permissions: <ul><li><b>Publish</b>: All accessible organisations, Marketplace (Publish)</li><li><b>Share</b>: All accessible organisations, Marketplace Publish</li><li><b>Install</b>: All accessible organisations or a specific organisation, Extensions (read and manage), Marketplace (acquire)</li><li><b>Query Version</b>: All accessible organisations, Marketplace (read)</li><li><b>Is Valid Extension</b>: All accessible organisations, Marketplace (read)</li></ul><br/><a href='https://www.visualstudio.com/docs/setup-admin/team-services/use-personal-access-tokens-to-authenticate'>More information</a>.",
"authenticationSchemes": [
{
"type": "ms.vss-endpoint.endpoint-auth-scheme-basic",
"inputDescriptors": [
{
"id": "username",
"name": "Username",
"description": "Username",
"inputMode": "textbox",
"isConfidential": false,
"validation": {
"isRequired": false,
"dataType": "string",
"maxLength": 300
},
"values": {
"inputId": "username",
"isDisabled": true,
"defaultValue": ""
}
},
{
"id": "password",
"name": "Personal access token",
"description": "Azure DevOps personal access token.",
"inputMode": "passwordbox",
"isConfidential": true,
"validation": {
"isRequired": true,
"dataType": "string",
"maxLength": 300
}
}
]
}
]
}
},
You may find other extensions that set or configure the authentication dialog on GitHub, there are quite a few. Useful docs are here in an old blog post.

SAPUI5, Datasource from 2 different URL

I am new in SAPUI5, currently I am building a SAPUI5 application and consumes ODATA services from backend. When I am creating an application and select service URL.
Service URL as following attachment.
However, now in my application I have to add another data source with totally url. In my manifest.json. I just can change different navigation of the same url as following
"dataSources": {
"mainService": {
"uri": "/sap/opu/odata/sap/SEPMRA_SHOP/",
"type": "OData",
"settings": {
"odataVersion": "2.0",
"localUri": "localService/metadata.xml"
}
},
"mainService2": {
"uri": "/odata/V2/OData/OData.svc/",
"type": "OData",
"settings": {
"odataVersion": "2.0",
"localUri": "localService/metadata.xml"
}
}
},
How can I add different url of datasource for SAPUI5 application?
You need to create a Destination in your SAP Cloud Platform cockpit.
You are already using one called "odata - odata" as per your screenshot.
You have the documentation here
And a screenshot below.
Fuertheremore, if you want to access a NW system in your private network from the cloud, you need to set up a Cloud connector. Check the documentation here
SAP Cloud Platform configuration
Once you have your destinations configured and you dataSource block already there (as you have in the screenshot), you need to create 2 OData models. So in your metadata.json file, go to the bottom, and there is a node called 'models' where you probably have something like this:
"models": {
"i18n": {
"type": "sap.ui.model.resource.ResourceModel",
"settings": {
...
}
},
"": {
"type": "sap.ui.model.odata.v2.ODataModel",
"settings": {
"defaultOperationMode": "Server",
"defaultBindingMode": "OneWay",
"defaultCountMode": "Request"
},
"dataSource": "mainService",
"preload": true
},
"dataModel2": {
"type": "sap.ui.model.odata.v2.ODataModel",
"settings": {
"defaultOperationMode": "Server",
"defaultBindingMode": "OneWay",
"defaultCountMode": "Request"
},
"dataSource": "mainService2",
"preload": true
}
}
You need to add the block "dataModel2":{ ... }
Where:
"dataModel2" is the name of your second model (choose the one you like, it doesn't need to be dataModel2).
Make sure that "dataSource": "mainService2" so this model points to your second data source
In the settings you can set the model as you want. Probably the most important is the "defaultBindingMode". Set it as "OneWay" if it is only readable, or "TwoWay" if it is read/write-able.
Then you just need to define all your bindings to this model with the modelName you choose as described in the first bullet point. For instance:
<Text text="{dataModel2>/path/to/my/property"}"></Text>
But be careful with this architecture, as per SAP Best Practices, you should try always to use only one OData service. Having more than one could be a nightmare to maintain, specially if you suscribe to model change/dataRequested/dataReceived events or for error handling with metadataFailed/metadataLoaded events, because you will have to manage them for both models and they are asynchronous...

'eventNotification' url is not being called

I am integrating DocuSign with our application. In testing phase we got a server with public IP and port (8086). On this port I have published my asp.net mvc web api and my url becomes:
http://XXX.XXX.XXX.XXX:8086/api/DocuSign/DocuSignDocumentStatus.
This url is going to be called from DocuSign whenever my document status changes.
Our network team has allowed access to following IPs for inbound access to this public IP/URL:
www.docusign.net 209.67.98.12
mailsea.docusign.net 209.67.98.59
NA2
na2.docusign.net 206.25.247.140
mailch.docusign.net 206.25.247.155
EU1
eu1.docusign.net 206.25.247.144
mailch.docusign.net 206.25.247.155
DAL/DR
demo.docusign.net 209.46.117.172
preview.docusign.net 209.46.117.174
mailda.docusign.net 209.46.117.17
I got this from the connect service reference pdf.
Whenever I am uploading the document from signing via docuSign web api and I am also sending this url with the request. I have checked - rechecked many times that my json request being created is correct (pasting it below):
{
"status": "sent",
"emailBlurb": "",
"emailSubject": "DocuSign API - Signature Request on Document Call",
"documents": [
{
"name": "someDocument.xls",
"documentId": "1"
}
],
"recipients": {
"signers": [
{
"recipientId": "1",
"email": "john.doe#someCompany.com",
"name": "John Doe",
"tabs": {
"signHereTabs": [
{
"xPosition": "100",
"yPosition": "100",
"documentId": "1",
"pageNumber": "1"
}
]
},
"routingOrder": "1"
}
],
"carbonCopies": [
{
"recipientId": "2",
"email": "some1.recipient#someCompany.com",
"name": "Some1 Recipient"
},
{
"recipientId": "3",
"email": "some2.recipient#someCompany.com",
"name": "Some2 Recipient"
},
{
"recipientId": "4",
"email": "some3.recipient#someCompany.com",
"name": "Some3 Recipient"
}
]
},
"eventNotification": {
"url": "http://XXX.XXX.XXX.XXX:8086/api/DocuSign/DocuSignDocumentStatus",
"loggingEnabled": true,
"requireAcknowledgement": true,
"includeDocuments": false,
"envelopeEvents": [
{
"envelopeEventStatusCode": "Completed"
},
{
"envelopeEventStatusCode": "Declined"
}
]
}
}
I am able successfully upload the document, emails are being sent successfully to all signers. Document being signed BUT DocuSign for some reason is not able to call my url with the status of the document. Please help. Let me know if you guys need any more information.
As specified by the Answer & subsequent Comment in this other question:
Regardless of whether you're using DocuSign Connect (configured at the account level) or using eventNotification (specified at the Envelope level), DocuSign will only publish messages to the "standard/default" ports:
In the DocuSign demo environment (demo.docusign.net) DocuSign Connect will publish to either port 80 (http) or port 443 (https). If the URL starts with "http", Connect will attempt to publish to port 80. If the URL starts with "https", Connect will attempt to publish to port 443.
In the DocuSign production environment (www.docusign.net), DocuSign Connect will only publish to port 443 (https). Publishing to port 80 (http) is not supported in the production environment -- the listener endpoint must be https.
Therefore, I'd suggest that you remove the port number from the URL that you've specified for "eventNotification", and ensure that your listener endpoint is located at either port 80 (for demo) or port 443 (for demo or prod).

how to get an openstack token and validate it?

I followed this guide: http://keystone.openstack.org/api_curl_examples.html
and it seemed that I got a valid token by ran:
curl -d '{"auth":{"passwordCredentials":{"username": "can", "password": "mypassword"}}}' -H "Content-type: application/json" http://url:35357/v2.0/tokens
and it returned:
{
"access":
{
"token":
{
"expires": "2012-05-21T14:35:17Z",
"id": "468da447bd1c4821bbc5def0498fd441"
},
"serviceCatalog": {},
"user":
{
"username": "can",
"roles_links": [],
"id": "bb6d3a09ad0c4924bf20c1a32ccb5781",
"roles": [],
"name": "can"
}
}
}
but when I came to the next few sections to validate this token, I encountered this magic number: X-Auth-Token:999888777666. At first I thought it's the token I got but I was wrong.
I think I may have missed something, so I read related sections in openstack documents( http://keystone.openstack.org/configuration.html and http://docs.openstack.org/api/openstack-compute/programmer/content/ ), but still no idea how the number comes from.
could anyone explain to me
what's the meaning of that magic number
how to get the right value of it so I can get a working token to manage other parts of openstack
That magic number (string really) is the admin_token setting in your keystone.conf file. Under the [DEFAULT] section in keystone.conf set
admin_token = abcd1234
If you don't use it for admin actions, you'll see something like
ubuntu#i-000004bc:~/devstack$ curl http://localhost:35357/v2.0/tenants
{"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Not Authorized"}}
If you do use it, you'll see something like
ubuntu#i-000004bc:~/devstack$ curl -H "X-Auth-Token: abcd1234" http://localhost:35357/v2.0/tenants
{"tenants_links": [], "tenants": [{"enabled": true, "description": null, "name": "demo", "id": "aee8a46babcb4e4286021c8f6ef996cd"}, {"enabled": true, "description": null, "name": "invisible_to_admin", "id": "de17fea45de148ada0a58e998e6c3e73"}, {"enabled": true, "description": null, "name": "admin", "id": "f34b0c8ab30e450489b121fbe723fde5"}, {"enabled": true, "description": null, "name": "service", "id": "fbe3e2e530fd47298cb2cba1b4afa3da"}]}
To get the list of tenants, in our current implementation, we authenticate with admin credentials and use the token returned to get list of tenants.
The implementation works smooth with the authentication token. It may work with admin_token but I have not verified.
If you see the examples you are referring to, there are 2 types of endpoints used
Endpoint pointing to port 5000 - public port
Endpoint pointing to port 35357 - admin port
In examples which are hitting admin port you would need to specify the "X-Auth-Token" header as the admin_token (specified in the keystone.conf file)
The token itself is in dict["access"]["token"]["id"] which is that part that will go in the header of subsequent HTTP requests, i.e
X-Auth-Token: 468da447bd1c4821bbc5def0498fd441
The value 999888777666 is from the curl example tutorial and will definitely not work.
As for the value of the token itself, it's randomly generated by the OpenStack service and shouldn't contain any useful information from your point of view.
(BTW, you probably shouldn't go pasting the tokens into forums as they're valid for 24 hours and anyone with a copy of the token and access to your compute endpoint can use it to impersonate you).

Resources