I'm trying to create a DAG on airflow that performs some query on snowflake. For our implementation we decided to connect using a private key in stead of using a password.
When I try to add the private_key_file to the extra field (see image) in the snowflake connection however it is not correctly stored. It says it saves it correctly but afterwards the key and value are not in the extra field. However the following is automatically inserted:
{"extra__snowflake__account": "",
"extra__snowflake__aws_access_key_id": "",
"extra__snowflake__aws_secret_access_key": "",
"extra__snowflake__database": "", "extra__snowflake__region": "",
"extra__snowflake__warehouse": ""}
In the placeholder example the "private_key_file" is shown however as shown in the image and in the source code the "private_key_file" key is also used to get a value from this field, so I'm pretty sure I'm putting it in the right spot.
airflow version 2.1.1
apache-airflow-providers-snowflake version 2.0.0
How do I add a private key file to a snowflake connection on airflow?
You may have already worked this out but here is how I got around this issue.
The web interface does not work for this you need to use the Airflow command line to create the connection with the private_key_file parameter.
You can use a python script as follows to create the Connection URI:
import json
from airflow.models.connection import Connection
c = Connection(
conn_id='SF_SSO',
conn_type='snowflake',
description='cli generated',
host='https://poc.us-east-1.snowflakecomputing.com/',
schema='TEST',
login='your_name#yourcompany.com',
password='private_key_file_password',
extra=json.dumps(dict(account='poc', database='TEST', region='us-east-1', warehouse='TEST_WH', private_key_file='/home/username/.snowsql/rsa_key.p8')),
)
print(f"AIRFLOW_CONN_{c.conn_id.upper()}='{c.get_uri()}'")
This should generate a URI that looks something like this:
snowflake://your.name%40yourcompany.com:private_key_file_password#https%3A%2F%2Fpoc.us-east-1.snowflakecomputing.com%2F/TEST?account=poc&database=TEST®ion=us-east-1&warehouse=TEST_WH&private_key_file=%2Fhome%2Fusername%2F.snowsql%2Frsa_key.p8
Now using the URI above create the connection using the cli:
airflow connections add 'SF_SSO' --conn-uri 'snowflake://your.name%40yourcompany.com:private_key_file_password#https%3A%2F%2Fpoc.us-east-1.snowflakecomputing.com%2F/TEST?account=poc&database=TEST®ion=us-east-1&warehouse=TEST_WH&private_key_file=%2Fhome%2Fusername%2F.snowsql%2Frsa_key.p8'
Related
I am writing an IBM Cloud Function which uses the python SDK to interface with a Cloudant service. I have the Cloudant service up, the databases populated, and service credentials / API key ready. However when I try to instantiate the CloudantV1 service inside my Function I get a runtime error "must provide authenticator".
I looked up the error in their git repos and it seems like it is trying to setup an authenticator object by looking up values from environment variables, which do not exist in the Function. I just want to pass my API key directly, but I have not found a method to do this. I am using basic code from the examples so I think my calls are correct.
I have considered injecting the environment variables inside the Function, but that sounds like a major hack. I must be doing something incorrectly. Please help me understand what it is. Here is basic Function python code which reproduces the error:
from ibmcloudant.cloudant_v1 import CloudantV1
def main(params_dict):
service = CloudantV1.new_instance()
# unreachable
return { "message" : "hello world" }
There is an example for programmatic authentication at https://cloud.ibm.com/apidocs/cloudant?code=python#programmatic-authentication - it basically looks like this:
from ibmcloudant.cloudant_v1 import CloudantV1
from ibm_cloud_sdk_core.authenticators import IAMAuthenticator
authenticator = IAMAuthenticator('yourAPIkey')
service = CloudantV1(authenticator=authenticator)
service.set_service_url('https://yourserviceurl.example')
I'm using great_expectations==0.13.4 and sqlalchemy==1.3.22 and am attempting to create the string which will provide the connection in URL format. These will be added manually to the systems, as the CLI does not allow for identification of the CERT or the Schema.
I'm using the following code to define and connect to the URL, while I had this working last night my computer rebooted before I could hit save. I believe the issue is with line "query={"ssl_ca": certificate_file}", but I cannot identify the correct keyword to allow the cert file to be listed in the URL.
Any and all help is appreciated. Thank you
import sqlalchemy
certificate_file = r'c:\Users\Dir\To\My\Certificate.crt'
sqlUrl = sqlalchemy.engine.url.URL(
drivername='presto',
username=actual_user,
password= quote_plus(actual_pswd),
host= hostname,
port=port_num,
database=db_name,
query={"ssl_ca": certificate_file}
)
engine = sqlalchemy.create_engine(sqlUrl)
with engine.connect() as connection:
result = connection.execute("show schemas")
The error I get is during the line above ("with engine.connect() as connecton") is
StatementError: (builtins.TypeError) __init__() got an unexpected keyword argument 'ssl_ca'
[SQL: show schemas]
I have tried options in the URL connection for query (request_kwargs, verify & ca, ssl_ca) and some variations there in. So far no luck
I am working on IB Gateway and want to get the historical data.
As i have completed the steps on IB Gateway software to enable the API.
I am using python notebook for this.
For now i am running this code and i am able to import the given library but rest of the code giving me this error. Important thing is connection is established as I have mention client id 1. then it is created and can be seen on IB Gateway application.
My code is here.
from ib_insync import *
#util.startLoop() # uncomment this line when in a notebook
ib = IB()
ib.connect('127.0.0.1', 5021, clientId=1)
bars = ib.reqHistoricalData(
contract=Stock('TSLA', 'SMART', 'USD'),
endDateTime='',
durationStr='30 D',
barSizeSetting='1 hour',
whatToShow='TRADES',
useRTH=True)
print(bars)
Here is the error.
Peer closed connection
clientId 1 already in use?
API connection failed: CancelledError()
As i am using notebook if i uncomment the second line (util.startLoop()) it adds one more error about timeout..
Need help to get this done.
Big Thanks
Assign a different clientID to this connecion:
ib.connect('127.0.0.1', 5021, clientId=2)
Apparently you already have another connection with clientId=1.
I am new to SSIS, here is the issue I am facing.
1) Created an OLEDB connection to connect to database. Storing the Connection String in SSIS variable and have added expression to Connection Manager to pick the connection string according to environment.
2) Have used Windows Authentication to connect to database, so that no need to provide user ID and Password. In development environment it worked perfectly fine. But when moved to testing environment, its failing with Error "SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "REGEDB" failed with error code 0xC0202009"
3) Using SQL Server Configuration to deploy the package. But getting error Failed to load at least one of the configuration entries for the package. Check configuration entries for "CBPSSIS" and previous warnings to see descriptions of which configuration failed.
Below is the Connection String
Data Source=abcd\ISQLQ02;Initial Catalog=DRIP;Provider=SQLNCLI10;Integrated Security=SSPI;Auto Translate=False;
COuld you please let me know how to set Password property? I tired in Script file but its not working. Thank you
ConnectionManager OldedbConn;
OldedbConn = Dts.Connections["QAREGE"]; Dts.Connections["QAREGE"].Properties["ServerName"].SetValue(OldedbConn, Dts.Variables["User::dbServerName"].Value); Dts.Connections["QAREGE"].Properties["InitialCatalog"].SetValue(OldedbConn, Dts.Variables["User::dbCatalog"].Value); Dts.Connections["QAREGE"].Properties["UserName"].SetValue(OldedbConn, Dts.Variables["User::dbUserID"].Value); Dts.Connections["QAREGE"].Properties["Password"].SetValue(OldedbConn, Dts.Variables["User::dbPwd"].Value);
Since you are using ssis-2008 you can create a ssis configFile with .dtsConfig format with xml configuration file type. Include all your Database connection string in the config file. So when you moved your ssis package to another server you just need to edit the config file and set your connection based on your target server.
I'm working on a Jenkins plugin where we make a call out to a remote service using Spring's RestTemplate. To configure the timeout values, I'm setting up some fields in the global configuration using the global.jelly file for Jenkins plugins using a number field as shown here:
<f:entry title="Read Timeout" field="readTimeout" description="Read timeout in ms.">
<f:number default="3000"/>
</f:entry>
Now, this works to save the values and retrieve the values no problem, so it looks like everything is setup correctly for my BuildStepDescriptor. However, when I first install the update to a Jenkins instance, instead of getting 3000 in the field by default as I would expect, instead I am getting 0. This is the same for all the fields that I'm using.
Given that the Jelly tag reference library says this attribute should be the default value, why do I keep seeing 0 when I first install the plugin?
Is there some more Java code that needs to be added to my plugin to tie the default in Jelly back to the global configuration?
I would think that when Jenkins starts, it goes to get the plugin configuration XML and fails to find a value and sets it to a default of 0.
I have got round this in the past by setting a default in the descriptor (in groovy) then this value will be saved into the global config the first time in and also be available if the user never visits the config page.
#Extension
static class DescriptorImpl extends AxisDescriptor {
final String displayName = 'Selenium Capability Axis'
String server = 'http://localhost:4444'
Boolean sauceLabs = false
String sauceLabsName
Secret sauceLabsPwd
String sauceLabsAPIURL =
'http://saucelabs.com/rest/v1/info/platforms/webdriver'
String sauceLabsURL = 'http://ondemand.saucelabs.com:80'
from here