I want to share database connections that I use for a transformation across other transformations.
Is it possible in Pentaho Kettle? If yes then how?
If no, is there any other workaround for the same except for manually defining all the DB connections again?
I found this solution many years ago, somewhere on the pentaho website and I still use it.
For sharing database connections I use a file named shared.xml in the directory ~/.kettle (in my HOME directory). You have to restart pentaho after writing / changing your file. After that you can select the database connection by name defined in your shared.xml in you transformations.
This is an example with some basic attributes. I use it as a template.
<?xml version="1.0" encoding="UTF-8"?>
<sharedobjects>
<connection>
<name>name</name>
<server>server</server>
<type>type</type>
<access>Native</access>
<database>database</database>
<port>1234</port>
<username>username</username>
<password>Encrypted ....</password>
<servername/>
<data_tablespace/>
<index_tablespace/>
<attributes>
<attribute><code>FORCE_IDENTIFIERS_TO_LOWERCASE</code><attribute>N</attribute></attribute>
<attribute><code>FORCE_IDENTIFIERS_TO_UPPERCASE</code><attribute>N</attribute></attribute>
<attribute><code>IS_CLUSTERED</code><attribute>N</attribute></attribute>
<attribute><code>PORT_NUMBER</code><attribute>3306</attribute></attribute>
<attribute><code>PRESERVE_RESERVED_WORD_CASE</code><attribute>Y</attribute></attribute>
<attribute><code>QUOTE_ALL_FIELDS</code><attribute>N</attribute></attribute>
<attribute><code>STREAM_RESULTS</code><attribute>Y</attribute></attribute>
<attribute><code>SUPPORTS_BOOLEAN_DATA_TYPE</code><attribute>Y</attribute></attribute>
<attribute><code>SUPPORTS_TIMESTAMP_DATA_TYPE</code><attribute>Y</attribute></attribute>
<attribute><code>USE_POOLING</code><attribute>N</attribute></attribute>
</attributes>
</connection> </sharedobjects>
short explanation:
name = choose a name you can identify your database...
server = localhost or IP address or FQDN
type = POSTGRESQL, MYSQL ... database type
database = the real database name used with CREATE ...
username = must be a known user to the database
Related
Possible?
Cannot find a "sink forwarder", where one sink can forward to several other sinks, possibly of the same type.
Serilogs documentation (https://github.com/serilog/serilog/wiki/AppSettings)
clearly states that
NOTE: When using serilog: keys need to be unique.*
so adding the same Seq sink several times doesnt seem to be a good idea.
I'm looking for the same concept as in log4net, where one logger can hold several appenders.
Unfortunately the <appSettings> config provider for Serilog doesn't support this case; the appSettings.json one does, if you're able to use it, otherwise configuring the sinks in code WriteTo.Seq(...).WriteTo.Seq(...) is the way to go.
Semi-workaround style of solution:
Put a "read these keys" in appsettigs
Example 1: Read one key
<add key="SerilogToHttpKeys" value="MyMachineA" />
Example 2 (which solves the problem): Read many keys
<add key="SerilogToHttpKeys" value="MyMachineA, MyLocalMachine, MachineOnTheMoon" />
Both cases "points" to an unlimited number of keys, that are then read via code (see 2) and hence be tweaked w/out recompiling
<add key="MyLocalMachine" value="http://localhost:5341/;juzOPqqqqqqqq" />
<add key="MyMachineA" value="http://10.107.14.57:5341/;m8QVnDaqqqqqqqqqqqqq" />
<add key="MachineOnTheMoon" value="http://10.107.14.62:5341/;Ah0tSzqqqqqqqqqqqq"
Loop the keys in code - each key points to a http address with an API key, which is used for logging to Seq, but change the structure of each entry, and you could log to file ect.
foreach (var aKey in System.Configuration.ConfigurationManager.AppSettings.Get("SerilogToHttpKeys")
.Split(',')//Use , as separator
.Select(s => s.Trim()))
{
var fields = System.Configuration.ConfigurationManager.AppSettings.Get(aKey);
var separator = ';';
string serverUrl = fields.Split(separator)[0];
string apiKey = fields.Split(separator)[1];
loggerConfiguration = loggerConfiguration.WriteTo.Seq(serverUrl: serverUrl, apiKey: apiKey);
}
I use this for logging both to my server and my dev machine at the same time - its easier to keep the localhost Seq open when errors occour, and see if I can find them there instead of logging into the server. However, in case my devmachine is not online, I have the logs on the server as well. Off course, if more than one persons accesses the server licenses are needed for Seq, but in a simple "one dev, one dev-machine, one server" it works.
I'm using FSharp.Data.SqlClient and trying to move my connectionString from a [<Literal>] to the app.config.
My app.config looks like this
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<connectionStrings>
<add name="DefaultConnection" connectionString="Data Source=(localdb)\MSSQLLocalDB;Initial Catalog=Test;Integrated Security=True" providerName="System.Data.SqlClient"/>
</connectionStrings>
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6.2" />
</startup>
</configuration>
And my SqlCommandProvider looks like the below, which should be correct according to http://fsprojects.github.io/FSharp.Data.SqlClient/configuration%20and%20input.html
new SqlCommandProvider<"SELECT ...",
ConnectionStringOrName = "name=DefaultConnection",
SingleRow = true,
AllParametersOptional = true>(??????)
Now the question is. What goes in the last part, the ?????? part.
I tried "name=DefaultConnection" but it gives me a runtime error with name being unsupported.
I can't seem to find any documentation explaining what goes there.
Update
Instaed of fixnig the issue I found this workaround.
https://fsprojects.github.io/FSharp.Configuration/
I don't get the purpose of ConnectionStringOrName if you have to supply the connection string anyway. Also why do you have to specify it twice. Makes very little sense to me :(
When using type providers, you often need two separate data sources:
Compile-time one that is used when you are editing the code and compiling the code. The type provider uses this connection or data source to infer the schema of the data - in case of SQL provider, this is connection to the database that is used to check that all your column names exist etc.
Run-time one is used when you actually run the program after it is deployed somewhere. This is where you'll read the actual data from.
The reason why you need two is that the run-time data source may be determined at runtime and it may not be accessible at compile-time (you typically have access to a dev database, but not to production database). The compile-time connection needs to be a constant, so that the provider can use it (when compiling your code) without running any part of your code.
In case of the SQL command provider:
type SelectCmd = SqlCommandProvider<"SELECT ...",
ConnectionStringOrName = "name=DefaultConnection",
SingleRow = true,
AllParametersOptional = true>
let cmd = new SelectCmd(??????)
"name=DefaultConnection" tells the provider to use an app.config key at compile-time
????? is where you need to specify the run-time connection string
To read the connection string from app.config, you can use the standard .NET methods like using ConfigurationManager:
open System.Configuration
let conn = ConfigurationManager.ConnectionStrings["DefaultConnection"].ConnectionString;
let cmd = new SelectCmd(conn)
Something like this would not work when passing the connection string to the SqlCommandProvider, because this needs to run some code to read the string and that's only possible at runtime. That's why the SQL command provider has a handy option to specify name=DefaultConnection as a special string.
I try to create a InterBase database on Delphi using TIBDatabase type.
ibdb:=TIBDatabase.Create(nil);
with ibdb do
begin
SQLDialect:=3;
DatabaseName:=Self.name;
Params.Clear;
Params.Add('USER "SYSDBA"');
Params.Add('PASSWORD "masterkey"');
Params.Add('PAGE_SIZE 4096');
LoginPrompt:=false;
try
CreateDatabase;
except on E: Exception do
ShowMessage('Can''t create database. '+E.Message);
end;
end;
Self.name is string like a 'localhost:C:\db.ib'.
Code works without errors. Database file created. But IBConsole doesn't show this database on IB server. How can I check that database was added on server?
The following works for me and avoids all the problems you've mentioned so far:
1 Start a new Delphi project
2 Drop a TIBDatabase onto the form
3 Add the following CreateDatabase procedure to the form
procedure TForm1.CreateDatabase;
begin
IBDatabase1.SQLDialect:=3;
IBDatabase1.DatabaseName := 'D:\aaad7\interbase\newdb.gdb'; //edit for
// your system & check that you have file-creation rights in the
// database folder you want to use.
IBDatabase1.Params.Clear;
IBDatabase1.Params.Add('USER ''SYSDBA'''); // Note that this line
// contains 5 single-quotes and no double quotes
// Also note that the omission of the usual '=' between the
// parameter name and value for the USER and PASSWORD is deliberate,
// because it's evidently required by the IB api
IBDatabase1.Params.Add('PASSWORD ''masterkey'''); // Ditto
IBDatabase1.LoginPrompt:=false;
try
IBDatabase1.CreateDatabase;
except on E: Exception do
ShowMessage('Can''t create database. '+E.Message);
end;
end;
4 Add a TButton and set its OnClick event to call CreateDatabase.
5 Compile & run and click Button1.
Using Delphi 7 and Interbase XE7, this code successfully creates the newdb.gdb database and this database stays on disk after the Delphi application terminates. I can then open the database in IBConsole using the procedure described below. I cannot delete the databse file via the windows shell while IBConsole is running but I can after I close it.
Do you get the same result and, if you do, does that solve your problem and if not, why not?
Btw, I've written the CreateDatabase code to avoid the use of "with", because it usually creates more problems than it avoids and I loathe it, and
to avoid the dynamic creation of the TIBDatabase component, because that serves no useful purpose.
Notes
Once you have created a database in code as above, there is no need to "attach" it to IBConsole.Exe or to the Interbase server in order to be able to use it.
You could set up an IBDatabase, IBTransaction and IBQuery to connect to it, and create tables in it by issuing "CREATE TABLE ..." SQL statements to it from the IBQuery.
If you want the database to be listed in the IBConsole utility, you can either
a) Add it manually using the procedure in Previous Answer below or
b) Add it in code to the IBConsole configuration file, IBConsole.XML, which you will find in C:\users[your user name]\Appdata\Roaming\Embarcadero\Interbase, using a Delphi XML access library of your choice. If you add it using the manual procedure below, then close IBConsole.Exe so that the config file is updated on disk, you'll be able to examine the XML file and find the node for the database under the Server/Databases node. It will look something like
<key name="Newdb">
<value name="Accessed">A32BFE2475B3E440</value>
<value name="CaseSensitiveRole">0</value>
<value name="CharacterSet"/>
<value name="Created">7DB1327474B3E440</value>
<value name="DatabaseFiles">D:\aaad7\Interbase\NEWDB.GDB</value>
<value name="Encrypted">0</value>
<value name="EUA Enabled">0</value>
<value name="Password">096e6376786a78726d82</value>
<value name="Role"/>
<value name="Save DBAlias">1</value>
<value name="SEP"/>
<value name="Use DBAlias">0</value>
<value name="Username">SYSDBA</value>
<value name="UseSSL">0</value>
</key>
Previous answer
Try this:
In IBConsole, click the Local Server node and select its Databases node
Right-click the Databases node and select Add
In the Add database and Connect pop-up, click the button to the RHS of the File: field
From there, you should be able to navigate to and select your new database.
Tested with Interbase XE7 IBConsole.
If that doesn't work for you, exactly which step fails and how?
It seems I find the answer. Code creates database file and add database on InterBase server. It can be checked by isql utility. Strange that IBConsole and IBExpert don't show all databases on server in current moment but only created by the same tool as I understand.
I have a WindowsXPSP3 op system, on it a DelphiXE and InterbaseXE installed.
I created a database in IB and it works OK through the IBConsole and ISQL and connection testing also works through TCP/IP localhost:3050.
Now I try to access it from Delphi.
I did:
var AC:tADOConnection;
...
AC:=tADOConnection(Self);
AC.ConnectionString:=
AC.Open;
I tried all possible version I could google for the ConnectionString, but all generated an error. I used various Provider= versions, etc., but none works.
Could someone provide me with a simple working ConnectionString? Do I need to install any ADO driver or similar additionally?
Thanks,
Zsolt
There are two ways to easily create a valid connection string
a.1) Click on the small button in the object inspector right of the connection string property.
a.2) Create your connection, test it, press OK
or
b.1) Create an empty file e.g. 'TEST.UDL'. Use Notepad.EXE for example.
b.2) Double click on the file in the explorer. This will open the connection string editor
b.3) Create your connection, test it. Press OK
b.4) Your file now contains the connection string which you may copy&past in your application
Another benefit of the second method is that you can even use the file as a connection string. This makes life alot easier if you have to configure your connection from time to time (Just double click on the UDL if you have to change the connection properties). Here's how a valid connection string for a file looks like:
FILE NAME=<Full path to your UDL file>
Hi this is my scenario,
I am trying to migrate an app from JBoss5 to JBoss7.
I am using jboss-as-7.1.1.Final.
The error I am getting is:
No EJB receiver available for handling [appName:,modulename:myapp-ejb,distinctname:] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext#6b9bb4bb
at org.jboss.ejb.client.EJBClientContext.requireEJBReceiver(EJBClientContext.java:584) [jboss-ejb-client-1.0.5.Final.jar:1.0.5.Final]
at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:119) [jboss-ejb-client-1.0.5.Final.jar:1.0.5.Final]
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:181) [jboss-ejb-client-1.0.5.Final.jar:1.0.5.Final]
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136) [jboss-ejb-client-1.0.5.Final.jar:1.0.5.Final]
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121) [jboss-ejb-client-1.0.5.Final.jar:1.0.5.Final]
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104) [jboss-ejb-client-1.0.5.Final.jar:1.0.5.Final]
I have looked at several discussions with the same error message but I just cant figure out what I am doing wrong.
In the deployments directory I have only one myapp.war. I do not deploy a .ear file. I have a dependency (myapp-ejb.jar) deployed as a module.
I have followed the instructions from https://docs.jboss.org/author/display/AS71/How+do+I+migrate+my+application+from+AS5+or+AS6+to+AS7 in section "Migrate EAP 5 Deployed Applications That Make Remote Invocations to AS 7".
SERVER
In the myapp-ejb.jar I have a bunch of JNDI names like:
public static final String ACCOUNT_REMOTE = "ejb:/myapp-ejb//AccountBean!com.company.myapp.ejb.account.AccountRemote";
The lookup is done from the client by invoking this static method which is defined in myapp-ejb.jar:
public static AccountRemote getAccountRemote() throws NamingException {
if (accountRemote == null){
InitialContext ic = new InitialContext();
Object ref = ic.lookup(JNDINames.ACCOUNT_REMOTE);
accountRemote = (AccountRemote) PortableRemoteObject.narrow(ref, AccountRemote.class);
}
return accountRemote;
}
All remote interfaces are for stateless EJB like:
#Stateless
#Remote(AccountRemote.class)
public class AccountBean implements AccountRemote {
CLIENT
From the myapp.war I make a remote invocation to the myapp-ejb.jar using the above static method getAccountRemote().
In the myapp.war/WEB-INF directory I have added a jndi.properties and a jboss-ejb-client.properties.
The jndi.properties contains only one value:
java.naming.factory.url.pkgs=org.jboss.ejb.client.naming
The jboss-ejb-client.properties contains:
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=default
remote.connection.default.host=localhost
remote.connection.default.port=4447
remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
I have removed the security realm on remoting from the standalone.xml:
<subsystem xmlns="urn:jboss:domain:remoting:1.1">
<connector name="remoting-connector" socket-binding="remoting" />
</subsystem>
I have added the JBOSS_HOME/bin/client/jboss-client.jar to the myapp.war/WEB-INF/lib.
The application deploys successfully without any errors but when I launch localhost:8080/ I get the No EJB receiver available for handling error.
Does anyone knows what I have missed? Any suggestions?
"EJB client API approach" for remote EJB invocation from one node to another node in clustered JBOSS:
------------------------------------------------------------------------------------
1. To call EJB from remote location we need to enable "remoting-ejb-receiver" on server side.
Please refer to “standalone_changes.xml” to know change details.
2. Also we need to register the "remoting-ejb-receiver" to the application, so that the application can receive remote EJB.
Please refer to “jboss-ejb-client.xml” section.
3. Now we need to call remote EJB in "EJB client API approach" way, which needs to have JNDI name pattern as:
ejb:<app-name>/<module-name>/<distinct-name>/<bean-name>!<fullclassname-of-the-remote-interface>
In our case it will be: ejb:myapp-ejb//node1/AccountBean!com.company.myapp.ejb.account.AccountRemote
Important to note that identification to remote location IP address is not based on InitialContext as InitialContext will not contain any IP address as normally happens with "remote://URL:Port".
The remote location identification is based on <distinct-name> passed in JNDI. JBOSS will internally identify the remote IP based on <distinct-name>.
Hence is required to provide unique <distinct-name> to the application running on different nodes.
Add "<distinct-name>${jboss.node.name}</distinct-name>" to “jboss-app.xml”. Make sure that jboss.node.name property is always unique.
But then jboss.node.name should be added as environmental property while server startup.
For test purpose we can provide hardcoded value like:
For node1: "<distinct-name>node1</distinct-name>" to “jboss-app.xml”.
For node2: "<distinct-name>node2</distinct-name>" to “jboss-app.xml”.
standalone_changes.xml:
------------------------------------------------------------------------------------
<subsystem xmlns="urn:jboss:domain:remoting:1.1">
<outbound-connections>
<remote-outbound-connection name="remote-ejb-connection-host2" outbound-socket-binding-ref="remote-ejb-host2" username="xxx" security-realm="ejb-security-realm">
<properties>
<property name="SASL_POLICY_NOANONYMOUS" value="false"/>
<property name="SSL_ENABLED" value="false"/>
</properties>
</remote-outbound-connection>
...........
...........
</outbound-connections>
</subsystem>
...........
...........
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
<outbound-socket-binding name="remote-ejb-host2">
<remote-destination host="${jboss.ejb.host2}" port="${jboss.ejb.host2.port}"/>
</outbound-socket-binding>
...........
...........
</socket-binding-group>
...........
...........
<management>
<security-realms>
<security-realm name="ejb-security-realm">
<server-identities>
<secret value="${jboss.ejb.remoting.password}"/>
</server-identities>
</security-realm>
...........
...........
</security-realms>
</management>
jboss-app.xml:
------------------------------------------------------------------------------------
<?xml version="1.0" encoding="UTF-8"?>
<jboss-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.jboss.com/xml/ns/javaee">
<distinct-name>node1</distinct-name>
<security-domain>xyz</security-domain>
<unauthenticated-principal>guest</unauthenticated-principal>
<library-directory>lib</library-directory>
</jboss-app>
jboss-ejb-client.xml
------------------------------------------------------------------------------------
<jboss-ejb-client xmlns="urn:jboss:ejb-client:1.0">
<client-context>
<ejb-receivers>
<remoting-ejb-receiver outbound-connection-ref="remote-ejb-connection-host2"/>
<!-- <remoting-ejb-receiver outbound-connection-ref="${jboss.remote.outbound.connection.host3}"/> -->
</ejb-receivers>
</client-context>
</jboss-ejb-client>
For more details refer to:
"https://docs.jboss.org/author/display/AS71/Remote+EJB+invocations+via+JNDI+-+EJB+client+API+or+remote-naming+project" which tells us different way of remote EJB location.
You don't actually require jboss-client.jar if you are using JBOSS as your App server. Please add the following property to your initialContext or jndi.propeties file everything would be fine.
jboss.naming.client.ejb.context=true
also please remove the property unless you are calling from a standalone client or server other than Jboss.
java.naming.factory.url.pkgs=org.jboss.ejb.client.naming
Try with these settings.