What needs to be done to enable pooling in a Delphi 7 app? My connection string is:
Provider=SQLOLEDB.1;Initial Catalog=%s;Data Source=%s;Password=%s;User ID=%s;OLE Db Services=-1
I can tell that connection pooling is not being achieved by looking at the SQLServer:GeneralStatistics UserConnections performance counter - it fluctuates wildly when my application runs. With connection pooling I'd expect it to achieve a steady state. Also, I see that Logins/sec and Logouts/sec counters are both very high - if connection pooling were used Logouts/sec would be at or near zero.
In searching I found this article on resource pooling:
http://www.ddj.com/database/184416942
It suggests that "If you are working at the OLEDB SDK (or COM) level using ATL, you have to write some more code" (aside from adding OLE Db Services=-1 to the connection string) to get connection pooling:
CDataSource db;
CDBPropSet dbinit(DBPROPSET_DBINIT);
dbinit.AddProperty(DBPROP_AUTH_USERID, "MyName);
dbinit.AddProperty(DBPROP_INIT_DATASOURCE, "MyServer);
dbinit.AddProperty(DBPROP_INIT_CATALOG, "MyDb );
dbinit.AddProperty(DBPROP_INIT_PROMPT, (short)4);
dbinit.AddProperty(DBPROP_INIT_LCID, (long)1033);
dbinit.AddProperty(DBPROP_INIT_OLEDBSERVICES, (long)DBPROPVAL_OS_ENABLEALL);
HRESULT hr = db.OpenWithServiceComponents(_T("sqloledb"), &dbinit);
Unfortunately that code is Greek to me and I'm not sure how to translate that to Delphi (or if its even necessary).
I'm also careful not to change the connection string at all. Any suggestions on what else I might need to do to enable resource pooling?
You need to keep one instance of the connection open at all times...if it drops to zero, then ADO will re-establish the connection to authenticate the user.
You don't mention it, but are you using Delphi's ADO implementation (dbGo for Delphi 7, IIRC) for your data access? If so, are you connecting everything through the same TADOConnection? If so, it should be doing the pooling for your application (meaning that one running copy of your application is using one connection to the DB server).
Related
I work with Firebird and Delphi, I want to implement access via internet with wirecompression;
But I am unable to activate it.
I have followed the steps inside this document for the new parameter(one of the few I was able to find)
How to enable WireCompression on Firebird 3.0 using FireDAC
In the tests I use
Windows server 2012 R2
Firebird : Firebird-3.0.4.33054_0_Win32(32 bits)
Also copied to executable folder.
fbclient.dll
zlib1.dll (idem server and client)
created firebird.conf with wirecompression=true.
and I am giving wirecompression=true inside the Firedac of the application.
Why am I unable to activate the P15:CZ compression ?
Sending connection info for the example:
================================
Connection definition parameters
================================
DriverID=FB
Database=miservidor001:C:\sysdat\C100\gestdat03.fdb
User_Name=SYSDBA
PassWord=*****
WireCompression=true
================================
FireDAC info
================================
Tool = RAD Studio 10.2
FireDAC = 16.0.0 (Build 88974)
Platform = Windows 32 bit
Defines = FireDAC_NOLOCALE_META;FireDAC_MONITOR
================================
Client info
================================
Loading driver FB ...
Brand = Firebird
Client version = 300049900
Client DLL name = C:\APPS\WC01\fbclient.dll
================================
Session info
================================
Current catalog =
Current schema =
Server version = WI-V3.0.4.33054 Firebird 3.0
WI-V3.0.4.33054 Firebird 3.0/tcp (WIN-2012LAGO003)/P15:C
WI-V3.0.4.33054 Firebird 3.0/tcp (nucleo)/P15:C'
NOTE: I don't know Delphi nor FireDAC, this answer is based on the general behavior of Firebird and my experience with maintaining its JDBC driver (Jaybird). So it is possible that there is a better answer specifically for FireDAC/Delphi.
Enabling or disabling wire compression is entirely determined by the client, not by the server. This means that configuration of the server is not necessary nor has it any effect, except in cases where the server itself acts as a client, for example with execute statement ... on external datasource.
To be able to use wire compression, you need three things:
fbclient.dll
zlib1.dll (in the same location as fbclient.dll, or on the search path)
A configuration to enable wire compression for the client
Point 3 is likely your problem: I'm not sure if FireDAC has a connection property WireCompression that actually enables wire compression.
I know of two ways to enable wire compression for the client:
Create a firebird.conf in the same directory as the fbclient.dll used by your application. In this configuration file, put the requested configuration options (one per line):
WireCompression = true
# maybe other config lines (eg AuthClient, WireCrypt, etc)
Instead of creating a firebird.conf file, pass the configuration (with linebreaks separating config options) in the isc_dpb_config (int 87) database parameter item.
The value is the same as the content of the firebird.conf file in the previous option. This may run into size issues if the client is using the old database parameter buffer format (where strings are max 255 bytes) and you want to pass (a lot) more config options.
Option 1 is probably the simplest and will work for all frameworks. Option 2 depends on whether or not the framework or driver exposes the database parameter buffer or if it has a connection property that maps to isc_dpb_config.
For example in Java using Jaybird, you can enable compression (only when using native connections) using:
Properties props = new Properties();
props.setProperty("user", "sysdba");
props.setProperty("password", "masterkey");
props.setProperty("config", "WireCompression=true");
try (var connection = DriverManager.getConnection(
"jdbc:firebirdsql:native:localhost:D:/data/db/fb3/fb3testdatabase.fdb", props)) {
FirebirdConnection fbCon = connection.unwrap(FirebirdConnection.class);
FbDatabase fbDatabase = fbCon.getFbDatabase();
System.out.println(fbDatabase.getServerVersion());
} catch (SQLException e) {
e.printStackTrace();
}
This prints out WI-V3.0.4.33054 Firebird 3.0,WI-V3.0.4.33054 Firebird 3.0/tcp (host)/P15:CZ,WI-V3.0.4.33054 Firebird 3.0/tcp (host)/P15:CZ (note this is <server version>,<server protocol info>,<client protocol info>). The Z in P15:CZ means that the connection is zlib compressed (the C that the connection is encrypted).
Here, the config property is an alias for isc_dpb_config.
Mark's answer is the best (and probably the only) source of information about this problem in the entire internet. Good luck finding anything on Delphi, FireDAC or Firebird documentation about what he said.
Based on his answer, here is what you need to use Firebird wire compression with FireDAC:
You need Delphi Rio 10.3.1 (Update 1) or later. Only in this version the config low level parameter (see below) was added to FireDAC.
You must pass WireCompression=true to the low level config connection parameter. This is NOT TFDConnection.Params (high level).
To accomplish this you need to set the IBAdvanced property of TFDPhysFBConnectionDefParams to config=WireCompression=true (yeah! Go figure it!)
Code:
FDConnection1.DriverName := 'FB';
with FDConnection1.Params as TFDPhysFBConnectionDefParams do
begin
Server := '...';
Database := '...';
UserName := '...';
Password := '...';
IBAdvanced := 'config=WireCompression=true';
end;
FDConnection1.Connected := True;
Using a connection definition file:
[FB_Demo]
DriverID=FB
Server=...
Database=...
User_Name=...
Password=...
IBAdvanced=config=WireCompression=true
You need zlib1.dll in the same path of your fbclient.dll. The catch here is that Firebird distribution DOES NOT have the 32-bit version of zlib1.dll in its C:\Program Files\Firebird\Firebird_3_0\WOW64 folder. So:
If your application is 64-bit you are probably fine. Just use both fbclient.dll and zlib1.dll from your C:\Program Files\Firebird\Firebird_3_0 folder.
If your application is 32-bit you have to download the 32-bit version of zlib1.dll from the 32-bit Firebird distribution. Use it together with the fbclient.dll you find in your C:\Program Files\Firebird\Firebird_3_0\WOW64 (which contains 32-bit libraries).
In Firebird 3.0.4 or later you can use the WIRE_COMPRESSED context variable to check if the connection was established as you expected:
SELECT
RDB$GET_CONTEXT('SYSTEM', 'WIRE_COMPRESSED') wire_compressed
FROM
rdb$database
This will return TRUE if the current connection is compressed.
I'm trying to process large amount of data using Python and maintaining processing status in MySQL. However, I'm surprised there is no standard connection pool for python-mysql (like HikariCP in Java).
I initially started with PyMySQL, things were great until the program ran for first few hours. After few hours, things started to fail. I was getting lot of errors like:
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on '127.0.0.1' ([Errno 99] Cannot assign requested address)")
Moreover, lot of ports were stuck in TIME_WAIT state because I'm opening and closing connections too frequently because of lack of connection pooling
/d/p/950 ❯❯❯ netstat -nt | wc -l
84752
Per this and this, I tried to set tcp_fin_timeout and ip_local_port_range, but hardly anything improved.
echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 15000 65000 > /proc/sys/net/ipv4/ip_local_port_range
Then I found out that MySQL provides mysql.connector which comes with pooling functionality. After doing all that performance actually deteriorated. More processes started to get failed. I'm using Python's multiprocessing module to simultaneously run 29 processes(multiprocessing.Pool picked this no by default) on a 24 core machine. Following was the code, of course I was using .my.cnf to pass all the credential to avoid committing them to git :
import mysql.connector
from mysql.connector import pooling
conn_pool = pooling.MySQLConnectionPool(pool_name="mypool1",
pool_size=pooling.CNX_POOL_MAXSIZE,
option_files=MYSQL_CONFIG,
option_groups=MYSQL_GROUP_NODE1,
allow_local_infile=True)
conn = conn_pool.get_connection()
Finally, reverted back to old code. Still using PyMySQL and though errors are less frequent it is still causing a major problem. I looked at SQLAlchemy and couldn't really found much of a documentation around pooling.
I'm wondering how's everyone else dealing with mysql-python connection pooling issue? I really believe there should be something out there so that I don't have to reinvent the wheel.
Any pointers are much appreciated.
DBUtils implements MySQL (and generally claims to support abritrary DB-API 2 compliant database interfaces) user-sized connection pool PooledDB, thead-mapped pool PersistentDB and SteadyDB (see functionality section). The latter should fit your case where multiprocessing.Pool creates worker processes with managed persistent database connection each. It is described as:
DBUtils.SteadyDB is a module implementing "hardened" connections to a database, based on ordinary connections made by any DB-API 2 database module. A "hardened" connection will transparently reopen upon access when it has been closed or the database connection has been lost or when it is used more often than an optional usage limit.
You can use it with PyMySQL like:
import pymysql
from DBUtils.SteadyDB import connect
db = connect(
creator = pymysql, # the rest keyword arguments belong to pymysql
user = 'guest', password = '', database = 'name',
autocommit = True, charset = 'utf8mb4',
cursorclass = pymysql.cursors.DictCursor)
Also see this related question for more examples.
I am using WAS and DB2 and my Application is coded in Java.
If I create a connection pooling in websphere Application server, then do I need to change anything in Java code? or Websphere will handle all connection pooling concept?
To take advantage of connection pooling provided by WebSphere Application Server, you need to obtain connections from data source. First configure the data source and assign a jndiName to it.
Then you can use resource injection to define a resource reference for it and inject into a Java EE component, for example,
#Resource(lookup = "jdbc/ds1", name = "java:comp/env/jdbc/ds1ref")
DataSource ds1;
or look it up in JNDI, for example,
DataSource ds = InitialContext.doLookup("java:comp/env/jdbc/ds1ref");
Always make sure to close connections that obtain from the data source when you are done with them so they can go back to the pool,
Connection con = ds.getConnection();
try {
...
} finally {
con.close();
}
If your application needs to access any JDBC vendor APIs (not part of the JDBC specification), use the JDBC wrapper pattern (unwrap method). For example,
OracleConnection oraCon = con.unwrap(OracleConnection.class);
Other than that, usage of connection pooling should be pretty much transparent.
I have a problem with changing the standard options used by an Axis 1.4 generated web service client code.
We consume a certain web service of a partner who is using the old RPC/Encoded style, which basically means we're not able to go for Axis 2 but are limited to Axis 1.4.
The service client is retrieving data from the remote server through our proxy which actually runs quite nicely.
Our application is deployed as a servlet. The retrieved response of the foreign web service is inserted into a (XML) document we provide to our internal systems/CMS.
But if the external service is not responding - which didn't happen yet but might happen at anytime - we want to degrade nicely and return our produced XML document without the calculated web service information within a resonable time.
The data retrieved is optional (if this specific calculation is missing it isn't a big issue at all).
So I tried to change the timeout settings. I did apply/use all methods and keys I could find in the documentation of axis to alter the connection and socket timeouts by searching the web.
None of these seems to influence the connection timeouts.
Can anyone give me advice how to alter the settings for an axis stub/service/port based on version 1.4?
Here's an example for the several configurations I tried:
MyService service = new MyServiceLocator();
MyServicePort port = null;
try {
port = service.getMyServicePort();
javax.xml.rpc.Stub stub = (javax.xml.rpc.Stub) port;
stub._setProperty("axis.connection.timeout", 10);
stub._setProperty(org.apache.axis.client.Call.CONNECTION_TIMEOUT_PROPERTY, 10);
stub._setProperty(org.apache.axis.components.net.DefaultCommonsHTTPClientProperties.CONNECTION_DEFAULT_CONNECTION_TIMEOUT_KEY, 10);
stub._setProperty(org.apache.axis.components.net.DefaultCommonsHTTPClientProperties.CONNECTION_DEFAULT_SO_TIMEOUT_KEY, 10);
AxisProperties.setProperty("axis.connection.timeout", "10");
AxisProperties.setProperty(org.apache.axis.client.Call.CONNECTION_TIMEOUT_PROPERTY, "10");
AxisProperties.setProperty(org.apache.axis.components.net.DefaultCommonsHTTPClientProperties.CONNECTION_DEFAULT_CONNECTION_TIMEOUT_KEY, "10");
AxisProperties.setProperty(org.apache.axis.components.net.DefaultCommonsHTTPClientProperties.CONNECTION_DEFAULT_SO_TIMEOUT_KEY, "10");
logger.error(AxisProperties.getProperties());
service = new MyClimateServiceLocator();
port = service.getMyServicePort();
}
I assigned the property changes before the generation of the service and after, I set the properties during initialisation, I tried several other timeout keys I found, ...
I think I'm getting mad about that and start to forget what I tried already!
What am I doing wrong? I mean there must be an option, mustn't it?
If I don't find a proper solution I thought about setting up a synchronized thread with a timeout within our code which actually feels quite awkward and somehow silly.
Can you imagine anything else?
Thanks in advance
Jens
axis1.4 java client soap wsdl2java rpc/encoded xml servlet generated alter change setup stub timeout connection socket keys methods
I think it may be a bug, as indicated here:
https://issues.apache.org/jira/browse/AXIS-2493?jql=text%20~%20%22CONNECTION_DEFAULT_CONNECTION_TIMEOUT_KEY%22
Typecast service port object to org.apache.axis.client.Stub.
(i.e)
org.apache.axis.client.Stub stub = (org.apache.axis.client.Stub) port;
Then set all the properties:
stub._setProperty(org.apache.axis.client.Call.CONNECTION_TIMEOUT_PROPERTY, 10);
stub._setProperty(org.apache.axis.components.net.DefaultCommonsHTTPClientProperties.CONNECTION_DEFAULT_CONNECTION_TIMEOUT_KEY, 10);
stub._setProperty(org.apache.axis.components.net.DefaultCommonsHTTPClientProperties.CONNECTION_DEFAULT_SO_TIMEOUT_KEY, 10);
Before inserting filestream data I'd like to check the following NTFS settings:
1) 8.3 naming status (this is disabled by using fsutil behavior set disable8dot3 1)
2) last access status (this is disabled by using fsutil behavior set disablelastaccess 1)
3) cluster size (this is set with format F: /FS:NTFS /V:MyFILESTREAMContainer /A:64K)
The filestream recomendation is to disable (1) and (2) and to set (3) at 64kb.
But before setting this I'd like to know the existing settings. How do I check this? Answer can be in Delphi but not necessarly.
The GetDiskFreeSpace Windows API call returns the sector_per_cluster and bytes_per_sector values. I think this function should be in Windows unit.
You can read the registry for points 1 and 2 (using xp_regread in SQL)
Number 3 is not essential but helps and has been SQL Server best practice for a decade or more. You'd have to use sp_OA% or a CLR function to read this in SQL.