I'm making a web app that needs to load and save UTF-8 (Korean, specifically) characters from a DB. I've been given an account on the Oracle 10g server, but it saves VARCHAR2 type columns as ASCII7, with each UTF-8 character taking 2 VARCHAR2 slots.
I assumed that since iBatis is writing in the same way that it is reading, if I treat everything from input to output as UTF-8 I will have no problems, but any Korean characters I input come out garbled.
Is there a way to do this properly without messing up the (someone else's) DB?
Further information:
I've previously been able to load Korean strings using:
ResultSet rs = ps.executeQuery();
String koreanString = new String(rs.getBytes("colname"), "euc-kr");
And write Korean strings to db using:
PreparedStatement ps = conn.prepareStatement(sql);
ps.setString(1, new String(koreanString.getBytes("euc-kr"), "ISO-8859-1"));
Attempts to change the JDBC connection url result in this message:
Description
Listener refused the connection with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor
The Connection descriptor used by the client was:
[ip]:myTablespace?useUnicode=true&characterEncoding=UTF-8
error dump
javax.servlet.ServletException: Listener refused the connection with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor
The Connection descriptor used by the client was:
[ip]:myTablespace?useUnicode=true&characterEncoding=UTF-8
at jeus.servlet.jsp2.runtime.PageContextImpl.doHandlePageException(PageContextImpl.java:859)
at jeus.servlet.jsp2.runtime.PageContextImpl.handlePageException(PageContextImpl.java:789)
at jeus_jspwork._jsp._500_managerAdmin_5fjsp._jspService(_500_managerAdmin_5fjsp.java:452)
at jeus.servlet.jsp2.runtime.HttpJspBase.service(HttpJspBase.java:95)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at jeus.servlet.jsp.JspServletWrapper.executeServlet(JspServletWrapper.java:147)
at jeus.servlet.servlets.JspServlet.execute(JspServlet.java:365)
at jeus.servlet.engine.HttpRequestProcessor.run(HttpRequestProcessor.java:284)
root cause
java.sql.SQLException: Listener refused the connection with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor
The Connection descriptor used by the client was:
[ip]:myTablespace?useUnicode=true&characterEncoding=UTF-8
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:261)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:441)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
at java.sql.DriverManager.getConnection(DriverManager.java:525)
at java.sql.DriverManager.getConnection(DriverManager.java:171)
at com.ibatis.common.jdbc.SimpleDataSource.popConnection(SimpleDataSource.java:580)
at com.ibatis.common.jdbc.SimpleDataSource.getConnection(SimpleDataSource.java:222)
at com.ibatis.sqlmap.engine.transaction.jdbc.JdbcTransaction.init(JdbcTransaction.java:48)
at com.ibatis.sqlmap.engine.transaction.jdbc.JdbcTransaction.getConnection(JdbcTransaction.java:89)
at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryForObject(MappedStatement.java:120)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(SqlMapExecutorDelegate.java:518)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(SqlMapExecutorDelegate.java:493)
at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForObject(SqlMapSessionImpl.java:106)
at com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.queryForObject(SqlMapClientImpl.java:82)
at [].admRole.getCount(admRole.java:44)
at jeus_jspwork._jsp._500_managerAdmin_5fjsp._jspService(_500_managerAdmin_5fjsp.java:145)
at jeus.servlet.jsp2.runtime.HttpJspBase.service(HttpJspBase.java:95)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at jeus.servlet.jsp.JspServletWrapper.executeServlet(JspServletWrapper.java:147)
at jeus.servlet.servlets.JspServlet.execute(JspServlet.java:365)
at jeus.servlet.engine.HttpRequestProcessor.run(HttpRequestProcessor.java:284)
As I stated in the question, strings are stored and retrieved correctly if they are re-encoded as EUC-KR before being turned into ISO-8859-1 (to save, or vice versa to retrieve).
I modified the two following classes:
com.ibatis.sqlmap.engine.mapping.parameter.ParameterMap
com.ibatis.sqlmap.engine.mapping.result.ResultMap
In both cases, I took the Object[] array (parameters and columnValues), casted to String, and applied the encoding transformations.
I am not using oracle for quite a while, but I have some confidence that this is reason of your "listener does not currently know of SID given" error: Can I force JDBC Driver use UTF-8 Charset to encode?
I am a Chinese developer so the character encoding problem is pretty much the same(we are mostly using GBK character set here). As far as I can remember, "but it saves VARCHAR2 type columns as ASCII7" means that your oracle instance is a non-unicode installation?
The force use of string.getBytes(charset) above JDBC layer is really really bad in terms of maintenance and data interpretability(the string data is displayed as a mess to DBA; DBA can not use SQL to perform any string comparison on this column, etc). So my advice is try to contact your DBA and get the database working with unicode first, since Oracle is very capable of handling unicode data.
Related
I have a stored procedure created in MySQL DB. This database is accessed as linked server through Microsoft SQL Server 2012 using Microsoft SQL Server Management Studio. The provider selected while creating linked server is "Microsoft OLE DB Provider for ODBC Drivers".
When I run following as text from Report builder 3.0 it runs fine and fetch data.
EXEC('CALL storedProcedureName(''string1'', ''string2'', ''string3'')') AT LinkedServerName;
But when I try to replace string1, string2, string3 with parameter name parameter1, parameter2, parameter3 as:
EXEC('CALL storedProcedureName(#parameter1, #parameter2, #parameter3)') AT LinkedServerName;
I get error:
Could not execute statement on remote server 'LinkedServerName'.
(Microsoft SQL Server, Error: 7215)
And when I try:
EXEC('CALL storedProcedureName('#parameter1', '#parameter2', '#parameter3')') AT LinkedServerName;
I get the prompt to enter values for parameter1, parameter2, parameter3. But when I enter the values and click ok, I get error:
Incorrect syntax near '#parameter1'. (Microsoft SQL Server, Error: 102)
Question: Am I missing something in syntax or is this a bug?
The linked server has:
"RPC" and "RPC out" set to True.
And the OLEDB provider has:
Enabled allow inprocess
Enabled dynamic parameter
I believe you have to call it like the below. So params become strings wrapped in single quotes:
EXEC('CALL storedProcedureName('''+#parameter1+''', '''+#parameter2+''', '''+#parameter3+''')') AT LinkedServerName;
I know this is an older question but the accepted answer is open to SQL Injection and I think it's important to offer a more secure method.
You really want to use a parameterized query
EXEC('CALL storedProcedureName(?,?,?)',#parameter1, #parameter2, #parameter3) AT LinkedServerName;
I know this works for oracle and it should work for MySql as well.
After many tries I have concluded that the optimal way to transfer with SSIS data from AS400 (non-unicode) to SQL Server is:
Use native transfer utility to dump data to tsv (tab delimited)
Convert files from utf-8 to unicode
Use bulk insert to put them into SQL Server
In #2 step I have found a ready made code that does this:
string from = #"\\appsrv02\c$\bg_f0101.tsv";
string to = #"\\appsrv02\c$\bg_f0101.txt";
using (StreamReader reader = new StreamReader(from, Encoding.UTF8, false, 1000000))
using (StreamWriter writer = new StreamWriter(to, false, Encoding.Unicode, 1000000))
{
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
if (line.Length > 0)
writer.WriteLine(line);
}
}
I need to fully understand what is happening here with the encoding and why this is necessary.
How can I replace this script task with a more elegant solution?
I don't have much insight into exactly why you need the utf-8 conversion task, except to say that SQL server - I believe - uses UCS-2 as its native storage format, and this is similiar to UTF-16 which is what your task converts the file to. I'm surprised SSIS can't work with a UTF-8 input source though.
My main point is to answer the "How could I replace this script task with a more elegant solution?":
I have had a lot of success using HiT OLEDB/400 Server. It allows you to set up your AS/400 / iSeries / System i / whatever IBM are calling it this week as a linked server in SQL server, and you can then access the 400's data directly from the server its linked to using the standard 4 part SQL syntax, e.g. SELECT * FROM my400.my400.myLib.myFile.
Or even better, it's much more efficient as a passthrough query using EXEC...AT.
Using this you would not need SSIS at all, you'd just need a simple stored proc with that does an insert into your destination table direct from the 400 data.
Dim NorthWindOledbConnection As String = "Provider=SQLOLEDB;DataSOurce=SARAN-PC\SQLEXPRESS;Integrated Security=ssp1;InitialCatalog=Sara"
Dim rs As New ADODB.Recordset()
rs.Open("select * from SecUserPassword", NorthWindOledbConnection, ADODB.CursorTypeEnum.adOpenStatic, ADODB.LockTypeEnum.adLockBatchOptimistic)
i tried to run this above code in visual studio 2008 - it shows the following error:
"Multiple-step OLE DB operation generated errors. Check each OLE DB status value, if available. No work was done"
Firstly, don't use ADO in VB.NET. Use ADO.NET.
Other than that, create a proper Connection object instead of passing around a string.
And fix your connection string. It's SSPI, not SSP1. And it's Data Source, not DataSOurce. And it's Initial Catalog, not InitialCatalog.
You are using a very very very old way to access a Database that has been used with Visual Basic 6 and older.
Check to use ADO.NET instead of old ADO. For example you can use this code that is "similar" to the code you are using (but is not the best way to access the data on VS2008)
OleDbConnection con= New OleDbConnection( **Your Connection String** )
con.Open()
Dim command As OleDbCommand = New OleDbCommand("select * from SecUserPassword", con)
sqlCommand .CommandType = CommandType.Text
Dim reader As OleDbDataReader = TheCommand.ExecuteReader()
While reader.Read()
System.Console.Write(reader(** Your Table Field Name** ).ToString())
End While
con.Close()
To view how to create a correct connection String see the site http://www.connectionstrings.com/
If you want to access to an SQL Server database also you can use the SQLClient namespace instead the OleDb. For example System.Data.SqlClient.SqlConnection instead the OleDbConnection to provide better performance for SQL Server
The link below is an article that gives a great breakdown of the 6 scenarios this error message can occur:
Scenario 1 - Error occurs when trying to insert data into a database
Scenario 2 - Error occurs when trying to open an ADO connection
Scenario 3 - Error occurs inserting data into Access, where a fieldname has a space
Scenario 4 - Error occurs inserting data into Access, when using adLockBatchOptimistic
Scenario 5 - Error occurs inserting data into Access, when using Jet.OLEDB.3.51 or ODBC driver (not Jet.OLEDB.4.0)
Scenario 6 - Error occurs when using a Command object and Parameters
http://www.adopenstatic.com/faq/80040e21.asp
Hope it may help others that may be facing the same issue.
I'm receiving messages from a JMS MQ queue which are supposedly utf-8 encoded. However on reading the out using msgText = ((TextMessage)msg).getText();
I get question marks where non standard characters were present. It seems possible to specify the encoding when using a bytemessage, but I cant find a way to specify encoding while reading out the TextMessage. Is there a way to solve this, or should I press for bytemessages?
We tried adding Dfile.encoding="UTF-8" to Websphere's jvm and we added
source = new StreamSource(new ByteArrayInputStream(
((TextMessage) msg).getText().getBytes("UTF-8")));
In our MessageListener. This worked for us, so then we took out the Dfile.encoding bit away and it still works for us.
Due to preferred minimum configuration for Websphere we decided to leave it this way, also taking into account that we may easier switch the UTF-8 string by a setting from file or database.
If the text is not decoded correctly, then probably the client is not sending the message with the utf-8 codec; this should work:
byte[] by = ((TextMessage) msg).getText().getBytes("ISO-8859-1");
String text = new String(by,"UTF-8");
What needs to be done to enable pooling in a Delphi 7 app? My connection string is:
Provider=SQLOLEDB.1;Initial Catalog=%s;Data Source=%s;Password=%s;User ID=%s;OLE Db Services=-1
I can tell that connection pooling is not being achieved by looking at the SQLServer:GeneralStatistics UserConnections performance counter - it fluctuates wildly when my application runs. With connection pooling I'd expect it to achieve a steady state. Also, I see that Logins/sec and Logouts/sec counters are both very high - if connection pooling were used Logouts/sec would be at or near zero.
In searching I found this article on resource pooling:
http://www.ddj.com/database/184416942
It suggests that "If you are working at the OLEDB SDK (or COM) level using ATL, you have to write some more code" (aside from adding OLE Db Services=-1 to the connection string) to get connection pooling:
CDataSource db;
CDBPropSet dbinit(DBPROPSET_DBINIT);
dbinit.AddProperty(DBPROP_AUTH_USERID, "MyName);
dbinit.AddProperty(DBPROP_INIT_DATASOURCE, "MyServer);
dbinit.AddProperty(DBPROP_INIT_CATALOG, "MyDb );
dbinit.AddProperty(DBPROP_INIT_PROMPT, (short)4);
dbinit.AddProperty(DBPROP_INIT_LCID, (long)1033);
dbinit.AddProperty(DBPROP_INIT_OLEDBSERVICES, (long)DBPROPVAL_OS_ENABLEALL);
HRESULT hr = db.OpenWithServiceComponents(_T("sqloledb"), &dbinit);
Unfortunately that code is Greek to me and I'm not sure how to translate that to Delphi (or if its even necessary).
I'm also careful not to change the connection string at all. Any suggestions on what else I might need to do to enable resource pooling?
You need to keep one instance of the connection open at all times...if it drops to zero, then ADO will re-establish the connection to authenticate the user.
You don't mention it, but are you using Delphi's ADO implementation (dbGo for Delphi 7, IIRC) for your data access? If so, are you connecting everything through the same TADOConnection? If so, it should be doing the pooling for your application (meaning that one running copy of your application is using one connection to the DB server).