SQL loader not loading all columns in some environments - sqlplus

I am facing a very weird problem, and the problem is related to SQL Loader. It loads all columns in some databases, while it skips the last column in other databases. And yes, the target table structure is identical among all databases.
Here is how csv file file looks like:
OPTIONS (ERRORS=50, DIRECT=TRUE, SKIP=1)
LOAD DATA
INFILE *
REPLACE
INTO TABLE STAGING.PRODUCTS
FIELDS TERMINATED BY ';'
TRAILING NULLCOLS (
"PRODUCT_ID",
"PRODUCT_NAME",
"CLIENT_TECHNOLOGY")
BEGINDATA
1;Product1;N/A
2;Product2;N/A
....
100;Product100;N/A
and this is the log file, and this is from dev01. This is what I expect, aka all data loaded successfully:
SQL*Loader: Release 11.2.0.3.0 - Production on Thu Dec 8 15:29:21 2016
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
Control File: products.csv
Data File: products.csv
Bad File: products.bad
Discard File: none specified
(Allow all discards)
Number to load: ALL
Number to skip: 1
Errors allowed: 50
Continuation: none specified
Path used: Direct
Table STAGING.PRODUCTS, loaded from every logical record.
Insert option in effect for this table: REPLACE
TRAILING NULLCOLS option in effect
Column Name Position Len Term Encl Datatype
------------------------------ ---------- ----- ---- ---- ---------------------
Message 3053 not found; product=RDBMS; facility=UL
Message 3054 not found; product=RDBMS; facility=UL
"PRODUCT_ID" FIRST * ; CHARACTER
"PRODUCT_NAME" NEXT * ; CHARACTER
"CLIENT_TECHNOLOGY" NEXT * ; CHARACTER
Table STAGING.PRODUCTS:
100 Rows successfully loaded.
0 Rows not loaded due to data errors.
0 Rows not loaded because all WHEN clauses were failed.
0 Rows not loaded because all fields were null.
On the other hand, say dev12 has different log. The last column is totally skipped the town.
SQL*Loader: Release 11.2.0.3.0 - Production on Thu Dec 8 15:29:21 2016
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
Control File: products.csv
Data File: products.csv
Bad File: products.bad
Discard File: none specified
(Allow all discards)
Number to load: ALL
Number to skip: 1
Errors allowed: 50
Continuation: none specified
Path used: Direct
Table STAGING.PRODUCTS, loaded from every logical record.
Insert option in effect for this table: REPLACE
TRAILING NULLCOLS option in effect
Column Name Position Len Term Encl Datatype
------------------------------ ---------- ----- ---- ---- ---------------------
Message 3053 not found; product=RDBMS; facility=UL
Message 3054 not found; product=RDBMS; facility=UL
"PRODUCT_ID" FIRST * ; CHARACTER
"PRODUCT_NAME" NEXT * ; CHARACTER
Table STAGING.PRODUCTS:
100 Rows successfully loaded.
0 Rows not loaded due to data errors.
0 Rows not loaded because all WHEN clauses were failed.
0 Rows not loaded because all fields were null.
Any idea what can be the reason behind it? The target table is the same in all environments.
create table STAGING.PRODUCTS
(
product_id VARCHAR2(64) not null,
product_name VARCHAR2(64) not null,
client_technology VARCHAR2(64)
)
Thanks in advance :-)

You may have bigger issues at play. Notice these lines in both log files:
Message 3053 not found; product=RDBMS; facility=UL
Message 3054 not found; product=RDBMS; facility=UL
They should not be there. According to a little searching and this post, a file is missing and perhaps a client reinstall is in order? http://www.orafaq.com/forum/t/51572/2
Also you don't need double-quotes around the column names in the control file. I wonder if the double-quotes imply case-sensitive column names like when in SQL and perhaps the column name in dev12 doesn't match? Just a guess.

Related

Looking up data in an Oracle (12.1) table using keys from a text file

I have a table with approximately 8 million rows in it. It has a uniqueness constraint on a column called Customer_Identifier. This is a varchar(10) field, is not the primary key, but is unique.
I wish to retrieve some customer rows from this table using SQL Developer. I have been given a text file with each record containing a search key value in the columns 1-10. This query will need to be reused a few times, with different customer_identifier values. Sometimes I will be given a few customer_identifier values (<1000 of them). Sometimes many (between 1000 and 10000 of them). For the times when I want fewer than 1000 values, it's pretty straightforward to use an IN clause. I can edit the text file to wrap the keys in quotes and insert commas as appropriate. But SQL developer has a hard limit of 1000 values in an IN clause.
I only have read rights to the database, so creating and managing a new physical table is out of the question :-(.
Is there a way that I can treat the text file as a table in Oracle 12.1, and thus use it to join to my customer table on the customer_identifier column?
Brgds
Chris
Yes, you can treat a text file as an external table. But you may need DBA assistance to create a new directory, if you don't have access to a directory defined in the database.
Thanks to Oracle Base
**Create a directory object pointing to the location of the files.**
CREATE OR REPLACE DIRECTORY ext_tab_data AS '/data';
**Create the external table using the CREATE TABLE..ORGANIZATION EXTERNAL syntax. This defines the metadata for the table describing how it should appear and how the data is loaded.**
CREATE TABLE countries_ext (
country_code VARCHAR2(5),
country_name VARCHAR2(50),
country_language VARCHAR2(50)
)
ORGANIZATION EXTERNAL (
TYPE ORACLE_LOADER
DEFAULT DIRECTORY ext_tab_data
ACCESS PARAMETERS (
RECORDS DELIMITED BY NEWLINE
FIELDS TERMINATED BY ','
MISSING FIELD VALUES ARE NULL
(
country_code CHAR(5),
country_name CHAR(50),
country_language CHAR(50)
)
)
LOCATION ('Countries1.txt','Countries2.txt')
)
PARALLEL 5
REJECT LIMIT UNLIMITED;
**Once the external table created, it can be queried like a regular table.**
SQL> SELECT *
2 FROM countries_ext
3 ORDER BY country_name;
COUNT COUNTRY_NAME COUNTRY_LANGUAGE
----- ---------------------------- -----------------------------
ENG England English
FRA France French
GER Germany German
IRE Ireland English
SCO Scotland English
USA Unites States of America English
WAL Wales Welsh
7 rows selected.
SQL>

How can you tell if a Influx Database contains data?

I'm currently trying to count the number of rows in an InfluxDB, but the following fails.
SELECT count(*) FROM "TempData_Quarantine_1519835017000_1519835137000"..:MEASUREMENT";
with the message
InfluxData API responded with status code=BadRequest, response={"error":"error parsing query: found :, expected ; at line 1, char 73"}
To my understanding this query should be checking all measurements and counting them?
(I inherited this code from someone else, so apologies for not understanding it better)
If you need a binary answer to the question "tell if a Influx Database contains data?" then just do
select count(*) from /.*/
In case if the current retention policy in the current database is empty (contains 0 rows) it will return just nothing. Otherwise it will return something like this:
name: api_calls
time count_value
---- -----------
0 5
name: cpu
time count_value
---- -----------
0 1
Also you can specify retention policy explicitly:
SELECT count(*) FROM "TempData_Quarantine_1519835017000_1519835137000"./.*/

View with LEFT JOIN is Updatable?

According to MariaDB documentation, a VIEW can't have a outer join to be updatable:
A view cannot be used for updating if it uses any of the following:
... an outer join ...
As far as i know, "outer join" include "left join" and "right join", right?
But when i test it (in mariaDB v10.1.25), it is updatable! What am I missing here?
DROP TABLE IF EXISTS t1,t2;
CREATE TABLE t1(id INT, a TEXT);
INSERT INTO t1(id,a) VALUES (1,'a'),(2,'b'),(3,'c');
CREATE TABLE t2(id INT, b TEXT);
INSERT INTO t2(id,b) VALUES (1,'+'),(1,'-'),(2,'*');
CREATE OR REPLACE VIEW v1 AS SELECT * FROM t1 LEFT JOIN t2 USING(id);
#UPDATE v1 SET a='x' WHERE id=3; # worked
#UPDATE v1 SET a='y' WHERE b IS NULL; # worked
#UPDATE v1 SET b='y' WHERE b IS NULL; # worked! even so, it does not make sense and does not update anything
SELECT * FROM t1;
SELECT * FROM t2;
Some entries in the ChangeLogs. (This is too long for a Comment; perhaps the Answer is buried in one of the links.)
----- 2015-08-03 5.7.8 Release Candidate -- Bugs Fixed -- -----
An assertion could be raised if a multiple-table UPDATE ( /doc/refman/5.7/en/update.html ) of a view, where the same column was used in the SET and JOIN clauses, was used as a prepared statement. (Bug #76962, Bug #21045724)
http://bugs.mysql.com/bug.php?id=76962
----- 2015-03-09 5.7.6 Milestone 16 -- Optimizer Notes -- -----
To handle a derived table (subquery in the FROM clause) or view reference, ...
mysql> SET optimizer_switch = 'derived_merge=off'; ...
http://bugs.mysql.com/bug.php?id=59203
----- 2015-06-04 MariaDB 10.1.5 & 2015-05-07 MariaDB 10.0.18 & 2015-05-01 MariaDB 5.5.43 -- -- -----
MDEV-7613 : MariaDB 5.5.40 server crash on update table left join with a view
http://localhost/kb/en/mariadb-5540-release-notes/
----- 2009-12-07 5.5.0 Milestone 2 & 2009-10-06 5.1.40 -- Bugs Fixed -- -----
A multiple-table UPDATE ( http://dev.mysql.com/doc/refman/5.5/en/update.html ) involving a natural join and a mergeable view raised an assertion. (Bug #47150)
http://bugs.mysql.com/bug.php?id=47150
Note: 5.5 predates MariaDB 10.1, and the fix should be included; 5.7 comes after, so it won't be in 10.1.

Hive Create big table from small tables

I have 1000 tables in hive. All have same columns. These tables will be incrementally updated.The columns are ID, name, dno, loc,sal …..
I want to create a big table by selecting only Id, name and sal from each table.
Table 1:
ID name dno loc sal ………
1 sam 201 HYD 2000 ………
Table2
ID name dno loc sal ………
2 Ram 203 BAN 3000 ………
Table 3
ID name dno loc sal ………
3 Bam 301 NY 4000 ………
And So on….
Big table:
ID name sal
1 sam 2000
2 Ram 3000
3 Bam 4000
And so on
This is what I want to achieve.
Say If there is anew record tomorrow insert into table 3 say with Id 100, name jack ….
Table 3 with new records
Table 3
ID name dno loc sal ………
3 Bam 301 NY 4000 ………
100 Jack 101 LA 5000 ……….
The new big table should be
ID name sal
1 sam 2000
2 Ram 3000
3 Bam 4000
100 Jack 5000
This is what I want to achieve without deleting the big table every time a new record is being inserted into the original 1000 tables
Slightly modified version of ravinder.
Create your child external tables as below.
create external table table1 (
column1 String,
Column2 String
)
row format delimited
fields terminated by ','
LOCATION '/user/cloudera/data_p/table_name=data1/';
Now your parent table will be created with a partitioned column table_name.
create external table parent_table (
column1 String,
Column2 String
)
partitioned by (table_name String)
row format delimited
fields terminated by ','
LOCATION '/user/cloudera/data_p/';
msck repair table parent_table;
if the ID is unique between all your tables, you can update your big table like
with q1 as (
select * from T1 where ID not in (select ID from BIGTABLE)
union
select * from T2 where ID not in (select ID from BIGTABLE)
.... so on
)
from q1
insert into table BIGTABLE select *;
if you can find the same ID between different tables (for exampleID=1 in T1 and T4), I would suggest use a extra column in the big table to identify the record source (from what tables it comes) or partition the data (depends on the size of the data). Post a comment if something doesn't make sense
Regards!
EDIT: like I said in the comment, if is not possible do an union for all the tables, I would recommend create a partitioned table. The idea is add a partition to this table pointing to the location of the other tables, in that way you should be able to apply the logic that I wrote before, this is possible to do if you are using avro/parquet format using schema evolution.
You can do couple of ways, but I like below if no constraints. make sure all small table location is one directory.
if following does not work,Need more info(criteria of naming for small tables, row formatted of a table ) (other way is to create a script selecting from meta data and get small table name and do a a script with union from small data and insert into big table)
small tables r1 & r2 and big table dept
create table r1
(dept int
,dept_name string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
STORED AS TEXTFILE
LOCATION '/apps/hive/warehouse/dept/r1';
create table r2
(dept int
,dept_name string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
STORED AS TEXTFILE
LOCATION '/apps/hive/warehouse/dept/r2';
[root#sandbox ~]# hadoop fs -ls -R /apps/hive/warehouse/dept
drwxrwxrwx - root hdfs 0 2017-01-25 17:43 /apps/hive/warehouse/dept/r1
-rwxrwxrwx 3 root hdfs 105 2017-01-25 17:43 /apps/hive/warehouse/dept/r1/000000_0
-rwxrwxrwx 3 root hdfs 0 2017-01-25 17:43 /apps/hive/warehouse/dept/r1/000001_0
drwxrwxrwx - root hdfs 0 2017-01-25 17:44 /apps/hive/warehouse/dept/r2
-rwxrwxrwx 3 root hdfs 105 2017-01-25 17:44 /apps/hive/warehouse/dept/r2/000000_0
-rwxrwxrwx 3 root hdfs 0 2017-01-25 17:44 /apps/hive/warehouse/dept/r2/000001_0
create external table dept
(dept int
,dept_name string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
STORED AS TEXTFILE
LOCATION '/apps/hive/warehouse/dept/';

SSIS - Imported string fields from Oracle are converted into wrong/bad charaters

Using SQL Server Integration Services (SSIS) I am migration a table from Oracle (11g) to SQL Server 2008 R2. The table's fields are int, string (data type Unicode strings [DT_WSTR]) and blob type converted to image data type.
SQL Server's collation is "Latin1_General_100_CI_AS".
The workflow is pretty straighforward:
1) An ADO NET element gathers data from the Oracle source.
2) A script component maps the inout column with the output columns with some data conversion
3) A SQL Server destination element stores the records to the target database
During the data migration (in total just 20'000 records) some string fields are stored with asian characters, while other that have same value are moved properly.
As example:
ID CODE USRNAME DOCNAME
---------------------------------------------------------
120 B-0000001 OAS2491 Help.pdf
121 D-0000465 Charlie Doc1.pdf
122 D-0000465 Charlie Doc2.pdf
123 殹榁鴀ځᡑ䇜쿫 Ɫ灿풑뾧껳쮏⽏� Doc3.pdf
124 D-0000465 Alpha Doc2.pdf
As first thing I thought to some special characters in the source table, but I checked the affected records and they are exactle the same as in the other rows properly migrated.
Row with ID 123 has the same values as row 122, that is displayed fine.
On Oracle: CODE is a VARCHAR2 (15 Byte) USRNAME is a VARCHAR2 (36 Byte)
On SQL Server: CODE is a nvarchar(15) USRNAME is a nvarchar(36)
Why some rows are migrated with wrong characters when others not, even if the content is the same?
I could solve the issue by replacing the "SQL Server destination" component with an "OLE DB Destination" element.
With the lattest all rows are properly imported without any wrong characters.

Resources