I'm trying to to import an excel file with a unique identifier column into SQL server but SSIS is giving me a lot of trouble.
I can get the unique identifier values from excel into the OLE DB Destination component but i can't get them into SQL Server table.
I get the error message below on trying to insert the unique identifier.
NOTE: column CreatedBy is of type uniqueidentifier in the destination SQL server table.
There was an error with OLE DB Destination.Inputs[OLE DB Destination Input].Columns[CreatedBy] on OLE DB Destination.Inputs[OLE DB Destination Input]. The column status returned was: "The value could not be converted because of a potential loss of data.".
I recommend you try this:
(DT_GUID)("{"+(DT_WSTR,36)[CreatedBy]+"}")
because your initial string doesn't contain any brackets you need to add them before converting to GUID.
Have fun!
You need to add a Derived Column with the following transformation
REPLACE(REPLACE((DT_WSTR,50)[CreatedBy], "{", ""), "}", "")
and map the new derived column to the output destination.
have fun!
Related
I have a geometry column of a geodataframe populated with polygons and I need to upload these to Snowflake.
I have been exporting the geometry column of the geodataframe to file and have tried both CSV and GeoJSON formats, but so far I either always get an error the staging table always winds up empty.
Here's my code:
design_gdf['geometry'].to_csv('polygons.csv', index=False, header=False, sep='|', compression=None)
import sqlalchemy
from sqlalchemy import create_engine
from snowflake.sqlalchemy import URL
engine = create_engine(
URL(<Snowflake Credentials Here>)
)
with engine.connect() as con:
con.execute("PUT file://<path to polygons.csv> #~ AUTO_COMPRESS=FALSE")
Then on Snowflake I run
create or replace table DB.SCHEMA.DESIGN_POLYGONS_STAGING (geometry GEOGRAPHY);
copy into DB.SCHEMA."DESIGN_POLYGONS_STAGING"
from #~/polygons.csv
FILE_FORMAT = (TYPE = CSV FIELD_DELIMITER = '|' SKIP_HEADER = 1 compression = None encoding = 'iso-8859-1');
Generates the following error:
"Number of columns in file (6) does not match that of the corresponding table (1), use file format option error_on_column_count_mismatch=false to ignore this error File '#~/polygons.csv.gz', line 3, character 1 Row 1 starts at line 2, column "DESIGN_POLYGONS_STAGING"[6] If you would like to continue loading when an error is encountered, use other values such as 'SKIP_FILE' or 'CONTINUE' for the ON_ERROR option. For more information on loading options, please run 'info loading_data' in a SQL client."
Can anyone identify what I'm doing wrong?
Inspired by #Simeon_Pilgrim's comment I went back to Snowflake's documentation. There I found an example of converting a string literal to a GEOGRAPHY.
https://docs.snowflake.com/en/sql-reference/functions/to_geography.html#examples
select to_geography('POINT(-122.35 37.55)');
My polygons looked like strings describing Polygons more than actual GEOGRAPHYs so I decided I needed to be treating them as strings and then calling TO_GEOGRAPHY() on them.
I quickly discovered that they needed to be explicitly enclosed in single quotes and copied into a VARCHAR column in the staging table. This was accomplished by modifying the CSV export code:
import csv
design_gdf['geometry'].to_csv(<path to polygons.csv>,
index=False, header=False, sep='|', compression=None, quoting=csv.QUOTE_ALL, quotechar="'")
The staging table now looks like:
create or replace table DB.SCHEMA."DESIGN_POLYGONS_STAGING" (geometry VARCHAR);
I ran into further problems copying into the staging table related to the presence of a polygons.csv.gz file I must have uploaded in a previous experiment. I deleted this file using:
remove #~/polygons.csv.gz
Finally, converting the staging table to GEOGRAPHY
create or replace table DB.SCHEMA."DESIGN_GEOGRAPHY_STAGING" (geometry GEOGRAPHY);
insert into DB.SCHEMA."DESIGN_GEOGRAPHY"
select to_geography(geometry)
from DB.SCHEMA."DESIGN_POLYGONS_STAGING"
and I wound up with a DESIGN_GEOGRAPHY table with a single column of GEOGRAPHYs in it. Success!!!
I am trying to analyse the code used in the stored procs on our Netezza server.
First step is to get the definitions/code contained in the stored procs - this can easily be done by either of the following:
Using the system views
select
PROCEDURE,
PROCEDURESOURCE
from _v_procedure
where
PROCEDURE = 'MY_PROC'
;
Or using the base table [view points to this table]
select
PRONAME,
PROSRC as PROCEDURESOURCE
from
DEFINITION_SCHEMA."_T_PROC" P
where
PRONAME= 'MY_PROC'
Now, once I run some analysis on the PROCEDURESOURCE column and try to write this information out to a table, I always get the following error:
ERROR: Type 'bytea' not supported by IBM Netezza SQL
Easy way to replicate this error is simply doing the following
create table MY_SCHEMA.TEST_TMP as
with rs as
(
select
PRONAME,
PROSRC
from
DEFINITION_SCHEMA."_T_PROC" P
where
PRONAME = 'MY_PROC'
)
select * from rs
I have determined that there is a column in DEFINITION_SCHEMA."_T_PROC" of type bytea (column name = PROBIN)
I am however not selecting this column, so I am not sure why I am getting this error
Can anyone help with a workaround on how to copy the PROCEDURESOURCE into a new table and bypass the 'bytea' error
Thanks
3 suggestions:
1) Sometimes the ‘limit all’ trick helps: What are the benefits of using LIMIT ALL in a subquery?
2) Alternatively, do a ‘create external table’ and put your data into a file, then another statement to read it back from the file
3) last guess is that you may be able to explicitly cast the column to a more benign data type (Varchar() or similar)
Using Delphi 10.2, SQLite and Teecharts. My SQLite database has two fields, created with:
CREATE TABLE HistoryRuntime ('DayTime' DateTime, Device1 INTEGER DEFAULT (0));
I access the table using a TFDQuery called qryGrpahRuntime with the following SQL:
SELECT DayTime AS TheDate, Sum(Device1) As DeviceTotal
FROM HistoryRuntime
WHERE (DayTime >= "2017-06-01") and (DayTime <= "2017-06-26")
Group by Date(DayTime)
Using the Field Editor in the Delphi IDE, I can add two persistent fields, getting TheDate as a TDateTimeField and DeviceTotal as a TLargeIntField.
I run this query in a program to create a TeeChart, which I created at design time. As long as the query returns some records, all this works. However, if there are no records for the requested dates, I get an EDatabaseError exception with the message:
qryGrpahRuntime: Type mismatch for field 'DeviceTotal', expecting: LargeInt actual: Widestring
I have done plenty of searching for solutions on the web on how to prevent this error on an empty query, but have had not luck with anything I found. From what I can tell, SQLite defaults to the wide string field when no data is returned. I have tried using CAST in the query and it did not seem to make any difference.
If I remove the persistent fields, the query will open without problems on an empty return set. However, in order to use the TeeChart editor in the IDE, it appears I need persistent fields.
Is there a way I can make this work with persistent fields, or am I going to have to throw out the persistent fields and then add the TeeChart Series at runtime?
This behavior is described in Adjusting FireDAC Mapping chapter of the FireDAC's SQLite manual:
For an expression in a SELECT list, SQLite avoids type name
information. When the result set is not empty, FireDAC uses the value
data types from the first record. When empty, FireDAC describes those
columns as dtWideString. To explicitly specify the column data type,
append ::<type name> to the column alias:
SELECT count(*) as "cnt::INT" FROM mytab
So modify your command e.g. this way (I used BIGINT, but you can use any pseudo data type that maps to a 64-bit signed integer data type and is not auto incrementing, which corresponds to your persistent TLargeIntField field):
SELECT
DayTime AS "TheDate",
Sum(Device1) AS "DeviceTotal::BIGINT"
FROM
HistoryRuntime
WHERE
DayTime BETWEEN {d 2017-06-01} AND {d 2017-06-26}
GROUP BY
Date(DayTime)
P.S. I did a small optimization by using BETWEEN operator (which evaluates the column value only once), and used an escape sequence for date constants (which, in real you replace by parameter, I guess; so just for curiosity).
This data type hinting is parsed by the FDSQLiteTypeName2ADDataType procedure that takes and parses column name in format <column name>::<type name> in its AColName parameter.
In Qlikview, I have an excel sheet that I use to map USERNAME to a TEAM value. But everytime I refresh the dashboard, new USERNAME values come up and since they are not in the excel sheet, these USERNAME values show up as their own value in the TEAM column. How would I make it so that any USERNAME that is not in the excel sheet shows up as 'Unidentified' or another value under the TEAM column instead of showing up as their own separate value?
First of all when posting question here if possible always include the source code so everybody will have more clear picture about your problem. Just saying.
On the topic ...
Use the mapping load in this case with supplying the third parameter. For example:
TeamMapping:
Mapping
Load
UserName,
Team
From
[User_to_Team_Mapping.xlsx] (ooxml, embedded labels, table is [Sheet1])
;
Transactions:
Load
Id,
Amount,
ApplyMap( 'TeamMapping', User, 'Unidentified') as Team
From
Transactions.qvd (qvd)
;
The third parameter in ApplyMap is the default string value when mapping value was not found in the mapping table (TeamMapping)
I want to make a copy of every record inserted in jobact to a new table jobactupdates. I am using a stored procedure for this purpose. Both tables are exactly the same and have same no of columns. When I insert data in jobact using insert query then, the stored procedure fails and show the Data Type mismatch error.
My code looks like this:
PROCEDURE insertData
INSERT INTO jobactupdates (jobcode ,jobdescr ,fileno ,port ,mastcode ,mastdescr ,mastdescr1 ,shipper ,goods ,unit1 ,qty ,unit ,vesname ,arremarks ,arrdate ,remarks ,docstat ,docdate ,blno ,bldate ,jastat ,rate ,demand ,received ,balance ,transpor,dldate);
VALUES(jobact.jobcode,jobact.jobdescr,jobact.fileno,jobact.port,jobact.mastcode,jobact.mastdescr,jobact.mastdescr1,jobact.shipper,jobact.goods,jobact.unit1,jobact.qty,jobact.unit,jobact.vesname,jobact.arremarks,jobact.arrdate,jobact.remarks,jobact.docstat,jobact.docdate,jobact.blno,jobact.bldate,jobact.jastat,jobact.rate,jobact.received,jobact.balance,jobact.transpor,jobact.dldate);
ENDPROC
A Data Type Mismatch error occurs when you try to insert an inappropriate data type into a field. For example, if you try to store a string into an integer field. I would double check the table structures and confirm that they are identical.
Another thing to be aware of is if any of the JOBACT field types are set to Integer (AutoInc). They will have to be set to just Integer in the JOBACTUPDATES table. Otherwise you will get a "Field is read-only" error message.
For
Character fields: write them into '' marks,
Numeric fields: just numbers for example 123,
Date fields: {^yyyy-mm-dd}
(There can also optionally be time in Date field.)
Is this your actual query? If so, the fact your Columns and Values clauses contain different field lists has certainly caused this error:
Insert Into ...
bldate,
jastat,
rate,
demand,
received,
balance ..
Values ...
jobact.bldate,
jobact.jastat,
jobact.rate,
jobact.received, <--
jobact.balance, <--
jobact.transpor <--
.