AS400 / SQL Server 2008 R2 Data Transfer Performance Improvement - stored-procedures

Question

We have recently converted our JD Edwards EnterpriseOne system from and AS400/DB2 platform to Windows & SQL Server. In the old system we had a RPG/CL program that would transfer data from AS400 library to the accounting system for further processing. The end users needed to initiate this process so it was executed via a menu command.
To replicate this behavior after the conversion I created a stored procedure in SQL Server 2008 R2 that inserts records into the SQL Server database from the as400 via a linked server and then updates the records on the as400 to indicate that the records have been processed. To allow the end users the ability to execute this process, I created a SSRS (2005) report that executes the stored procedure.
When the SSRS report is executed interactively, we intermittently get an error 'For security resasons DTD is prohibited in this XML document' which from my research is caused by SSRS running out of memory.
Does anyone know of another/better way to transfer the data?
The transfer/update of the stored procedures is essentially
INSERT INTO [SQL DEST TABLE]
SELECT *
FROM [AS400 Linked Server/Table]
UPDATE OPENQUERY (AS400_LINK, 'MY SELECT QUERY')
SET FLAG = PROCESSED;

Answer 1

You should get better database performance can allow a server to perform work with it's own data, where possible, rather than having to transmit it back and forth.
I will make a few guesses about the circumstances, and if you correct me, I will gladly adjust the answer to fit your situation. This sounds like you are extracting data from an accounting transaction table in DB2, and that when done, you want to update the flag in those same records. That might indicate that the records could stay in that table essentially forever, or perhaps that some other process clears them out. There is no WHERE on your SELECT, so I will assume they do get removed. I will assume that we don't know if more records might get added to the transaction table at any time, including any period between extracting the info and updating the flag.
I wonder if you could update the flag immediately upon extraction, before they have actually been processed SQL Server? Would this be allowed logistically, and within your business requirements?
Suppose you...
extract the DB2 unprocessed transaction data into a workfile,
transfer the workfile to SQL Server,
perform whatever processing you want to do in SQL Server
tell DB2 to update the transaction table based on the workfile
So in DB2, #1 might look like
INSERT INTO workfile
SELECT *
FROM transactions
WHERE flag = unprocessed
During step 3, your SQL Server job could update the flag in the workfile to an error status, for any records that SQL Server cannot process properly.
Step 4 on DB2 could be
UPDATE transactions
SET flag = processed
WHERE transid IN (SELECT transid
FROM workfile
WHERE flag <> error
)
Hopefully, processing errors on SQL Server would be fairly rare. If that process updates transactions in the workfile, only on an error, this should be faster than transmitting updates for each success. The UPDATE statement above, should be able to to run faster on DB2, since it is driven by the workfile on the same server, rather than data be transmitted back up to DB2.