OpenXml and HttpResponse.OutputStream - asp.net-mvc

I have an Asp.Net Mvc application. In this application i have a functionality to download data from Database to Excel file (with OpenXml Sdk). It works now. But when data is large, time from user request to response with download window becomes 10+ minutes. This is because of two long process:
Taking data from MSSQL server.
Generating Excel document in memory on server. (Downloading begins only when Excel document completed)
First problem was solved through using of DataReader. Now generating of excel file begins just after user request becomes to webserver.
For solving second problem we need to generate Excel document on HttpResponse.OutputStream, but this stream is not Seekable and generation fails before begining.
Does anyone knows any workaround that can help to work with this problem?
Sample of my generating function:
public void GenerateSpreadSheetToStream(IDataReader dataReader, Stream outputStream)
{
var columnCaptions = FillColumnCaptionsFromDataReader(dataReader.GetSchemaTable());
//fails on next line with exception "Cannot open package because FileMode or FileAccess value is not valid for the stream."
using (var spreadsheetDocument = SpreadsheetDocument.Create(outputStream, SpreadsheetDocumentType.Workbook))
{
spreadsheetDocument.AddWorkbookPart();
var workSheetPart = spreadsheetDocument.WorkbookPart.AddNewPart<WorksheetPart>();
OpenXmlWriter writer;
using (writer = OpenXmlWriter.Create(workSheetPart))
{
using (writer.Write(new Worksheet()))
{
using (writer.Write(new SheetData()))
{
using (writer.Write(w =>
w.WriteStartElement(new Row(), new[] {new OpenXmlAttribute("r", null, 1.ToString(CultureInfo.InvariantCulture))})))
{
var cells =
columnCaptions.Select(caption => new Cell()
{
CellValue = new CellValue(caption.Item2),
DataType = CellValues.String
});
foreach (var cell in cells)
{
writer.WriteElement(cell);
}
}
var i = 2;
while (dataReader.Read())
{
var oxa = new[] { new OpenXmlAttribute("r", null, i.ToString(CultureInfo.InvariantCulture)) };
using (writer.Write(w => w.WriteStartElement(new Row(), oxa)))
{
var cells =
columnCaptions.Select(
(c, j) =>
new Cell
{
CellValue = new CellValue(dataReader[c.Item1].ToString()),
DataType = CellValues.String,
CellReference = new StringValue(GetSymbolByCellNumber(j))
});
foreach (var cell in cells)
{
writer.WriteElement(cell);
}
}
i++;
}
}
}
}
using (writer = OpenXmlWriter.Create(spreadsheetDocument.WorkbookPart))
{
using (writer.Write(new Workbook()))
{
using (writer.Write(new Sheets()))
{
var sheet = new Sheet
{
Id = spreadsheetDocument.WorkbookPart.GetIdOfPart(workSheetPart),
SheetId = 1,
Name = SheetName
};
writer.WriteElement(sheet);
}
}
}
}
}
private static string GetSymbolByCellNumber(int number)
{
var r = number/26;
var s = (char) ((number%26) + 65);
return new string(s, r);
}
My FileStreamResultWithTransformation (for working with HttpResponse.OutputStream):
public class FileStreamResultWithTransformation : FileResult
{
private readonly Action<Stream> _action;
public FileStreamResultWithTransformation(Action<Stream> action, string contentType, string fileName) : base(contentType)
{
_action = action;
FileDownloadName = fileName;
}
protected override void WriteFile(HttpResponseBase response)
{
response.BufferOutput = false;
_action(response.OutputStream); ->> it fails there
}
}
StackTrace:
[IOException: Cannot open package because FileMode or FileAccess value
is not valid for the stream.]
System.IO.Packaging.Package.ValidateModeAndAccess(Stream s, FileMode
mode, FileAccess access) +784533
System.IO.Packaging.Package.Open(Stream stream, FileMode packageMode,
FileAccess packageAccess, Boolean streaming) +89
System.IO.Packaging.Package.Open(Stream stream, FileMode packageMode,
FileAccess packageAccess) +10
DocumentFormat.OpenXml.Packaging.OpenXmlPackage.CreateCore(Stream
stream) +192
DocumentFormat.OpenXml.Packaging.SpreadsheetDocument.Create(Stream
stream, SpreadsheetDocumentType type, Boolean autoSave) +215
DocumentFormat.OpenXml.Packaging.SpreadsheetDocument.Create(Stream
stream, SpreadsheetDocumentType type) +44
-------.GenerateSpreadSheetToStream(IDataReader dataReader, Stream outputStream) in
d:\Work\Epsilon\development\Web\trunk\Sources\Epsilon.DocumentGenerator\XlsXGenerator.cs:119

It seems to me, that this problem cannot be solved. On finalization of writing, OpenXmlWriter seeks, read and write in different positions of stream and without this actions xlsx file is broken.
I think, there is something wrong in design of OpenXml library.

The problem is a little deeper. Xlsx file is zip archive and OpenXml internally uses the ZipArchive class. Each file in archive has a header, it is placed before the data. ZipArchive writes data to the stream, then returns to the beginning of the file and writes file's header. It uses the Stream.Seek method and HttpResponse.OutputStream cannot work this way.

Related

How to keep the hyperlink in an pdf merge in ITextSharp? [duplicate]

How to merge multiple pdf files (generated on run time) through ItextSharp then printing them.
I found the following link but that method requires the pdf names considering that the pdf files stored and this is not my case .
I have multiple reports i'll convert them to pdf files through this method :
private void AddReportToResponse(LocalReport followsReport)
{
string mimeType;
string encoding;
string extension;
string[] streams = new string[100];
Warning[] warnings = new Warning[100];
byte[] pdfStream = followsReport.Render("PDF", "", out mimeType, out encoding, out extension, out streams, out warnings);
//Response.Clear();
//Response.ContentType = mimeType;
//Response.AddHeader("content-disposition", "attachment; filename=Application." + extension);
//Response.BinaryWrite(pdfStream);
//Response.End();
}
Now i want to merge all those generated files (Bytes) in one pdf file to print it
If you want to merge source documents using iText(Sharp), there are two basic situations:
You really want to merge the documents, acquiring the pages in their original format, transfering as much of their content and their interactive annotations as possible. In this case you should use a solution based on a member of the Pdf*Copy* family of classes.
You actually want to integrate pages from the source documents into a new document but want the new document to govern the general format and don't care for the interactive features (annotations...) in the original documents (or even want to get rid of them). In this case you should use a solution based on the PdfWriter class.
You can find details in chapter 6 (especially section 6.4) of iText in Action — 2nd Edition. The Java sample code can be accessed here and the C#'ified versions here.
A simple sample using PdfCopy is Concatenate.java / Concatenate.cs. The central piece of code is:
byte[] mergedPdf = null;
using (MemoryStream ms = new MemoryStream())
{
using (Document document = new Document())
{
using (PdfCopy copy = new PdfCopy(document, ms))
{
document.Open();
for (int i = 0; i < pdf.Count; ++i)
{
PdfReader reader = new PdfReader(pdf[i]);
// loop over the pages in that document
int n = reader.NumberOfPages;
for (int page = 0; page < n; )
{
copy.AddPage(copy.GetImportedPage(reader, ++page));
}
}
}
}
mergedPdf = ms.ToArray();
}
Here pdf can either be defined as a List<byte[]> immediately containing the source documents (appropriate for your use case of merging intermediate in-memory documents) or as a List<String> containing the names of source document files (appropriate if you merge documents from disk).
An overview at the end of the referenced chapter summarizes the usage of the classes mentioned:
PdfCopy: Copies pages from one or more existing PDF documents. Major downsides: PdfCopy doesn’t detect redundant content, and it fails when concatenating forms.
PdfCopyFields: Puts the fields of the different forms into one form. Can be used to avoid the problems encountered with form fields when concatenating forms using PdfCopy. Memory use can be an issue.
PdfSmartCopy: Copies pages from one or more existing PDF documents. PdfSmartCopy is able to detect redundant content, but it needs more memory and CPU than PdfCopy.
PdfWriter: Generates PDF documents from scratch. Can import pages from other PDF documents. The major downside is that all interactive features of the imported page (annotations, bookmarks, fields, and so forth) are lost in the process.
I used iTextsharp with c# to combine pdf files. This is the code I used.
string[] lstFiles=new string[3];
lstFiles[0]=#"C:/pdf/1.pdf";
lstFiles[1]=#"C:/pdf/2.pdf";
lstFiles[2]=#"C:/pdf/3.pdf";
PdfReader reader = null;
Document sourceDocument = null;
PdfCopy pdfCopyProvider = null;
PdfImportedPage importedPage;
string outputPdfPath=#"C:/pdf/new.pdf";
sourceDocument = new Document();
pdfCopyProvider = new PdfCopy(sourceDocument, new System.IO.FileStream(outputPdfPath, System.IO.FileMode.Create));
//Open the output file
sourceDocument.Open();
try
{
//Loop through the files list
for (int f = 0; f < lstFiles.Length-1; f++)
{
int pages =get_pageCcount(lstFiles[f]);
reader = new PdfReader(lstFiles[f]);
//Add pages of current file
for (int i = 1; i <= pages; i++)
{
importedPage = pdfCopyProvider.GetImportedPage(reader, i);
pdfCopyProvider.AddPage(importedPage);
}
reader.Close();
}
//At the end save the output file
sourceDocument.Close();
}
catch (Exception ex)
{
throw ex;
}
private int get_pageCcount(string file)
{
using (StreamReader sr = new StreamReader(File.OpenRead(file)))
{
Regex regex = new Regex(#"/Type\s*/Page[^s]");
MatchCollection matches = regex.Matches(sr.ReadToEnd());
return matches.Count;
}
}
Here is some code I pulled out of an old project I had. It was a web application but I was using iTextSharp to merge pdf files then print them.
public static class PdfMerger
{
/// <summary>
/// Merge pdf files.
/// </summary>
/// <param name="sourceFiles">PDF files being merged.</param>
/// <returns></returns>
public static byte[] MergeFiles(List<Stream> sourceFiles)
{
Document document = new Document();
MemoryStream output = new MemoryStream();
try
{
// Initialize pdf writer
PdfWriter writer = PdfWriter.GetInstance(document, output);
writer.PageEvent = new PdfPageEvents();
// Open document to write
document.Open();
PdfContentByte content = writer.DirectContent;
// Iterate through all pdf documents
for (int fileCounter = 0; fileCounter < sourceFiles.Count; fileCounter++)
{
// Create pdf reader
PdfReader reader = new PdfReader(sourceFiles[fileCounter]);
int numberOfPages = reader.NumberOfPages;
// Iterate through all pages
for (int currentPageIndex = 1; currentPageIndex <=
numberOfPages; currentPageIndex++)
{
// Determine page size for the current page
document.SetPageSize(
reader.GetPageSizeWithRotation(currentPageIndex));
// Create page
document.NewPage();
PdfImportedPage importedPage =
writer.GetImportedPage(reader, currentPageIndex);
// Determine page orientation
int pageOrientation = reader.GetPageRotation(currentPageIndex);
if ((pageOrientation == 90) || (pageOrientation == 270))
{
content.AddTemplate(importedPage, 0, -1f, 1f, 0, 0,
reader.GetPageSizeWithRotation(currentPageIndex).Height);
}
else
{
content.AddTemplate(importedPage, 1f, 0, 0, 1f, 0, 0);
}
}
}
}
catch (Exception exception)
{
throw new Exception("There has an unexpected exception" +
" occured during the pdf merging process.", exception);
}
finally
{
document.Close();
}
return output.GetBuffer();
}
}
/// <summary>
/// Implements custom page events.
/// </summary>
internal class PdfPageEvents : IPdfPageEvent
{
#region members
private BaseFont _baseFont = null;
private PdfContentByte _content;
#endregion
#region IPdfPageEvent Members
public void OnOpenDocument(PdfWriter writer, Document document)
{
_baseFont = BaseFont.CreateFont(BaseFont.HELVETICA,
BaseFont.CP1252, BaseFont.NOT_EMBEDDED);
_content = writer.DirectContent;
}
public void OnStartPage(PdfWriter writer, Document document)
{ }
public void OnEndPage(PdfWriter writer, Document document)
{ }
public void OnCloseDocument(PdfWriter writer, Document document)
{ }
public void OnParagraph(PdfWriter writer,
Document document, float paragraphPosition)
{ }
public void OnParagraphEnd(PdfWriter writer,
Document document, float paragraphPosition)
{ }
public void OnChapter(PdfWriter writer, Document document,
float paragraphPosition, Paragraph title)
{ }
public void OnChapterEnd(PdfWriter writer,
Document document, float paragraphPosition)
{ }
public void OnSection(PdfWriter writer, Document document,
float paragraphPosition, int depth, Paragraph title)
{ }
public void OnSectionEnd(PdfWriter writer,
Document document, float paragraphPosition)
{ }
public void OnGenericTag(PdfWriter writer, Document document,
Rectangle rect, string text)
{ }
#endregion
private float GetCenterTextPosition(string text, PdfWriter writer)
{
return writer.PageSize.Width / 2 - _baseFont.GetWidthPoint(text, 8) / 2;
}
}
I didn't write this, but made some modifications. I can't remember where I found it. After I merged the PDFs I would call this method to insert javascript to open the print dialog when the PDF is opened. If you change bSilent to true then it should print silently to their default printer.
public Stream addPrintJStoPDF(Stream thePDF)
{
MemoryStream outPutStream = null;
PRStream finalStream = null;
PdfDictionary page = null;
string content = null;
//Open the stream with iTextSharp
var reader = new PdfReader(thePDF);
outPutStream = new MemoryStream(finalStream.GetBytes());
var stamper = new PdfStamper(reader, (MemoryStream)outPutStream);
var jsText = "var res = app.setTimeOut('this.print({bUI: true, bSilent: false, bShrinkToFit: false});', 200);";
//Add the javascript to the PDF
stamper.JavaScript = jsText;
stamper.FormFlattening = true;
stamper.Writer.CloseStream = false;
stamper.Close();
//Set the stream to the beginning
outPutStream.Position = 0;
return outPutStream;
}
Not sure how well the above code is written since I pulled it from somewhere else and I haven't worked in depth at all with iTextSharp but I do know that it did work at merging PDFs that I was generating at runtime.
Tested with iTextSharp-LGPL 4.1.6:
public static byte[] ConcatenatePdfs(IEnumerable<byte[]> documents)
{
using (var ms = new MemoryStream())
{
var outputDocument = new Document();
var writer = new PdfCopy(outputDocument, ms);
outputDocument.Open();
foreach (var doc in documents)
{
var reader = new PdfReader(doc);
for (var i = 1; i <= reader.NumberOfPages; i++)
{
writer.AddPage(writer.GetImportedPage(reader, i));
}
writer.FreeReader(reader);
reader.Close();
}
writer.Close();
outputDocument.Close();
var allPagesContent = ms.GetBuffer();
ms.Flush();
return allPagesContent;
}
}
To avoid the memory issues mentioned, I used file stream instead of memory stream(mentioned in ITextSharp Out of memory exception merging multiple pdf) to merge pdf files:
var parentDirectory = Directory.GetParent(SelectedDocuments[0].FilePath);
var savePath = parentDirectory + "\\MergedDocument.pdf";
using (var fs = new FileStream(savePath, FileMode.Create))
{
using (var document = new Document())
{
using (var pdfCopy = new PdfCopy(document, fs))
{
document.Open();
for (var i = 0; i < SelectedDocuments.Count; i++)
{
using (var pdfReader = new PdfReader(SelectedDocuments[i].FilePath))
{
for (var page = 0; page < pdfReader.NumberOfPages;)
{
pdfCopy.AddPage(pdfCopy.GetImportedPage(pdfReader, ++page));
}
}
}
}
}
}
****/*For Multiple PDF Print..!!*/****
<button type="button" id="btnPrintMultiplePdf" runat="server" class="btn btn-primary btn-border btn-sm"
onserverclick="btnPrintMultiplePdf_click">
<i class="fa fa-file-pdf-o"></i>Print Multiple pdf</button>
protected void btnPrintMultiplePdf_click(object sender, EventArgs e)
{
if (ValidateForMultiplePDF() == true)
{
#region Declare Temp Variables..!!
CheckBox chkList = new CheckBox();
HiddenField HidNo = new HiddenField();
string Multi_fofile, Multi_listfile;
Multi_fofile = Multi_listfile = "";
Multi_fofile = Server.MapPath("PDFRNew");
#endregion
for (int i = 0; i < grdRnew.Rows.Count; i++)
{
#region Find Grd Controls..!!
CheckBox Chk_One = (CheckBox)grdRnew.Rows[i].FindControl("chkOne");
Label lbl_Year = (Label)grdRnew.Rows[i].FindControl("lblYear");
Label lbl_No = (Label)grdRnew.Rows[i].FindControl("lblCode");
#endregion
if (Chk_One.Checked == true)
{
HidNo .Value = llbl_No .Text.Trim()+ lbl_Year .Text;
if (File.Exists(Multi_fofile + "\\" + HidNo.Value.ToString() + ".pdf"))
{
#region Get Multiple Files Name And Paths..!!
if (Multi_listfile != "")
{
Multi_listfile = Multi_listfile + ",";
}
Multi_listfile = Multi_listfile + Multi_fofile + "\\" + HidNo.Value.ToString() + ".pdf";
#endregion
}
}
}
#region For Generate Multiple Pdf..!!
if (Multi_listfile != "")
{
String[] Multifiles = Multi_listfile.Split(',');
string DestinationFile = Server.MapPath("PDFRNew") + "\\Multiple.Pdf";
MergeFiles(DestinationFile, Multifiles);
Response.ContentType = "pdf";
Response.AddHeader("Content-Disposition", "attachment;filename=\"" + DestinationFile + "\"");
Response.TransmitFile(DestinationFile);
Response.End();
}
else
{
}
#endregion
}
}
private void MergeFiles(string DestinationFile, string[] SourceFiles)
{
try
{
int f = 0;
/**we create a reader for a certain Document**/
PdfReader reader = new PdfReader(SourceFiles[f]);
/**we retrieve the total number of pages**/
int n = reader.NumberOfPages;
/**Console.WriteLine("There are " + n + " pages in the original file.")**/
/**Step 1: creation of a document-object**/
Document document = new Document(reader.GetPageSizeWithRotation(1));
/**Step 2: we create a writer that listens to the Document**/
PdfWriter writer = PdfWriter.GetInstance(document, new FileStream(DestinationFile, FileMode.Create));
/**Step 3: we open the Document**/
document.Open();
PdfContentByte cb = writer.DirectContent;
PdfImportedPage page;
int rotation;
/**Step 4: We Add Content**/
while (f < SourceFiles.Length)
{
int i = 0;
while (i < n)
{
i++;
document.SetPageSize(reader.GetPageSizeWithRotation(i));
document.NewPage();
page = writer.GetImportedPage(reader, i);
rotation = reader.GetPageRotation(i);
if (rotation == 90 || rotation == 270)
{
cb.AddTemplate(page, 0, -1f, 1f, 0, 0, reader.GetPageSizeWithRotation(i).Height);
}
else
{
cb.AddTemplate(page, 1f, 0, 0, 1f, 0, 0);
}
/**Console.WriteLine("Processed page " + i)**/
}
f++;
if (f < SourceFiles.Length)
{
reader = new PdfReader(SourceFiles[f]);
/**we retrieve the total number of pages**/
n = reader.NumberOfPages;
/**Console.WriteLine("There are"+n+"pages in the original file.")**/
}
}
/**Step 5: we Close the Document**/
document.Close();
}
catch (Exception e)
{
string strOb = e.Message;
}
}
private bool ValidateForMultiplePDF()
{
bool chkList = false;
foreach (GridViewRow gvr in grdRnew.Rows)
{
CheckBox Chk_One = (CheckBox)gvr.FindControl("ChkSelectOne");
if (Chk_One.Checked == true)
{
chkList = true;
}
}
if (chkList == false)
{
divStatusMsg.Style.Add("display", "");
divStatusMsg.Attributes.Add("class", "alert alert-danger alert-dismissable");
divStatusMsg.InnerText = "ERROR !!...Please Check At Least On CheckBox.";
grdRnew.Focus();
set_timeout();
return false;
}
return true;
}

Write zip files to MemoryStream

I have a controller action that creates a zip file and sends back to user for download. The problem is that the zip file gets created but it is empty. Somehow it's not writing the image files to the MemoryStream. I wonder what I am missing. If I write the zip file to the disk everything will work as expected, but I'd rather not save files to the disk if I can avoid it. This is what I have tried using dotnetzip:
public ActionResult DownloadGraphs()
{
var state = Session["State"];
using (ZipFile zip = new ZipFile())
{
if (state == "IA")
{
zip.AddFile(Server.MapPath("~/Content/DataVizByState/FallGraphs/Watermarked/Fall_IA.jpg"), "");
zip.AddFile(Server.MapPath("~/Content/DataVizByState/SpringGraphs/Watermarked/Spring_IA.jpg"), "");
}
MemoryStream output = new MemoryStream();
zip.Save(output);
output.Seek(0, SeekOrigin.Begin);
var fileName = state + "Graphs.zip";
return File(output, "application/zip", fileName);
}
}
This forces download in the view based on click of a button:
$('#graphDwnldBtn').click(function (evt) {
window.location = '#Url.Action("DownloadGraphs", "DataSharing")';
})
Do I need to use StreamWriter or Reader or something? This is the first time I have ever attempted something like this and it's been cobbled together by reading various stackoverflow posts...
Dumb mistakes: Session["State"] is an object, so the state variable was coming out as object instead of a string like I need it to be for my conditional statement to evaluate correctly. I cast state to a string to fix it. Fixed code:
public ActionResult DownloadGraphs()
{
var state = Session["State"].ToString();
using (ZipFile zip = new ZipFile())
{
if (state == "IA")
{
zip.AddFile(Server.MapPath("~/Content/DataVizByState/FallGraphs/Watermarked/Fall_IA.jpg"), "");
zip.AddFile(Server.MapPath("~/Content/DataVizByState/SpringGraphs/Watermarked/Spring_IA.jpg"), "");
}
MemoryStream output = new MemoryStream();
zip.Save(output);
output.Seek(0, SeekOrigin.Begin);
var fileName = state + "Graphs.zip";
return File(output, "application/zip", fileName);
}
}

jqgrid + EF + MVC: How to export in excel? Which method you suggest?

I am using jqgrid (standard) with EF 4 + MVC3. I'd like to implement excel export. Which method you would suggest me?
To generate excel, I'd like to use this library by Dr Stephen Walther, which has three types of output and allows to define headers too. Please tell me if you find it valid for my purpose.
I ask this question because I am still approaching to implement excel export and I found several techniques. Some suggest making a csv export, others indicate that it should return a JSON output and it is not clear to me whether this capability is present in the free version of jqgrid. In any case, I would like to pass the data to Walther's object.
About the jqgrid code, I found this interesting answer by Oleg, but I do not understand if could be applied to my needs.
Unfortunately, by now I only found parts of solutions for excel export with EF MVC, but no solution or complete examples...
About the MVC logic, I am going to implement and develop this code as kindly suggested by #Tommy.
Please sorry if the question could be silly, I am just a (enthusiast) beginner.
Thanks for your precious help!
Best Regards
As I wrote before (see here and here for example) the best way to export grid data to XML is the usage of Open XML SDK 2.0.
The post of Dr Stephen Walther shows how to create HTML file which can be read by Excel. It's not Excel file and have to be still converted to Excel format. The usage of CSV has even more problems. Depend on the content in the source table the automatic conversion to Excel data types can be absolutely wrong. In one project which I developed for a customer the grid contained information about software products: product name, version, and so on. The software version looks sometime as the date (1.3.1963 for example) and such cells will be wrong converted (in German one use '.' as the separator in the date). As the result one had really hard problems. The usage of CSV with texts having commas inside will be also frequently wrong imported. Even when one quotes the cells having commas (,) and escaped the texts having quotas the import still be wrong especially in the first column. I don't want to explain here the whole history of all attempts and errors, but after all I decide to give up with the usage of CSV and HTML and started to use Open XML SDK 2.0 which allows to create real Excel files with extension XLSX. The way seems me perfect because one don't need any Office
components installed on the server, no additional licenses.
The only restriction is that one should be able to use DocumentFormat.OpenXml.dll, so your server program should run on any Windows operation system. As it's well known, XLSX file is ZIP file which contains some XML files inside. If you still don't know that I recommend you to rename the XLSX file to ZIP file and extract it. The Open XML SDK 2.0 is the library which works with XLSX file like with XML files. So no additional Office components are required.
One can find a lot of information how to use Open XML SDK 2.0 (see here, here and here). Many helpful code examples one cam find directly on the MSDN (see here). Nevertheless the practical usage of Open XML SDK 2.0 is not so easy at least at the first time. So I created a demo from the parts of the code which I used myself.
You can download the demo project from here. The demo is an extension of the demos from the answer and this one.
To export data I use the DataForExcel helper class. It has constructor in the form
DataForExcel(string[] headers, DataType[] colunmTypes, List<string[]> data,
string sheetName)
or in a little simplified form
DataForExcel(string[] headers, List<string[]> data, string sheetName)
and the only public method
CreateXlsxAndFillData(Stream stream)
The usage of the class to create Excel file can be like the following
var excelData = new DataForExcel (
// column Header
new[]{"Col1", "Col2", "Col3"},
new[]{DataForExcel.DataType.String, DataForExcel.DataType.Integer,
DataForExcel.DataType.String},
new List<string[]> {
new[] {"a", "1", "c1"},
new[] {"a", "2", "c2"}
},
"Test Grid");
Stream stream = new FileStream ("Test.xlsx", FileMode.Create);
excelData.CreateXlsxAndFillData (stream);
stream.Close();
The usage in the demo from ASP.NET MVC is the following
static readonly string[] HeadersQuestions = {
"Id", "Votes", "Title"
};
static readonly DataForExcel.DataType[] ColunmTypesQuestions = {
DataForExcel.DataType.Integer,
DataForExcel.DataType.Integer,
DataForExcel.DataType.String
};
public ActionResult ExportAllQuestionsToExcel () {
var context = new HaackOverflowEntities ();
var questions = context.Questions;
questions.MergeOption = MergeOption.NoTracking; // we don't want to update the data
// to be able to use ToString() below which is NOT exist in the LINQ to Entity
// we should include in query only the properies which we will use below
var query = questions.ToList ();
if (query.Count == 0)
return new EmptyResult ();
var data = new List<string[]> (query.Count);
data.AddRange (query.Select (item => new[] {
item.Id.ToString(CultureInfo.InvariantCulture),
item.Votes.ToString(CultureInfo.InvariantCulture),
item.Title
}));
return new ExcelResult (HeadersQuestions, ColunmTypesQuestions, data,
"Questions.xlsx", "Questions");
}
where ExcelResult are defined as
public class ExcelResult : ActionResult {
private readonly DataForExcel _data;
private readonly string _fileName;
public ExcelResult (string[] headers, List<string[]> data, string fileName, string sheetName) {
_data = new DataForExcel (headers, data, sheetName);
_fileName = fileName;
}
public ExcelResult (string[] headers, DataForExcel.DataType[] colunmTypes, List<string[]> data, string fileName, string sheetName) {
_data = new DataForExcel (headers, colunmTypes, data, sheetName);
_fileName = fileName;
}
public override void ExecuteResult (ControllerContext context) {
var response = context.HttpContext.Response;
response.ClearContent();
response.ClearHeaders();
response.Cache.SetMaxAge (new TimeSpan (0));
using (var stream = new MemoryStream()) {
_data.CreateXlsxAndFillData (stream);
//Return it to the client - strFile has been updated, so return it.
response.AddHeader ("content-disposition", "attachment; filename=" + _fileName);
// see http://filext.com/faq/office_mime_types.php
response.ContentType = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";
response.ContentEncoding = Encoding.UTF8;
stream.WriteTo (response.OutputStream);
}
response.Flush();
response.Close();
}
}
To make the code full I have to include the code of the class DataForExcel:
public class DataForExcel {
public enum DataType {
String,
Integer
}
private readonly string[] _headers;
private readonly DataType[] _colunmTypes;
private readonly List<string[]> _data;
private readonly string _sheetName = "Grid1";
private readonly SortedSet<string> _os = new SortedSet<string> ();
private string[] _sharedStrings;
private static string ConvertIntToColumnHeader(int index) {
var sb = new StringBuilder ();
while (index > 0) {
if (index <= 'Z' - 'A') // index=0 -> 'A', 25 -> 'Z'
break;
sb.Append (ConvertIntToColumnHeader (index / ('Z' - 'A' + 1) - 1));
index = index % ('Z' - 'A' + 1);
}
sb.Append ((char)('A' + index));
return sb.ToString ();
}
private static Row CreateRow(UInt32 index, IList<string> data) {
var r = new Row { RowIndex = index };
for (var i = 0; i < data.Count; i++)
r.Append (new OpenXmlElement[] { CreateTextCell (ConvertIntToColumnHeader (i), index, data[i]) });
return r;
}
private Row CreateRowWithSharedStrings(UInt32 index, IList<string> data) {
var r = new Row { RowIndex = index };
for (var i = 0; i < data.Count; i++)
r.Append (new OpenXmlElement[] { CreateSharedTextCell (ConvertIntToColumnHeader (i), index, data[i]) });
return r;
}
private Row CreateRowWithSharedStrings(UInt32 index, IList<string> data, IList<DataType> colunmTypes) {
var r = new Row { RowIndex = index };
for (var i = 0; i < data.Count; i++)
if (colunmTypes != null && i < colunmTypes.Count && colunmTypes[i] == DataType.Integer)
r.Append (new OpenXmlElement[] { CreateNumberCell (ConvertIntToColumnHeader (i), index, data[i]) });
else
r.Append (new OpenXmlElement[] { CreateSharedTextCell (ConvertIntToColumnHeader (i), index, data[i]) });
return r;
}
private static Cell CreateTextCell(string header, UInt32 index, string text) {
// create Cell with InlineString as a child, which has Text as a child
return new Cell (new InlineString (new Text { Text = text })) {
// Cell properties
DataType = CellValues.InlineString,
CellReference = header + index
};
}
private Cell CreateSharedTextCell(string header, UInt32 index, string text) {
for (var i=0; i<_sharedStrings.Length; i++) {
if (String.Compare (_sharedStrings[i], text, StringComparison.Ordinal) == 0) {
return new Cell (new CellValue { Text = i.ToString (CultureInfo.InvariantCulture) }) {
// Cell properties
DataType = CellValues.SharedString,
CellReference = header + index
};
}
}
// create Cell with InlineString as a child, which has Text as a child
throw new InstanceNotFoundException();
}
private static Cell CreateNumberCell(string header, UInt32 index, string numberAsString) {
// create Cell with CellValue as a child, which has Text as a child
return new Cell (new CellValue { Text = numberAsString }) {
// Cell properties
CellReference = header + index
};
}
private void FillSharedStringTable(IEnumerable<string> data) {
foreach (var item in data)
_os.Add (item);
}
private void FillSharedStringTable(IList<string> data, IList<DataType> colunmTypes) {
for (var i = 0; i < data.Count; i++)
if (colunmTypes == null || i >= colunmTypes.Count || colunmTypes[i] == DataType.String)
_os.Add (data[i]);
}
public DataForExcel(string[] headers, List<string[]> data, string sheetName) {
_headers = headers;
_data = data;
_sheetName = sheetName;
}
public DataForExcel(string[] headers, DataType[] colunmTypes, List<string[]> data, string sheetName) {
_headers = headers;
_colunmTypes = colunmTypes;
_data = data;
_sheetName = sheetName;
}
private void FillSpreadsheetDocument(SpreadsheetDocument spreadsheetDocument) {
// create and fill SheetData
var sheetData = new SheetData ();
// first row is the header
sheetData.AppendChild (CreateRow (1, _headers));
//const UInt32 iAutoFilter = 2;
// skip next row (number 2) for the AutoFilter
//var i = iAutoFilter + 1;
UInt32 i = 2;
// first of all collect all different strings in OrderedSet<string> _os
foreach (var dataRow in _data)
if (_colunmTypes != null)
FillSharedStringTable (dataRow, _colunmTypes);
else
FillSharedStringTable (dataRow);
_sharedStrings = _os.ToArray ();
foreach (var dataRow in _data)
sheetData.AppendChild (_colunmTypes != null
? CreateRowWithSharedStrings (i++, dataRow, _colunmTypes)
: CreateRowWithSharedStrings (i++, dataRow));
var sst = new SharedStringTable ();
foreach (var text in _os)
sst.AppendChild (new SharedStringItem (new Text (text)));
// add empty workbook and worksheet to the SpreadsheetDocument
var workbookPart = spreadsheetDocument.AddWorkbookPart ();
var worksheetPart = workbookPart.AddNewPart<WorksheetPart> ();
var shareStringPart = workbookPart.AddNewPart<SharedStringTablePart> ();
shareStringPart.SharedStringTable = sst;
shareStringPart.SharedStringTable.Save ();
// add sheet data to Worksheet
worksheetPart.Worksheet = new Worksheet (sheetData);
worksheetPart.Worksheet.Save ();
// fill workbook with the Worksheet
spreadsheetDocument.WorkbookPart.Workbook = new Workbook (
new FileVersion { ApplicationName = "Microsoft Office Excel" },
new Sheets (
new Sheet {
Name = _sheetName,
SheetId = (UInt32Value)1U,
// generate the id for sheet
Id = workbookPart.GetIdOfPart (worksheetPart)
}
)
);
spreadsheetDocument.WorkbookPart.Workbook.Save ();
spreadsheetDocument.Close ();
}
public void CreateXlsxAndFillData(Stream stream) {
// Create workbook document
using (var spreadsheetDocument = SpreadsheetDocument.Create (stream, SpreadsheetDocumentType.Workbook)) {
FillSpreadsheetDocument (spreadsheetDocument);
}
}
}
The above code create new XLSX file directly. You can extend the code to support more data types as String and Integer which I used in the code.
In more professional version of your application you can create some XLSX templates for exporting different tables. In the code you can place the data in the cells instead, so modify the spreadsheet instead of creating. In the way you can create perfect formatted XLSX files. The examples from the MSDN (see here) will help you to implement the way when it will be required.
UPDATED: The answer contains updated code which allows generate Excel documented with more cell formatting.
I looked at Stephen's post and it's old as hell, which btw doesn't make it wrong.
If you don't need custom formatting, headers and styles, then I think use CSV as it's very simple.
More importantly, don't think that excel export from MVC site that internally uses EF for data access is harder than, say, Ruby on Rails site that uses ActiveRecord. For me it's independent concerns, export shouldn't new anything about underlying technologies (at least not directly), just the structure of your data, that's all.
Search for codeplex libraries that allows to do Excel reading/writing and export, there are plenty of them these days, many really good solutions that's regularly maintained and tested by thousand of developers all over the globe. If I were you I won't use Stephen solution because it looks like he occasionally typed it in a notepad and then pasted to the post - no unit tests, no extensibility points + it's in VB so it even harder to understand, but may be that's just me.
Hope this help and good luck

sqlbulkcopy mem. management

I'm using SQLBULKCOPY to copy some data-tables into a database table, however, because the size of the files I'm copying run sometimes in excess of 600mb, I keep running out of memory.
I'm hoping to get some advice about managing the table size before I commit it to the database so I can free up some memory to continue writing.
Here are some examples of my code (some columns and rows eliminated for simplicity)
SqlBulkCopy sqlbulkCopy = new SqlBulkCopy(ServerConfiguration); //Define the Server Configuration
System.IO.StreamReader rdr = new System.IO.StreamReader(fileName);
Console.WriteLine("Counting number of lines...");
Console.WriteLine("{0}, Contains: {1} Lines", fileName, countLines(fileName));
DataTable dt = new DataTable();
sqlbulkCopy.DestinationTableName = "[dbo].[buy.com]"; //You need to define the target table name where the data will be copied
dt.Columns.Add("PROGRAMNAME");
dt.Columns.Add("PROGRAMURL");
dt.Columns.Add("CATALOGNAME");
string inputLine = "";
DataRow row; //Declare a row, which will be added to the above data table
while ((inputLine = rdr.ReadLine()) != null) //Read while the line is not null
{
i = 0;
string[] arr;
Console.Write("\rWriting Line: {0}", k);
arr = inputLine.Split('\t'); //splitting the line which was read by the stream reader object (tab delimited)
row = dt.NewRow();
row["PROGRAMNAME"] = arr[i++];
row["PROGRAMURL"] = arr[i++];
row["CATALOGNAME"] = arr[i++];
row["LASTUPDATED"] = arr[i++];
row["NAME"] = arr[i++];
dt.Rows.Add(row);
k++;
}
// Set the timeout, 600 secons (10 minutes) given table size--damn that's a lota hooch
sqlbulkCopy.BulkCopyTimeout = 600;
try
{
sqlbulkCopy.WriteToServer(dt);
}
catch (Exception e)
{
Console.WriteLine(e);
}
sqlbulkCopy.Close();//Release the resources
dt.Dispose();
Console.WriteLine("\nDB Table Written: \"{0}\" \n\n", sqlbulkCopy.DestinationTableName.ToString());
}
I continued to have problems getting SQLBulkCopy to work, and I realized I needed to do more work on each record before it was entered into the database, so I developed a simple LinQ to Sql method to do record by record updates, so I could edit other information and create more record information as it was being run,
Problem: This method's been running pretty slow (even on Core i3 machine), any ideas on how to speed it up (threading?) -- on a single processor core, with 1gb of memory it crashes or takes sometimes 6-8 hours to write the same amount of data as one SQLBulkCopy that takes a few moments. It does manage memory better though.
while ((inputLine = rdr.ReadLine()) != null) //Read while the line is not null
{
Console.Write("\rWriting Line: {0}", k);
string[] arr;
arr = inputLine.Split('\t');
/* items */
if (fileName.Contains(",,"))
{
Item = Table(arr);
table.tables.InsertOnSubmit(Item);
/* Check to see if the item is in the db */
bool exists = table.tables.Where(u => u.ProductID == Item.ProductID).Any();
/* Commit */
if (!exists)
{
try
{
table.SubmitChanges();
}
catch (Exception e)
{
Console.WriteLine(e);
// Make some adjustments.
// ...
// Try again.
table.SubmitChanges();
}
}
}
With helper method:
public static class extensionMethods
{
/// <summary>
/// Method that provides the T-SQL EXISTS call for any IQueryable (thus extending Linq).
/// </summary>
/// <remarks>Returns whether or not the predicate conditions exists at least one time.</remarks>
public static bool Exists<TSource>(this IQueryable<TSource> source, Expression<Func<TSource, bool>> predicate)
{
return source.Where(predicate).Any();
}
}
Try specifying the BatchSize property to 1000 which will batch up the insert in a 1000 record batch rather than the whole lot. You can tweak this value to find what is optimal. I have used sqlbulkcopy for similar size data and it works well.
Faced with the same issue, found that problem of OutOfMemory Exception was in DataTable.Rows maximum quantity limitations.
Solved with recreating table, with maximum 500000 rows limit.
Hope, my solution will be helpfull:
var myTable = new System.Data.DataTable();
myTable.Columns.Add("Guid", typeof(Guid));
myTable.Columns.Add("Name", typeof(string));
int counter = 0;
foreach (var row in rows)
{
++counter;
if (counter < 500000)
{
myTable.Rows.Add(
new object[]
{
row.Value.Guid,
row.Value.Name
});
}
else
{
using (var dbConnection = new SqlConnection("Source=localhost;..."))
{
dbConnection.Open();
using (var s = new SqlBulkCopy(dbConnection))
{
s.DestinationTableName = "MyTable";
foreach (var column in myTable.Columns)
s.ColumnMappings.Add(column.ToString(), column.ToString());
try
{
s.WriteToServer(myTable);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
s.Close();
}
}
}
myTable = new System.Data.DataTable();
myTable.Columns.Add("Guid", typeof(Guid));
myTable.Columns.Add("Name", typeof(string));
myTable.Rows.Add(
new object[]
{
row.Value.Guid,
row.Value.Name
});
counter = 0;
}
}

asp.net mvc serving txt gets truncated

I'm trying to serve a txt file made from the database using an action. The action is the following:
public ActionResult ATxt()
{
var articulos = _articulosService.ObteTotsArticles();
return File(CatalegATxt.ATxt(articulos), "text/plain");
}
and the CatalegATxt class is:
using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using WebDibaelsaMVC.DTOs.Busqueda;
namespace WebDibaelsaMVC.TxtLib
{
public static class CatalegATxt
{
public static Stream ATxt(IEnumerable<ArticuloBusquedaDTO> articles)
{
var stream = new MemoryStream();
var streamWriter = new StreamWriter(stream, Encoding.UTF8);
foreach (ArticuloBusquedaDTO article in articles)
{
streamWriter.WriteLine(article.ToStringFix());
}
stream.Seek(0, SeekOrigin.Begin);
return stream;
}
public static string ToStringFix(this ArticuloBusquedaDTO article)
{
string result = "";
result += article.CodigoArticulo.PadRight(10, ' ').Substring(0, 10);
result += article.EAN.Trim().PadLeft(13, '0').Substring(0, 13);
result += article.NombreArticulo.PadRight(100, ' ').Substring(0, 100);
result += article.Marca.PadRight(100, ' ').Substring(0, 100);
result += article.Familia.PadRight(50, ' ').Substring(0, 50);
result += article.PrecioCesion.ToStringFix();
result += article.PVP.ToStringFix();
return result;
}
private static string ToStringFix(this double numero)
{
var num = (int)Math.Round(numero * 100, 0);
string result = num.ToString().PadLeft(10, '0');
return result;
}
}
}
it just writes the file lines based on the stuff I got from the database. But when I look at the file it looks truncated. The file is about 8Mb. I also tried converting to byte[] before returning from ATxt with the same result.
Any idea?
Thanks,
Carles
Update: I also tried to serve XML from the same content and it also gets truncated. It doesn't get truncated on the data (I thought it might have been an EOF character in it) but it truncates in the middle of a label...
I was having the exact same problem. The text file would always be returned as truncated.
It crossed my mind that it might be a "flushing" problem, and indeed it was. The writer's buffer hasn't been flushed at the end of the operation - since there's no using block, or the Close() call - which would flush automatically.
You need to call:
streamWriter.Flush();
before MVC takes over the stream.
Here's how your method should look like:
public static Stream ATxt(IEnumerable<ArticuloBusquedaDTO> articles)
{
var stream = new MemoryStream();
var streamWriter = new StreamWriter(stream, Encoding.UTF8);
foreach (ArticuloBusquedaDTO article in articles)
{
streamWriter.WriteLine(article.ToStringFix());
}
// Flush the stream writer buffer
streamWriter.Flush();
stream.Seek(0, SeekOrigin.Begin);
return stream;
}
Why are you using an ActionResult?
ASP.NET MVC 1 has a FileStreamResult for just what you are doing. It expects a Stream object, and returns it.
public FileStreamResult Test()
{
return new FileStreamResult(myMemoryStream, "text/plain");
}
Should work fine for what you want to do. No need to do any conversions.
In your case, just change your method to this:
public FileStreamResult ATxt()
{
var articulos = _articulosService.ObteTotsArticles();
return new FileStreamResult(CatalegATxt.ATxt(articulos), "text/plain");
}
You probably want to close the MemoryStream. It could be getting truncated because it expects more data still. Or to make things even simpler, try something like this:
public static byte[] ATxt(IEnumerable<ArticuloBusquedaDTO> articles)
{
using(var stream = new MemoryStream())
{
var streamWriter = new StreamWriter(stream, Encoding.UTF8);
foreach (ArticuloBusquedaDTO article in articles)
{
streamWriter.WriteLine(article.ToStringFix());
}
return stream.ToArray();
}
}

Resources