I have an MVC 4 application that currently reads data from a CSV file (this is based on client requirements, even if I wanted to have a database for it). All is working well when I debug and run it from visual studio. However, when I deploy it on IIS 7.5, it is unable to read large CSV files (currently, the largest i have is around 6000kb). I tried different techniques on reading the files, but it just produces the same result. But small files are being read perfectly.
Here is my code in parsing the file:
using (CsvReader csv =
new CsvReader(new StreamReader(_filePath), false, ';'))
{
while (csv.ReadNextRecord())
{
int fieldCount = csv.FieldCount;
string currentRow = "";
for (int i = 0; i < fieldCount; i++)
{
currentRow += csv[i] + ";";
}
this.AddKYCFolder(this.CreateKYCFolder(currentRow.Split(';')));
}
}
Any ideas on this?
Many thanks!
Thank you for taking time to look into my question. Apparently, there were some methods that uses parallelism (Parallel.For) that is, I suppose, not compatible with my IIS setup. The log4net error logging greatly helped me to find the source of error.
Related
I am updating archaic code that creates memos. The code was written to use bookmarks inside of manually created template.doc files that aspose can write to. The problem comes from this chunk of code.
foreach (Addressee infoAddressee in ConfigManager.GetConfig().Addressees)
{
if (infoAddressee.Abbreviation == Memo.AddresseeAbbr.ToUpper() &&
infoAddressee.NeedsThisLine)
{
WriteMeString = "FOO BARR ________";
break;
}
}
if (WriteMeString != "")
{
builder.MoveToBookmark("BOOKMARK");
builder.Write(WriteMeString);
}
}
This works for me, but the two people who have tested this chunk of code have the "FOO BARR _______" line appear as "FOO BARR "
the seven underlines are replaced with spaces(the spacing exists on the word doc, but Stack overflow concatenates consecutive spaces). I am not sure what could cause this.
To test we need to copy the file from the remote dev environment into our local environment, I believe this to be the source of the issue, but i do not know for sure.
What I have already tried:
The testers and me are supplying the exact same input for the document.
The testers and I had a slightly different way to save the document and copy paste it over to the local environment, but doing it my way did not change anything.
I am unsure of what could do this for some users but not for others, any suggestions for things i could check out, be it literature with information on the subject or proposed solutions, would be greatly appreciated
I checked the scenario on my side and cannot reproduce the problem. Underscores are properly displayed in the output document. Here are few things to try.
Try setting bookmark text instead of moving to it and writing text.
doc.Range.Bookmarks["BOOKMARK"].Text = WriteMeString;
Try checking whether string is written correctly into the document.
builder.MoveToBookmark("BOOKMARK");
builder.Write("FOO BARR ________");
Assert.AreEqual("FOO BARR ________", builder.Document.Range.Bookmarks["BOOKMARK"].Text);
using (MemoryStream ms = new MemoryStream())
{
builder.Document.Save(ms, SaveFormat.Doc);
ms.Position = 0;
Document tempDoc = new Document(ms);
Assert.AreEqual("FOO BARR ________", tempDoc.Range.Bookmarks["BOOKMARK"].Text);
}
Compare the documents produced on your side and on the testers side yourself (I suppose, you have already done this, but just in case). Probably the documents are correct, but there is difference in viewer used on your and testers side.
Disclosure: I work at Aspose.Words team.
I made an ASP.NET MVC web application that is existed in a Sub-Directory. The problem is every time I try to upload file I get this error "Could not find a part of the path".
The code is working perfectly on my local machine and other web apps, so I think the problem is related to the web app being exist in the sub-directory, but I don't know how to solve it.
Thanks in advance.
This is my function
public byte newImage(HttpPostedFileBase newFile, string uploadPath)
{
if (newFile != null && newFile.ContentLength > 0)
{
if (newFile.ContentLength > 3000000) //means file size maximum is 3 MB
return 1; //means the file size is more than 3 MB
var fileName = Path.GetFileName(newFile.FileName);
var path = Path.Combine(System.Web.HttpContext.Current.Server.MapPath(uploadPath), fileName);
newFile.SaveAs(path);
return 0; //means file uploaded successfuly
}
return 2; //means no file was chosen
}//Upload New Image
Did you "map" the upload folder properly? You can read more here.
DirectoryInfo yourUploadDir = new DirectoryInfo(HostingEnvironment.MapPath("~/YourUploadFolder"));
I figured out what's wrong. It seems I forgot to add ~ -_-
So if anyone faced this error, first check if you forgot to add ~. I suppose it worked well on my local machine and other web apps because they were on the root, but once it's placed on a sub-directory, We need to use relative path using this tilde ~.
I am working on WTX design Studio with copybooks,i have a copybook but i dont have any corresponding sample input regarding that .cpy file .
Are there any means to generate a sample text file from copy book rather than writing the text document manually?
As bill said there are lots of ways depending on the tools available:
Ask for sample file
There are packages that can generate data. As I do not work at your site I do not know if any have been installed. They tend to be expensive though
Write a Cobol program (I presume the source is a mainframe) so it would have to be done on the mainframe.
I presume the source is a Mainframe, get on the Mainframe and use FileAid (or FileMaster or whatever they have) to setup the file. FileAid and its ilk let you edit files with a Cobol copybook.
Use the RecordEditor to create the file. You can import Cobol Copybooks into the RecordEditor and then use them to edit Cobol-Data files.
Use a RecordEditor macro to generate a file;
Write a Java / jython / JRuby program with Cobol interface package (have a look on sourceforge)
There are a lot of other possibilities, as I do not know what software or skills you have I can not really advise.
RecordEditor Macro to generate some numeric data:
/******************************************************************
* Purpose: RecordEditor Example Macroto generate numeric data for a file
*
* It is best to run this script from a Single Record Screen rather than
* a Table screen
*
*******************************************************************/
var rec = layout.getRecord(0)
var lines = RecordEditorData.view.createLines(20)
for (lineNo = 0; lineNo < 20; lineNo++) {
print(lineNo);
for (i=0; i < rec.getFieldCount(); i++) {
try {
lines[lineNo].getFieldValue(0, i).set(lineNo * 100 + i)
} catch(err) {
lines[lineNo].getFieldValue(0, i).set(i % 10)
}
}
}
RecordEditorData.view.addLines(-1,1, lines)
Output from the Macro:
Is there a way to set or override a project deployment folder in Mono for Android? For example, my application right now deploys to /data/data/SolutionEngine/files/.__override__
The nature of the application is that it loads plug-ins using Reflection, and by default it looks in the /Adapters sub-folder from the app root. This is how it works on the desktop and the Compact Framework, so for simplicity we'd like to continue to do the same on Android.
If I have a single solution that has the app and some plug-ins in it, I'd like those files to get deployed in the proper structure when I start debugging.
You could write out the plugins as android assets (see screenshot below). Please Note: You might need to change the extension to .mp3. See here. I didn't have this issue though.
Once you do that, you should be able to get the assets by using the Asset Manager. You can copy them to a different folder or do whatever with them. Here is a sample of reading them into memory and them writing out the name.
const String pluginPath = "Plugins";
var pluginAssets = Assets.List(pluginPath);
foreach (var pluginAsset in pluginAssets)
{
var file = Assets.Open(pluginPath + Java.IO.File.Separator + pluginAsset);
using (var memStream = new MemoryStream())
{
file.CopyTo(memStream);
//do something fun.
var assembly = System.Reflection.Assembly.Load(memStream.ToArray());
Console.WriteLine(String.Format("Loaded: {0}", assembly.FullName));
}
}
I'm new to search engines and web crawlers. Now I want to store all the original pages in a particular web site as html files, but with Apache Nutch I can only get the binary database files. How do I get the original html files with Nutch?
Does Nutch support it? If not, what other tools can I use to achieve my goal.(The tools that support distributed crawling are better.)
Well, nutch will write the crawled data in binary form so if if you want that to be saved in html format, you will have to modify the code. (this will be painful if you are new to nutch).
If you want quick and easy solution for getting html pages:
If the list of pages/urls that you intend to have is quite low, then better get it done with a script which invokes wget for each url.
OR use HTTrack tool.
EDIT:
Writing a your own nutch plugin will be great. Your problem will get solved plus you can contribute to nutch by submitting your work !!! If you are new to nutch (in terms of code & design), then you will have to invest lot of time building a new plugin ... else its easy to do.
Few pointers for helping your initiative:
Here is a page which talks about writing own nutch plugin.
Start with Fetcher.java. See lines 647-648. That is the place where you can get the fetched content on per url basis (for those pages which got fetched successfully).
pstatus = output(fit.url, fit.datum, content, status, CrawlDatum.STATUS_FETCH_SUCCESS);
updateStatus(content.getContent().length);
You should add code right after this to invoke your plugin. Pass content object to it. By now, you would have guessed that content.getContent() is the content for url you want. Inside the plugin code, write it to some file. Filename should be based on the url name else it will be difficult to work with that. Url can be obtained by fit.url.
You must do modifications in run Nutch in Eclipse.
When you are able to run, open Fetcher.java and add the lines between "content saver" command lines.
case ProtocolStatus.SUCCESS: // got a page
pstatus = output(fit.url, fit.datum, content, status, CrawlDatum.STATUS_FETCH_SUCCESS, fit.outlinkDepth);
updateStatus(content.getContent().length);'
//------------------------------------------- content saver ---------------------------------------------\\
String filename = "savedsites//" + content.getUrl().replace('/', '-');
File file = new File(filename);
file.getParentFile().mkdirs();
boolean exist = file.createNewFile();
if (!exist) {
System.out.println("File exists.");
} else {
FileWriter fstream = new FileWriter(file);
BufferedWriter out = new BufferedWriter(fstream);
out.write(content.toString().substring(content.toString().indexOf("<!DOCTYPE html")));
out.close();
System.out.println("File created successfully.");
}
//------------------------------------------- content saver ---------------------------------------------\\
To update this answer -
It is possible to post process the data from your crawldb segment folder, and read in the html (including other data nutch has stored) directly.
Configuration conf = NutchConfiguration.create();
FileSystem fs = FileSystem.get(conf);
Path file = new Path(segment, Content.DIR_NAME + "/part-00000/data");
SequenceFile.Reader reader = new SequenceFile.Reader(fs, file, conf);
try
{
Text key = new Text();
Content content = new Content();
while (reader.next(key, content))
{
System.out.println(new String(content.GetContent()));
}
}
catch (Exception e)
{
}
The answers here are obsolete. Now, it is simply possible to get the plain HTML-files with nutch dump. Please see this answer.
In apache Nutch 2.3.1
You can save the raw HTML by edit the Nutch code firstly run the nutch in eclipse by following https://wiki.apache.org/nutch/RunNutchInEclipse
After you finish ruunning nutch in eclipse edit file FetcherReducer.java , add this code to the output method, run ant eclipse again to rebuild the class
Finally the raw html will added to reportUrl column in your database
if (content != null) {
ByteBuffer raw = fit.page.getContent();
if (raw != null) {
ByteArrayInputStream arrayInputStream = new ByteArrayInputStream(raw.array(), raw.arrayOffset() + raw.position(), raw.remaining());
Scanner scanner = new Scanner(arrayInputStream);
scanner.useDelimiter("\\Z");//To read all scanner content in one String
String data = "";
if (scanner.hasNext()) {
data = scanner.next();
}
fit.page.setReprUrl(StringUtil.cleanField(data));
scanner.close();
}