Generic Data Reader causing memory leak - memory

I created a generic method in reading sql statement, but I am having a memory leak whenever I do a select query and using while read.
Sample Query:
public CMItemPackagingType GetItemPackagingType(int itemID)
{
try
{
List<CommandParameter> param = new List<CommandParameter>();
StringBuilder sb = new StringBuilder();
using (BaseConnection db = new BaseConnection())
{
sb.Append("SELECT RATIO, PACKAGING_TYPE_CODE FROM ITEM_PACKAGING_TYPE WHERE ROUND_UP = 0.01 AND ITEM_ID = #itemID");
param.Add(new CommandParameter("#itemID", itemID));
using (var rs = db.ExecSQL(sb.ToString(), param.ToArray()))
{
CMItemPackagingType cmItemInfo = new CMItemPackagingType();
while (rs.Read())
{
CMItemPackagingType list = new CMItemPackagingType();
if (!rs.IsDBNull(0))
list.Ratio = Convert.ToInt32(rs.GetValue(0));
if (!rs.IsDBNull(1))
list.PackagingTypeCode = rs.GetValue(1).ToString();
cmItemInfo.ItemPackagingTypeList.Add(list);
}
return cmItemInfo;
}
}
}
catch (Exception ex)
{
GlobalFramework.HandleException(ex);
}
return null;
}
Generic Reader:
public DbDataReader ExecSQL(string sqlStmt, CommandParameter[] param)
{
List<MySqlParameter> p = ParameterMySql(param);
_mySqlConn = new MySqlConnection(szConnect);
if (_mySqlConn.State == ConnectionState.Open)
{
_mySqlConn.Close();
}
_mySqlConn.Open();
_mySqlComm = new MySqlCommand(sqlStmt, _mySqlConn);
_mySqlComm.Parameters.AddRange(p.ToArray());
MySqlDataReader reader = _mySqlComm.ExecuteReader();
return reader;
}

I'm assuming the BaseConnection is a wrapper around a SqlConnection and _mySqlConn is an instance of BaseConnection. I suspect the issue is that you are opening and closing the connection in ExecSQL and at the same time have a using statement around BaseConnection creating this leak. I would refactor your code with proper placement of using statements to ensure correct disposal of the objects and freeing of resources.
Example
var query = "YOUR QUERY";
using (var connection = new SqlConnection("YOUR CONNECTION STRING"))
{
using (var command = new SqlCommand(query, connection))
{
await connection.OpenAsync();
using (var reader = await command.ExecuteReaderAsync())
{
if (reader != null)
{
while (await reader.ReadAsync())
{
// your logic
}
}
} // reader closed and disposed up here
} // command disposed here
} //connection closed and disposed here
}
Also notice how I'm using the async versions of the ADO.NET methods. Async commands are critical in achieving scale, throughput, and latency.
I recommend you use Dapper over trying to develop a generic data reader and writing all the boilerplate ADO.NET code yourself.

Related

How to stop current job and start new job use quartz.net in asp.net mvc?

I've made a crawler on a website it has to read a website and fetch some values from it website.I've made use quartz.net and Asp.net MVC. but what is my problem? in fact,My problem is that for example,he/she the first time start for scraping a "Stackoverflow.com" about 5 hours and then he/she is decided stop "stackoverflow.com" and start a scrap new website.So,How can i do it?
[HttpPost]
public ActionResult Index(string keyword, string url)
{
IScheduler scheduler = StdSchedulerFactory.GetDefaultScheduler();
scheduler.Start();
IJobDetail job = JobBuilder.Create<ScrapJob>()
.WithIdentity("MyScrapJob")
.UsingJobData("url", url)
.UsingJobData("keyword", keyword)
.Build();
ITrigger trigger = TriggerBuilder.Create().WithDailyTimeIntervalSchedule(
s => s.WithIntervalInSeconds(20).OnEveryDay().StartingDailyAt(TimeOfDay.HourAndMinuteOfDay(0, 0))
).Build();
scheduler.ScheduleJob(job, trigger);
return View(db.Scraps.ToList());
}
public List<ScrapJob> Scraping(string url, string keyword)
{
int count = 0;
List<ScrapJob> scraps = new List<ScrapJob>();
ScrapJob scrap = null;
HtmlDocument doc = new HtmlDocument();
try
{
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
using (var response = (HttpWebResponse)request.GetResponse())
{
using (var stream = response.GetResponseStream())
{
doc.Load(stream, Encoding.GetEncoding("UTF-8"));
foreach (HtmlNode node in doc.DocumentNode.SelectNodes("//text()"))
{
if (node.InnerText.ToString().Contains(keyword))
{
count++;
scrap = new ScrapJob { Keyword = keyword, DateTime = System.DateTime.Now.ToString(), Count = count, Url = url };
}
}
}
}
}
catch (WebException ex)
{
Console.WriteLine(ex.Message);
}
// scraps.Add(scrap);
var isExist = db.Scraps.Where(s => s.Keyword == keyword && s.Count == scrap.Count).Max(s => (int?)s.Id) ?? 0;
if (isExist == 0)
{
db.Scraps.Add(scrap);
db.SaveChanges();
}
return scraps;
}
public void Execute(IJobExecutionContext context)
{
//ScrapJob scraps = null;
using (var scrap = new ScrapJob())
{
JobKey key = context.JobDetail.Key;
JobDataMap dataMap = context.JobDetail.JobDataMap;
string url = dataMap.GetString("url");
string keyword = dataMap.GetString("keyword");
scrap.Scraping(url, keyword);
}
}
I'm not sure why you picked QUARTZ, but here is something that I think will help you.
This is a code sample that interrupt and delete job by unique identifier
public void DeleteJob(JobKey jobKey)
{
var scheduler = StdSchedulerFactory.GetDefaultScheduler();
var executingJobs = scheduler.GetCurrentlyExecutingJobs();
if (executingJobs.Any(x => x.JobDetail.Key.Equals(jobKey)))
{
scheduler.Interrupt(jobKey);
}
scheduler.DeleteJob(jobKey);
}
But I believe you need to define what behavior you expect, because it can be a bit more complex for example:
If you like to just pause the job and resume it after finish with the other website /persist some state and progress/ or just log the progress
If you want them to run in parallel and process multiple sites simultaneously. (You just need to give different names instead of the hardcoded .WithIdentity("MyScrapJob") )
Also with scheduler.GetCurrentlyExecutingJobs() you can get the currently executing jobs, show them to the user and let him decide what to do.
Also looking at your action method I'm not sure whether this is the behavior you expect of that trigger. Also what bothers me is db.Scraps.ToList() you will materialize the whole table you can consider adding pagination as well in your case is not necessary because you will only show count but its mandatory if you have a lot of records in the grid.
About the scraping method
Instead of
var isExist = db.Scraps.Where(s => s.Keyword == keyword && s.Count == scrap.Count).Max(s => (int?)s.Id) ?? 0;
you can use .Any
var exists = db.Scraps.Any(s => s.Keyword == keyword && s.Count == scrap.Count);
this will return boolean and you can check if(!exists)
You can check https://github.com/AngleSharp/AngleSharp it's high performance web parsing library. Super easy to use as well.
I see possibility of duplicated records by keyword if you check them by keyword and count - not sure whether you want this or just want to update the existing record with it's counter
Good luck! I hope this answer helps you :)

How to get data from data base with asp mvc 4 async controller and EntityFramework 6 async read from database

I have next controller
public async Task<ActionResult> ImageAsync(int id)
{
var img = await _repository.GetImageAsync(id);
if (img != null)
{
return File(img, "image/jpg"); //View(img);
}
byte[] res = new byte[0];
return File(res, "image/jpg");
}
and method in repository
public async Task<byte[]> GetImage(int imageId)
{
try
{
var dbCtx = new smartbags_storeEntities();
var res = await dbCtx.GoodImages.SingleAsync(d => d.ImageId == imageId);
return res != null ? res.ImageData : null;
}
catch (Exception ex)
{
throw ex;
}
}
public async Task<byte[]> GetImageAsync(int imageId)
{
byte[] img = await Task.Run(() =>
{
var res = GetImage(imageId).Result;
if (res != null)
{
var wi = new System.Web.Helpers.WebImage(res);
wi.AddTextWatermark("info");
return wi.GetBytes();
}
return null;
});
return img;
}
but execution of image reading is freezing on line
var res = await dbCtx.GoodImages.SingleAsync(d => d.ImageId == imageId);
What I am doing in wrong way when try to read data from data base in async style ?
The call to the property Result of a Task is a blocking call and the continuation of the await won't be able to be posted to run.
Once you already have a Task returning method, why didn't you just use await?
public async Task<byte[]> GetImageAsync(int imageId)
{
var res = await GetImage(imageId);
if (res != null)
{
var wi = new System.Web.Helpers.WebImage(res);
wi.AddTextWatermark("info");
return wi.GetBytes();
}
return null;
}
The funny thing about that line is that it calls SingleAsync, which is a TAP extension method for observables.
I have never used a data repository that exposed its collections as observables, though I suppose it is possible. My first guess is that [the task returned by] SingleAsync isn't completing because the GoodImages observable isn't completing. Note that SingleAsync must continue scanning after it sees a match to ensure that it is the only match; FirstAsync is more forgiving and will complete as soon as it sees the first match.
On a side note, I do recommend using await instead of Result and not using Task.Run on the server. So Paulo's answer is good in that regard, though in this case Result is not causing a deadlock.

Memory issue when using CGImage.ScreenImage in a loop using Mono Touch

I'm trying to create an app to read QR codes using Monotouch and C# port of Zxing but I'm hitting memory issues. While the app processes captured screen frames the app receives memory warnings and is then shut down. I have removed the call to Zxing to track down where the memory issue stems from and can reproduce the issue with just capturing the screen image in a loop.
Here is the code:
using System;
using System.Drawing;
using System.Collections.Generic;
using System.Threading;
using MonoTouch.UIKit;
using MonoTouch.Foundation;
using MonoTouch.CoreGraphics;
using com.google.zxing;
using com.google.zxing.common;
using System.Collections;
using MonoTouch.AudioToolbox;
using iOS_Client.Utilities;
namespace iOS_Client.Controllers
{
public class CameraOverLayView : UIView
{
private Thread _thread;
private CameraViewController _parentViewController;
private Hashtable hints;
private static com.google.zxing.MultiFormatReader _multiFormatReader = null;
private static RectangleF picFrame = new RectangleF(0, 146, 320, 157);
private static UIImage _theScreenImage = null;
public CameraOverLayView(CameraViewController parentController) : base()
{
Initialize();
_parentViewController = parentController;
}
private void Initialize()
{
}
private bool Worker()
{
Result resultb = null;
if(DeviceHardware.Version == DeviceHardware.HardwareVersion.iPhone4
|| DeviceHardware.Version == DeviceHardware.HardwareVersion.iPhone4S)
{
picFrame = new RectangleF(0, 146*2, 320*2, 157*2);
}
if(hints==null)
{
var list = new ArrayList();
list.Add (com.google.zxing.BarcodeFormat.QR_CODE);
hints = new Hashtable();
hints.Add(com.google.zxing.DecodeHintType.POSSIBLE_FORMATS, list);
hints.Add (com.google.zxing.DecodeHintType.TRY_HARDER, true);
}
if(_multiFormatReader == null)
{
_multiFormatReader = new com.google.zxing.MultiFormatReader();
}
using (var screenImage = CGImage.ScreenImage.WithImageInRect(picFrame))
{
using (_theScreenImage = UIImage.FromImage(screenImage))
{
Bitmap srcbitmap = new System.Drawing.Bitmap(_theScreenImage);
LuminanceSource source = null;
BinaryBitmap bitmap = null;
try {
source = new RGBLuminanceSource(srcbitmap, screenImage.Width, screenImage.Height);
bitmap = new BinaryBitmap(new HybridBinarizer(source));
try {
_multiFormatReader.Hints = hints;
resultb = null;
//_multiFormatReader.decodeWithState(bitmap);
if(resultb != null && resultb.Text!=null)
{
InvokeOnMainThread( () => _parentViewController.BarCodeScanned(resultb));
}
}
catch (ReaderException re)
{
//continue;
}
} catch (Exception ex) {
Console.WriteLine(ex.Message);
}
finally {
if(bitmap!=null)
bitmap = null;
if(source!=null)
source = null;
if(srcbitmap!=null)
{
srcbitmap.Dispose();
srcbitmap = null;
}
}
}
}
return resultb != null;
}
public void StartWorker()
{
if(_thread==null)
{
_thread = new Thread(()=> {
bool result = false;
while (result == false)
{
result = Worker();
Thread.Sleep (67);
}
});
}
_thread.Start();
}
public void StopWorker()
{
if(_thread!=null)
{
_thread.Abort();
_thread = null;
}
//Just in case
_multiFormatReader = null;
hints = null;
}
protected override void Dispose(bool disposing)
{
StopWorker();
base.Dispose(disposing);
}
}
}
Interestingly I took a look at http://blog.reinforce-lab.com/2010/02/monotouchvideocapturinghowto.html to try and see how others were capturing and processing video and this code suffers from the same as mine, quitting after about 40 seconds with memory warnings.
Hopefully the QR codes will be scanned in less than 40 seconds but I'm not sure if the memory ever gets released so the problem may crop up after many codes have been scanned. Either way it should be possible to capture a video feed continuously without memory issues right?
This is somewhat counter-intuitive, but the ScreenImage property will create a new CGImage instance every time you call it, so you must call Dispose on that object as well:
using (var img = CGImage.ScreenImage) {
using (var screenImage = img.WithImageInRect(picFrame))
{
}
}
I will just add the actual solution that worked for me which combined information from previous answers. The code inside the loop looks like:
using (var pool = new NSAutoreleasePool ())
{
using (var img = CGImage.ScreenImage)
{
using (var screenImage = img.WithImageInRect(picFrame))
{
using (_theScreenImage = UIImage.FromImage(screenImage))
{
}
}
}
}
GC.Collect();
The original System.Drawing.Bitmap from zxing.MonoTouch suffered from a lack of Dispose which made it never release the unmanaged memory it allocated.
The more recent one (from your link) does free the unmanaged memory when Dispose is called (it's better). However it creates a bitmap context (in it's constructor) and does not dispose it manually (e.g. with a using). So it relies on the garbage collector (GC) to do it later...
In many cases this is not a big issue since the GC will, eventually, free this context instance and will reclaim the associated memory. However if you're doing this in a loop it's possible you'll run out of (unmanaged) memory before the GC kicks in. That will get you memory warnings and iOS can decide to kill your application (or it could crash by itself).
but I'm not sure if the memory ever gets released
Yes, it should be - but maybe not as fast as you need the memory back. Implementing (and using) IDisposable correctly will solve this.
Either way it should be possible to capture a video feed continuously without memory issues right?
Yes. Make sure you're releasing your memory as soon as possible, e.g. with using (var ...) { }, and ensure the 3rd party code you use does the same.

What is using connection in the code

What is the purpose of using (connection) in the code - please explain me
static void HasRows(SqlConnection connection)
{
using (connection)/// what is this line
{
SqlCommand command = new SqlCommand(
"SELECT CategoryID, CategoryName FROM Categories;",
connection);
connection.Open();
SqlDataReader reader = command.ExecuteReader();
if (reader.HasRows)
{
while (reader.Read())
{
Console.WriteLine("{0}\t{1}", reader.GetInt32(0),
reader.GetString(1));
}
}
else
{
Console.WriteLine("No rows found.");
}
reader.Close();
}
}
using (connection){
connection.Open();
}
makes sure that connection is closed when the application is done using it.
similar to a Try Catch.
try{
connection.Open();
}
catch{
}
finally{
connection.Dispose();
}
Disposing of the connection is another way of saying closing a connection. An open connection can leak memory and if you have too many it can slow down or freeze up whatever you are connecting to.
the using function closes the connection even after you return something from the class you are in. same as the try catch. it always closes the connection no matter what happens inside the brackets. even if there is an exception that breaks out of the class/application the connection still gets closed
Using
Quote from site:
The using statement allows the programmer to specify when objects that use resources should release them. The object provided to the using statement must implement the IDisposable interface. This interface provides the Dispose method, which should release the object's resources.
I would probably re-write the method to something like the following:
static void HasRows(string connectionString)
{
using (var connection = new SqlConnection(connectionString))
using(var command = new SqlCommand("SELECT CategoryID, CategoryName FROM Categories;",
connection))
{
connection.Open();
using (var reader = command.ExecuteReader())
{
if (reader.HasRows)
{
while (reader.Read())
{
Console.WriteLine("{0}\t{1}", reader.GetInt32(0),
reader.GetString(1));
}
}
else
{
Console.WriteLine("No rows found.");
}
}
}
}
In your original implementation the caller could be at the receiving end of an ObjectDisposedException exception, because the SqlConnection is passed in as a parameter.

sqlbulkcopy mem. management

I'm using SQLBULKCOPY to copy some data-tables into a database table, however, because the size of the files I'm copying run sometimes in excess of 600mb, I keep running out of memory.
I'm hoping to get some advice about managing the table size before I commit it to the database so I can free up some memory to continue writing.
Here are some examples of my code (some columns and rows eliminated for simplicity)
SqlBulkCopy sqlbulkCopy = new SqlBulkCopy(ServerConfiguration); //Define the Server Configuration
System.IO.StreamReader rdr = new System.IO.StreamReader(fileName);
Console.WriteLine("Counting number of lines...");
Console.WriteLine("{0}, Contains: {1} Lines", fileName, countLines(fileName));
DataTable dt = new DataTable();
sqlbulkCopy.DestinationTableName = "[dbo].[buy.com]"; //You need to define the target table name where the data will be copied
dt.Columns.Add("PROGRAMNAME");
dt.Columns.Add("PROGRAMURL");
dt.Columns.Add("CATALOGNAME");
string inputLine = "";
DataRow row; //Declare a row, which will be added to the above data table
while ((inputLine = rdr.ReadLine()) != null) //Read while the line is not null
{
i = 0;
string[] arr;
Console.Write("\rWriting Line: {0}", k);
arr = inputLine.Split('\t'); //splitting the line which was read by the stream reader object (tab delimited)
row = dt.NewRow();
row["PROGRAMNAME"] = arr[i++];
row["PROGRAMURL"] = arr[i++];
row["CATALOGNAME"] = arr[i++];
row["LASTUPDATED"] = arr[i++];
row["NAME"] = arr[i++];
dt.Rows.Add(row);
k++;
}
// Set the timeout, 600 secons (10 minutes) given table size--damn that's a lota hooch
sqlbulkCopy.BulkCopyTimeout = 600;
try
{
sqlbulkCopy.WriteToServer(dt);
}
catch (Exception e)
{
Console.WriteLine(e);
}
sqlbulkCopy.Close();//Release the resources
dt.Dispose();
Console.WriteLine("\nDB Table Written: \"{0}\" \n\n", sqlbulkCopy.DestinationTableName.ToString());
}
I continued to have problems getting SQLBulkCopy to work, and I realized I needed to do more work on each record before it was entered into the database, so I developed a simple LinQ to Sql method to do record by record updates, so I could edit other information and create more record information as it was being run,
Problem: This method's been running pretty slow (even on Core i3 machine), any ideas on how to speed it up (threading?) -- on a single processor core, with 1gb of memory it crashes or takes sometimes 6-8 hours to write the same amount of data as one SQLBulkCopy that takes a few moments. It does manage memory better though.
while ((inputLine = rdr.ReadLine()) != null) //Read while the line is not null
{
Console.Write("\rWriting Line: {0}", k);
string[] arr;
arr = inputLine.Split('\t');
/* items */
if (fileName.Contains(",,"))
{
Item = Table(arr);
table.tables.InsertOnSubmit(Item);
/* Check to see if the item is in the db */
bool exists = table.tables.Where(u => u.ProductID == Item.ProductID).Any();
/* Commit */
if (!exists)
{
try
{
table.SubmitChanges();
}
catch (Exception e)
{
Console.WriteLine(e);
// Make some adjustments.
// ...
// Try again.
table.SubmitChanges();
}
}
}
With helper method:
public static class extensionMethods
{
/// <summary>
/// Method that provides the T-SQL EXISTS call for any IQueryable (thus extending Linq).
/// </summary>
/// <remarks>Returns whether or not the predicate conditions exists at least one time.</remarks>
public static bool Exists<TSource>(this IQueryable<TSource> source, Expression<Func<TSource, bool>> predicate)
{
return source.Where(predicate).Any();
}
}
Try specifying the BatchSize property to 1000 which will batch up the insert in a 1000 record batch rather than the whole lot. You can tweak this value to find what is optimal. I have used sqlbulkcopy for similar size data and it works well.
Faced with the same issue, found that problem of OutOfMemory Exception was in DataTable.Rows maximum quantity limitations.
Solved with recreating table, with maximum 500000 rows limit.
Hope, my solution will be helpfull:
var myTable = new System.Data.DataTable();
myTable.Columns.Add("Guid", typeof(Guid));
myTable.Columns.Add("Name", typeof(string));
int counter = 0;
foreach (var row in rows)
{
++counter;
if (counter < 500000)
{
myTable.Rows.Add(
new object[]
{
row.Value.Guid,
row.Value.Name
});
}
else
{
using (var dbConnection = new SqlConnection("Source=localhost;..."))
{
dbConnection.Open();
using (var s = new SqlBulkCopy(dbConnection))
{
s.DestinationTableName = "MyTable";
foreach (var column in myTable.Columns)
s.ColumnMappings.Add(column.ToString(), column.ToString());
try
{
s.WriteToServer(myTable);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
s.Close();
}
}
}
myTable = new System.Data.DataTable();
myTable.Columns.Add("Guid", typeof(Guid));
myTable.Columns.Add("Name", typeof(string));
myTable.Rows.Add(
new object[]
{
row.Value.Guid,
row.Value.Name
});
counter = 0;
}
}

Resources