How to read bytes in chunks to avoid out of memory exception? Or something else - memory

after searching and trying to find a solution for 2 days i finnaly decided to post here to ask for help...
I have a code to read and write bytes of a file (like an hex editor) working very good taken from here: http://www.novahq.net/forum/showthread.php?t=42592
Here goes the code:
Public Class Form1
Private Shared Function HexStringToByteArray(ByRef strInput As String)
As Byte()
Dim length As Integer
Dim bOutput As Byte()
Dim c(1) As Integer
length = strInput.Length / 2
ReDim bOutput(length - 1)
For i As Integer = 0 To (length - 1)
For j As Integer = 0 To 1
c(j) = Asc(strInput.Chars(i * 2 + j))
If ((c(j) >= Asc("0")) And (c(j) <= Asc("9"))) Then
c(j) = c(j) - Asc("0")
ElseIf ((c(j) >= Asc("A")) And (c(j) <= Asc("F"))) Then
c(j) = c(j) - Asc("A") + &HA
ElseIf ((c(j) >= Asc("a")) And (c(j) <= Asc("f"))) Then
c(j) = c(j) - Asc("a") + &HA
End If
Next j
bOutput(i) = (c(0) * &H10 + c(1))
Next i
Return (bOutput)
End Function
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As
System.EventArgs) Handles Button1.Click
Dim myFile As String = "..\..\Test.jpg"
Dim myBytes As Byte() =
My.Computer.FileSystem.ReadAllBytes(myFile)
Dim txtTemp As New System.Text.StringBuilder()
For Each myByte As Byte In myBytes
txtTemp.Append(myByte.ToString("X2"))
Next
RichTextBox1.Text = txtTemp.ToString()
End Sub
Private Sub Button2_Click(ByVal sender As System.Object, ByVal e As
System.EventArgs) Handles Button2.Click
Dim myFile As String = "..\..\Test2.jpg"
Dim myBytes As Byte() = HexStringToByteArray(RichTextBox1.Text)
My.Computer.FileSystem.WriteAllBytes(myFile, myBytes, False)
End Sub
End Class
This is working very good, BUT only with small/medium size files. If i try to load a 512MB or a bigger file i will get "System.OutOfMemoryException" error.
I know that in order to make this work (read bigger files without error) i have to read files in small pieces (or chunks whatever you wanna name it), or maybe read it byte by byte i suppose...
The only problem is that i couldn't find a way to do it. I did find some codes but for some reason i wasn't able to make that codes work :(
I apreciate if someone could give me a push on this.
...However, i've thinking and maybe i don't need to read the whole file because i just need to add some bytes to the end of the file and then remove those again.
I'll try to explain what i need:
I only need to add this "504B0506000000000000000000000000000000000000" bytes at the end of a zip in onder do blind/hide contents of zip and then i need to read in again and remove those bytes.
a) Append bytes to the end of file
b) Remove those bytes again
This sounds really simple but from all codes and examples i have found i couldn't adapt them to my code.
Thank you

Related

Using AKSampleDescriptor

Using AKSamplerDescriptor
I am using an adapted AKSampler example, in which I try to use the sforzando output of Fluid.sf3 melodicSounds. Sforzando creates .sfz files for each instrument, but all pointing for the global sample to a huge .wav file.
In all the instrument.sfz files there is an offset and endpoint description for the part of the wave file to be used.
When I load the .sfz file I get a crash due to memory problems. It seems that for every defined region in the .sfz file the complete .wav file (140 mB) is loaded again.
The most likely is that loading the sample file with the AKSampleDescriptor as done in the AKSampler example will ignore offset and endpoint (AKSampleDescriptor.startPoint and AKSampleDescriptor.endPoint) while reloading the complete .wav file.
Is there a way to load just the part start-to-end wanted from the sample file, because the complete file has al the sample data for all the instruments (I know and use polyphony that extracts only one instrument at the time and works fine, but this is for other use)
Or, and that seems the best to me, just load the file once and than have the sampledescriptors point to the data in memory
Good suggestions, Rob. I just ran into this one-giant-WAV issue myself, having never seen it before. I was also using Sforzando for conversion. I'll look into adding the necessary capabilities to AKSampler. In the meantime, it might be easier to write a program to cut up the one WAV file into smaller pieces and adjust the SFZ accordingly.
Here is some Python 2.7 code to do this, which I have used successfully with a Sforzando-converted sf2 soundfont. It might need changes to work for you--there is huge variability among sfz files--but at least it might help you get started. This code requires the PyDub library for manipulating WAV audio.
import os
import re
from pydub import AudioSegment
def stripComments(text):
def replacer(match):
s = match.group(0)
if s.startswith('/'):
return " " # note: a space and not an empty string
else:
return s
pattern = re.compile(
r'//.*?$|/\*.*?\*/|\'(?:\\.|[^\\\'])*\'|"(?:\\.|[^\\"])*"',
re.DOTALL | re.MULTILINE
)
return re.sub(pattern, replacer, text)
def updateSplitList(splitList, regionLabels, values):
if len(values) > 3:
start = int(values['offset'])
length = int(values['end']) - start
name = regionLabels.pop(0)
splitList.add((name, start, length))
def lookupSplitName(splitList, offset, end):
for (name, start, end) in splitList:
if offset == start and end == end:
return name
return None
def outputGroupAndRegion(outputFile, splitList, values):
if values.has_key('lokey') and values.has_key('hikey') and values.has_key('pitch_keycenter'):
outputFile.write('<group> lokey=%s hikey=%s pitch_keycenter=%s\n' % (values['lokey'], values['hikey'], values['pitch_keycenter']))
elif values.has_key('key') and values.has_key('pitch_keycenter'):
outputFile.write('<group> key=%s pitch_keycenter=%s\n' % (values['key'], values['pitch_keycenter']))
if len(values) > 3:
outputFile.write(' <region> ')
if values.has_key('lovel') and values.has_key('hivel'):
outputFile.write('lovel=%s hivel=%s ' % (values['lovel'], values['hivel']))
if values.has_key('tune'):
outputFile.write('tune=%s ' % values['tune'])
if values.has_key('volume'):
outputFile.write('volume=%s ' % values['volume'])
if values.has_key('offset'):
outputFile.write('offset=0 ')
if values.has_key('end'):
outputFile.write('end=%d ' % (int(values['end']) - int(values['offset'])))
if values.has_key('loop_mode'):
outputFile.write('loop_mode=%s ' % values['loop_mode'])
if values.has_key('loop_start'):
outputFile.write('loop_start=%d ' % (int(values['loop_start']) - int(values['offset'])))
if values.has_key('loop_end'):
outputFile.write('loop_end=%d ' % (int(values['loop_end']) - int(values['offset'])))
outputFile.write('sample=samples/%s' % lookupSplitName(splitList, int(values['offset']), int(values['end'])) + '.wav\n')
def process(inputFile, outputFile):
# create a list of region labels
regionLabels = list()
for line in open(inputFile):
if line.strip().startswith('region_label'):
regionLabels.append(line.strip().split('=')[1])
# read entire input SFZ file
sfz = open(inputFile).read()
# strip comments and create a mixed list of <header> tags and key=value pairs
sfz_list = stripComments(sfz).split()
inSection = "none"
default_path = ""
global_sample = None
values = dict()
splitList = set()
# parse the input SFZ data and build up splitList
for item in sfz_list:
if item.startswith('<'):
inSection = item
updateSplitList(splitList, regionLabels, values)
values.clear()
continue
elif item.find('=') < 0:
#print 'unknown:', item
continue
key, value = item.split('=')
if inSection == '<control>' and key == 'default_path':
default_path = value.replace('\\', '/')
elif inSection == '<global>' and key == 'sample':
global_sample = value.replace('\\', '/')
elif inSection == '<region>':
values[key] = value
# split the wav file
bigWav = AudioSegment.from_wav(global_sample)
#print "%d channels, %d bytes/sample, %d frames/sec" % (bigWav.channels, bigWav.sample_width, bigWav.frame_rate)
frate = float(bigWav.frame_rate)
for (name, start, length) in splitList:
startMs = 1000 * start / frate
endMs = 1000 * (start + length) / frate
wav = bigWav[startMs : endMs]
wavName = 'samples/' + name + '.wav'
wav.export(wavName, format='wav')
# parse the input SFZ data again and generate the output SFZ
for item in sfz_list:
if item.startswith('<'):
inSection = item
outputGroupAndRegion(outputFile, splitList, values)
values.clear()
continue
elif item.find('=') < 0:
#print 'unknown:', item
continue
key, value = item.split('=')
if inSection == '<control>' and key == 'default_path':
default_path = value.replace('\\', '/')
elif inSection == '<global>' and key == 'sample':
global_sample = value.replace('\\', '/')
elif inSection == '<region>':
values[key] = value
dirPath = '000'
fileNameList = os.listdir(dirPath)
for fileName in fileNameList:
if fileName.endswith('.sfz'):
inputFile = os.path.join(dirPath, fileName)
outputFile = open(fileName, 'w')
print fileName
process(inputFile, outputFile)

Google Closure Compile REST Api suddenly Throws Error 405 "Method not allowed"

I've been using Closure as part of y application to compress Javascript for a while now, but just started getting an error 405 - Method not allowed
My code as below was working for a couple of years, but has now stopped.
NOTE: this is not called frequently, just when ever any changes are detected in the Javascript files in my application.
I get no further error information from the Closure app than this.
Obviously this code performs a POST operation. If I use the form found here https://closure-compiler.appspot.com/home, it works, but if I either browser to the URL or use my code I get Error 405, it's almost as if my code is trying a GET method... but it's not...
Any ideas?
Public Class Closure
Property OriginalCode As String
Property CompiledCode As String
Property Errors As ArrayList
Property Warnings As ArrayList
Property Statistics As Dictionary(Of String, Object)
Public Sub New(Input As String, Optional CompliationLevel As String = "SIMPLE_OPTIMIZATIONS")
Dim _HttpWebRequest As HttpWebRequest
Dim _Result As StringBuilder
Dim ClosureWebServiceURL As String = "http://closure-compiler.appspot.com/compile?"
Dim ClosureWebServicePOSTData As String = "output_format=json&output_info=compiled_code" &
"&output_info=warnings" &
"&output_info=errors" &
"&output_info=statistics" &
"&compilation_level=" & CompliationLevel & "" &
"&warning_level=default" &
"&js_code={0}"
'// Create the POST data
Dim Data = String.Format(ClosureWebServicePOSTData, HttpUtility.UrlEncode(Input))
_Result = New StringBuilder
_HttpWebRequest = DirectCast(WebRequest.Create(ClosureWebServiceURL), HttpWebRequest)
_HttpWebRequest.Method = "POST"
_HttpWebRequest.ContentType = "application/x-www-form-urlencoded"
'//Set the content length to the length of the data. This might need to change if you're using characters that take more than 256 bytes
_HttpWebRequest.ContentLength = Data.Length
'//Write the request stream
Using SW As New StreamWriter(_HttpWebRequest.GetRequestStream())
SW.Write(Data)
End Using
Try
Dim response As WebResponse = _HttpWebRequest.GetResponse()
Using responseStream As Stream = response.GetResponseStream
Dim encoding As Encoding = System.Text.Encoding.GetEncoding("utf-8")
Using readStream As New StreamReader(responseStream, encoding)
Dim read(256) As Char
Dim count As Integer = readStream.Read(read, 0, 256)
While count > 0
Dim str As New String(read, 0, count)
_Result.Append(str)
count = readStream.Read(read, 0, 256)
End While
End Using
End Using
Dim js As New JavaScriptSerializer
js.MaxJsonLength = Int32.MaxValue
Dim d As Dictionary(Of String, Object) = js.Deserialize(Of Dictionary(Of String, Object))(_Result.ToString())
Me.CompiledCode = d.NullKey("compiledCode")
Me.Warnings = TryCast(d.NullKey("warnings"), ArrayList)
Me.Errors = TryCast(d.NullKey("errors"), ArrayList)
Me.Statistics = TryCast(d.NullKey("statistics"), Dictionary(Of String, Object))
Catch ex As Exception
Me.CompiledCode = ""
If Me.Errors Is Nothing Then
Dim er As New List(Of String)
er.Add(ex.ToString())
Me.Errors = New ArrayList(er)
Else
Me.Errors.Add(ex.ToString())
End If
End Try
Me.OriginalCode = Input
End Sub
End Class
Closure REST api is redirecting to https, you may want to try to POST to "https://closure-compiler.appspot.com/compile" directly in order to avoid redirection.

VBA to read in tab delimited file

I have some code which reads in a tab delimited file where cell reference B2 matches the reference in the first column in the tab delimited file. This works fine where the text file is small. The below works on a sample file with aa bb and cc as the headers with dummy data.
Option Explicit
Sub TestImport()
Call ImportTextFile(Sheet1.Range("B1"), vbTab, Sheet2.Range("A4"))
End Sub
Public Sub ImportTextFile(strFileName As String, strSeparator As String, rngTgt As Range)
Dim lngTgtRow As Long
Dim lngTgtCol As Long
Dim varTemp As Variant
Dim strWholeLine As String
Dim intPos As Integer
Dim intNextPos As Integer
Dim intTgtColIndex As Integer
Dim wks As Worksheet
Set wks = rngTgt.Parent
intTgtColIndex = rngTgt.Column
lngTgtRow = rngTgt.Row
Open strFileName For Input Access Read As #1
While Not EOF(1)
Line Input #1, strWholeLine
varTemp = Split(strWholeLine, strSeparator)
If CStr(varTemp(0)) = CStr(Range("B2")) Then
wks.Cells(lngTgtRow, intTgtColIndex).Resize(, UBound(varTemp) + 1).Value = varTemp
lngTgtRow = lngTgtRow + 1
End If
Wend
Close #1
Set wks = Nothing
End Sub
I am trying to get the below bit of code to work using ADO as this will run much faster on a text file with a couple of million records however I am getting an error on the '.Open str' part of the code (no value given for one or more required parameters).
It looks like it is to do with how I am defining the string- could you review and see if there is something I am missing...?
Sub QueryTextFile()
t = Timer
Dim cnn As Object
Dim str As String
Set cnn = CreateObject("ADODB.Connection")
cnn.Provider = "Microsoft.Jet.OLEDB.4.0"
cnn.ConnectionString = "Data Source=C:\Users\Davids Laptop\Documents\Other Ad Hoc\Test Files\;Extended Properties=""text;HDR=Yes;FMT=Delimited;"""
cnn.Open
Dim rs As Object
Set rs = CreateObject("ADODB.Recordset")
str = "SELECT * FROM [test1.txt] WHERE [aa]=" & Chr(34) & Range("B2") & Chr(34)
With rs
.ActiveConnection = cnn
.Open str
Sheet1.Range("A4").CopyFromRecordset rs
.Close
End With
cnn.Close
MsgBox Timer - t
End Sub

How to filter Flat File Source using script component

I have the following scenario:
I have thousands of text files with the below format.The column names are written in separate lines where as the row values are delimited by Pipe(|).
START-OF-FILE
PROGRAMNAME=getdata
DATEFORMAT=yyyymmdd
#Some Text
#Some Text
#Some Text
#Some Text
#Some Text
START-OF-FIELDS
Field1
Field2
Field3
------
FieldN
END-OF-FIELDS
TIMESTARTED=Tue May 12 16:04:42 JST 2015
START-OF-DATA
Field1Value|Field2value|Field3Value|...|Field N Value
Field1Value|Field2value|Field3Value|...|Field N Value
------|...........|----|-------
END-OF-DATA
DATARECORDS=30747
TIMEFINISHED=Tue May 12 16:11:53 JST 2015
END-OF-FILE
Now I have a corresponding SQL Server table, where I can easily load the data as destination.
Since I am new to SSIS, having trouble as to how to write the Script Component so that I can filter the Source Text files and easily load into sql server table.
Thanks in advance!
There are a few ways to do it. If the format of the files are constant, there are some useful properties of the flat file connection manager editor. For example, you can add a new flat file connection into the connection managers. There are some properties such as "Rows to skip" for the above file, you could set this to 18. Then it would start at the columns line with the "|".
Another property of the flat file connection manager that may be useful is that if you open the flat file connection manager, and then click on columns in the side menu, you can set the column delimter to the pipe "|"
But if the format of the file will change, e.g. variable number of header rows, you can use a script task to remove any non-piped rows. e.g. the header and footer.
For example, you can add a method such as file.readalllines and then edit or remove the lines as needed then save the file.
Info about that method is here:
https://msdn.microsoft.com/en-us/library/s2tte0y1%28v=vs.110%29.aspx
e.g. to remove last line in script task
string[] lines = File.ReadAllLines( "input.txt" );
StringBuilder sb = new StringBuilder();
int count = lines.Length - 1; // all except last line
for (int i = 0; i < count; i++)
{
sb.AppendLine(lines[i]);
}
File.WriteAllText("output.txt", sb.ToString());
USE Below VB Script in your SSIS SCript Component Task as source
enter code here
Imports System
Imports System.Data
Imports System.Math
Imports System.IO
Imports Microsoft.SqlServer.Dts.Runtime
Imports Microsoft.SqlServer.Dts.Pipeline.Wrapper
Imports Microsoft.SqlServer.Dts.Runtime.Wrapper
<Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute()> _
<CLSCompliant(False)> _
Public Class ScriptMain
Inherits UserComponent
'Private strSourceDirectory As String
'Private strSourceFileName As String
Private strSourceSystem As String
Private strSourceSubSystem As String
Private dtBusinessDate As Date
Public Overrides Sub PreExecute()
MyBase.PreExecute()
'
' Add your code here for preprocessing or remove if not needed
''
End Sub
Public Overrides Sub PostExecute()
MyBase.PostExecute()
'
' Add your code here for postprocessing or remove if not needed
' You can set read/write variables here, for example:
Dim strSourceDirectory As String = Me.Variables.GLOBALSourceDirectory.ToString()
Dim strSourceFileName As String = Me.Variables.GLOBALSourceFileName.ToString()
'Dim strSourceSystem As String = Me.Variables.GLOBALSourceSystem.ToString()
'Dim strSourceSubSystem As String = Me.Variables.GLOBALSourceSubSystem.ToString()
'Dim dtBusinessDate As Date = Me.Variables.GLOBALBusinessDate.Date
End Sub
Public Overrides Sub CreateNewOutputRows()
'
' Add rows by calling the AddRow method on the member variable named "<Output Name>Buffer".
' For example, call MyOutputBuffer.AddRow() if your output was named "MyOutput".
'
Dim sr As System.IO.StreamReader
Dim strSourceDirectory As String = Me.Variables.GLOBALSourceDirectory.ToString()
Dim strSourceFileName As String = Me.Variables.GLOBALSourceFileName.ToString()
'Dim strSourceSystem As String = Me.Variables.GLOBALSourceSystem.ToString()
'Dim strSourceSubSystem As String = Me.Variables.GLOBALSourceSubSystem.ToString()
'Dim dtBusinessDate As Date = Me.Variables.GLOBALBusinessDate.Date
'sr = New System.IO.StreamReader("C:\QRM_SourceFiles\BBG_BONDS_OUTPUT_YYYYMMDD.txt")
sr = New System.IO.StreamReader(strSourceDirectory & strSourceFileName)
Dim lineIndex As Integer = 0
While (Not sr.EndOfStream)
Dim line As String = sr.ReadLine()
If (lineIndex <> 0) Then 'remove header row
Dim columnArray As String() = line.Split(Convert.ToChar("|"))
If (columnArray.Length > 1) Then
Output0Buffer.AddRow()
Output0Buffer.Col0 = columnArray(0).ToString()
Output0Buffer.Col3 = columnArray(3).ToString()
Output0Buffer.Col4 = columnArray(4).ToString()
Output0Buffer.Col5 = columnArray(5).ToString()
Output0Buffer.Col6 = columnArray(6).ToString()
Output0Buffer.Col7 = columnArray(7).ToString()
Output0Buffer.Col8 = columnArray(8).ToString()
Output0Buffer.Col9 = columnArray(9).ToString()
Output0Buffer.Col10 = columnArray(10).ToString()
Output0Buffer.Col11 = columnArray(11).ToString()
Output0Buffer.Col12 = columnArray(12).ToString()
Output0Buffer.Col13 = columnArray(13).ToString()
Output0Buffer.Col14 = columnArray(14).ToString()
Output0Buffer.Col15 = columnArray(15).ToString()
Output0Buffer.Col16 = columnArray(16).ToString()
Output0Buffer.Col17 = columnArray(17).ToString()
Output0Buffer.Col18 = columnArray(18).ToString()
Output0Buffer.Col19 = columnArray(19).ToString()
Output0Buffer.Col20 = columnArray(20).ToString()
Output0Buffer.Col21 = columnArray(21).ToString()
Output0Buffer.Col22 = columnArray(22).ToString()
Output0Buffer.Col23 = columnArray(23).ToString()
Output0Buffer.Col24 = columnArray(24).ToString()
End If
End If
lineIndex = lineIndex + 1
End While
sr.Close()
End Sub
End Class
Code End

Can't get listbox to display

I am stuck here. I can't get the solution to print out into the listbox. I just can't find out what I am missing. I need help. Anything you see would be a big help. This is all the code from my program:
Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
For years As Integer = 3 To 20
Lifebox.Items.Add(years.ToString)
Next years
Lifebox.SelectedIndex = 0
End Sub
Private Sub CloseButton_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles CloseButton.Click
Me.Close()
End Sub
Private Sub DisplayButton_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles DisplayButton.Click
Dim cost As Double
Dim life As Double
Dim period As Double
Dim numberperiod As Double
Dim salvage As Double
Dim depreciation As Double
Dim isConvertedCost As Boolean
Dim isConvertedLife As Boolean
Dim isConvertedSalvage As Boolean
isConvertedCost = Double.TryParse(Assetbox.Text, cost)
isConvertedLife = Double.TryParse(Lifebox.Text, life)
isConvertedSalvage = Double.TryParse(SalavageBox.Text, salvage)
For numberperiod = 1 To period Step 1
depreciation = Financial.DDB(cost, salvage, life, numberperiod)
DepBox.Text += numberperiod.ToString & " -> " & Convert.ToString(depreciation) _
& ControlChars.NewLine
Next numberperiod
If isConvertedCost AndAlso isConvertedLife AndAlso isConvertedSalvage Then
DepBox.Text = " Year Depreciation "
Else
MessageBox.Show("The cost, life, and salvage values must be numeric.",
"Sonheim Manufacturing Company", MessageBoxButtons.OK,
MessageBoxIcon.Information)
Assetbox.Focus()
End If
End Sub
End Class

Resources