I was working on generating some excel reports and those sheets contains lot of data. I cannot use below line as I am not going in linear way
worksheet.add_row [array_of_data]
For this I first initialize a row in the below way
worksheet.add_row Array.new(maxmimum_columns, nil)
Then updated the value accordingly. Here 'maxmimum_columns' is calculated dynamically. I found that when this value becomes large and tries to open file in excel then while opening file I encounter a warning that file might have some virus. What should I do? I am unable to find some proper documentation. The maximum_colmns is less then the maximum columns of excel 2010 16384.
Thanks in advance.
Related
I want to work with the ages of individuals from their eeg records for research. I am using these tuh_eeg dataset (https://isip.piconepress.com/projects/tuh_eeg/downloads/tuh_eeg_abnormal/). I have tried different methods in matlab and python to somehow get the age but i failed. Please help me with this.
As you can see in the image, when i opened an eeg file with "edfbrowser" , there is a text showing "Age". That means the age is somewhere in the file but i can not access it
Use the function getPatient() of edfreader.py in EDFlib
Unfortunately, your EDF file is in an older (deprecated) version of EDF
that does not yet support the birthdate of the subject but you can filter
the string returned by getPatient() for "age:".
FYI, in EDFbrowser go to File -> Info to see all information stored in the EDF header.
Here you find a code example of how to use edfreader.py.
So i'm working on a OBJ/GLTF2 converter, and for simplicity i've decided to use one file for every kind of buffer, i have positions.bin (vertex) indices.bin Normals.bin and Uvs.bin the exported files open with windows 10 visualizer but the GLTF validator prints a bunch of MESH_PRIMITIVE_ACCESSOR_WITHOUT_BYTESTRIDE errors.
The file is structured so every buffer binary file have just one view and many accessor with offset (one for each face)
I'm doing something wrong ? or the validator isn't working as expected ? my data is tightly packed so i see no reason to have a ByteStride ...
I haven't an hosting so i'm using we transfer here, sorry for that
Example file
This question has been answered here: https://github.com/KhronosGroup/glTF/issues/1198
To sum up the explainations is that the bytestride can be deducted by the software that reads the GLTF as long as the bufferview isn't shared among accessors, tightly packed data still have the bytestride, it just happen to be equal to the data length and MUST be specified when it can't be deducted.
I need to use maxima software to deal with data. I try to read data from a text file constructed as
1 2 3
11 22 33
ect.
Following comands allow for loading data sufficiently.
load(numericalio);
read_matrix("path to the file");
The problem arises when I apply them to a more realistic (larger) data set. In this case the message appears Expression longer than allowed by the configuration setting.
How to overcome this problem? I cannot see any option in configuration menu. I would be grateful for advice.
I ran into the same error message today, at it seems to be related to the size of the output that wxMaxima receives from the Maxima executable.
If you wish to display the output regardless, you can change it in the configuration here:
Edit>Configure>Worksheet>Show long expressions
Note that showing a massive expression or amount of data may dramatically slow the program down, so consider hiding the output (use a $ instead of a ; at the end of your lines) if you don't need to visualize the data.
I do not know if the heading is correct - so editing is allowed to make it proper.
Problem - using Vb.net code, when I read an excel file of 100,000 records, using connection string and sql query, it takes 3 minutes ( too long to me - I want a solution, please) to complete.
But, when I submit another excel file of 300,000 records ( my requirement is to read 50 Million records) - the time taken was more than 30 minutes ( I could not tolerate and killed the program)
Please help me understand this disparity and why it takes so long to read.
(I did not give any code samples because thousands of such sample codes are available on the net on how to establish a connection to a excel file ( Office 2010) and how to run a query to read a record )
Thanks in advance for your help and time. As a solution, I thought of chopping the 300,000 record file into files of 10,000 records each - but, how do I do that without wasting opening and reading time ?
Sabya
P.S - using core 2 duo with 8 GB RAM with Windows Server 2008 and Windows 7
So, i don't work with vb.net but if you familiar with java i can advice you Apache POI library. POI load all data in memory and for my cases it works perfect, after that you can store it to mysql or anything else i read a hundred of files with poi and it helps me great.
Here i find a question which looks like similar to yours.
And here you can find POI performance discussion.
And another solution can be to export excel file to csv and read it after that, i think it'll also fast.
You could temporarily disable the Macro run as soon as Excel loads.
Memory limitation is another reason, as excel could use very large amount of memory. I would exhaust out the memory banks to 16GB if I am running this large spreadsheet (100K) cells).
Make sure the excel file and the hard drive is defragmented (you can see a real impact).
If never shutdown the PC, try shutdown and restart. This can liberate processes to unload unused dlls.
Increase the pagefile.sys size to at least 2.5 times RAM so that data transaction occurs smoothly.
Ishikawa asked, if vb.net is essential - my answer is yes, because, it is a part of an application written in VB.Net Framework 4.0. He also talked about exporting the excel to csv and try - but, I am afraid, if opening and reading is taking so many hours, ( it took 9 hours !!) - converting will not help. User will be killing the process - I am sure.
Soandos asked for the query - it is - "Select top 1* from excel-file" - I am reading one by one. I think, the problem is not this query because this same query reads 100,000 records quite well.
KronoS supported Soandos and I have answered above. To his/her 2nd point, the answer is - I have to have excel as - this is whatthe user provides. I can not change it.
I do not see who answered this - but the idea of disabling Macros - is a very good point. Should I not disable all macro, all filters and unhide all - to read all data in simple way ? I will try that.
The total size of the 300,000 record excel file is 61 MB - it is not very large !! to create a memory problem ?
I found that the speed of simply reading records in excel is not linear. It reads 10,000 records in 4 sec but, reads 50,000 in 27 sec and 100,000 in 60 sec etc..I wish - if anyone can tell me how to index an excel file to read large files. I do not know what will be the problem size, when I get an excel file of 50 Million rows ?
I had similar problems with updating large excel file. my solution - update part of it, close, kill excel process, reopen, update again
oexcel.DisplayAlerts = False
obook.SaveAs(fnExcNew, 1)
obook.Close()
obook = Nothing
KillExcel()
oexcel = CreateObject("Excel.Application")
oexcel.Workbooks.Open(fnExcNew)
obook = oexcel.ActiveWorkbook
osheet = oexcel.Worksheets(1)
Private Sub KillExcel()
' Kill all excel processes
Dim pList() As Process
Dim pExcelProcess As System.Diagnostics.Process
pList = pExcelProcess.GetProcesses
For Each pExcelProcess In pList
If pExcelProcess.ProcessName.ToUpper = "EXCEL" Then
pExcelProcess.Kill()
End If
Next
End Sub
Workbook names in Excel 2007 are supposed to be limited in size only by memory, but this appears not to be the case. Eventually, an array saved to a name will get big enough that when you try to save you get (paraphrased): "one or more of the formulas in this workbook is larger than the 8192 character limit, please save as binary file".
OK, so then save it to a binary file format... but even here, an array can get big enough to make saving the file impossible.
What gives? How are names being stored within Excel that this occurs? Is this something particular to the installation? Is there a way around it?
Try it out yourself with the code below. It will run perfectly and the name will be properly populated, but saving will give you somenasty errors. 3351 elements is too many but 3350 saves just fine:
Public Sub TestNameLimits()
Dim v As Variant
ReDim v(1)
Dim index As Integer
For index = 1 To 3351
ReDim Preserve v(index)
v(index) = "AAAA"
Next
Call Application.Names.Add("NameLimit", v)
End Sub
The Names collection is a feature of Excel that has been around for a very long time. The formula length limitation in Excel 2003 is 1,024 (2^10) but was expanded for Excel 2007 to 8,192 (2^13).
These two articles describe the main size limitations for Excel 2003 and Excel 2007:
Excel 2003 specifications and limits
Excel 2007 specifications and limits
To solve this, I would have a look at the Excel.Worksheet.CustomProperties collection. I believe that the Worksheet.CustomProperties item size is limited only by memory. You will have to test this on your system, and probably in various versions of Excel as well, but I think you should be easily able to save well over a 10,000,000 characters.
However, when using the Worksheet.CustomProperties collection, you will be responsible for converting your array to and from a string yourself, unlike the Names collection, which can convert your array to a string automatically.