I have a question about reading MNIST dataset. I got the idea of how the MNIST dataset is constructed. However, I have no clue, how does it read through a following code. Some of you may think that the result of couts are obvious( I wrote values as a comment). But for me it doesn't make sense because it uses the same exact function four times with the same input, but it gets the different output every time.. How does it possible? Please let me know If there is any ambiguity in my question.
Thank you.
Code start:
typedef unsigned char BYTE;
int main()
{
...
FILE *fp = fopen("MNIST/train-images.idx3-ubyte", "rb");
// delcare function;
int magicNumber = readFlippedInteger(fp);
int numImages = readFlippedInteger(fp);
int numRows = readFlippedInteger(fp);
int numCols = readFlippedInteger(fp);
cout << magicNumber << endl; // 2051
cout << numImages << endl; // 60000
cout << numRows << endl; // 28
cout << numCols << endl; // 28
...
}
int readFlippedInteger(FILE *fp)
{
int ret = 0;
BYTE *temp;
temp = (BYTE*)(&ret);
fread(&temp[3], sizeof(BYTE), 1, fp);
fread(&temp[2], sizeof(BYTE), 1, fp);
fread(&temp[1], sizeof(BYTE), 1, fp);
fread(&temp[0], sizeof(BYTE), 1, fp);
return ret;
}
Please don't mix C and C++ unless it is absolutely necessary. The underlying confusion is that the call to fread "moves" the file pointer through the file for you. As #RetiredNinja noted, you are advancing the file pointer 4 bytes at a time. That's how it "knows" how to read the next value even though you didn't tell it to explicitly. You can read all about file pointers here.
An implementation using slightly more idiomatic C++ could be
#include <fstream>
#include <iostream>
#include <algorithm>
int readFlippedInteger(std::istream &in) {
char temp[sizeof(int)];
in.read(temp, sizeof(int));
std::reverse(temp, temp+sizeof(int));
return *reinterpret_cast<int*>(temp);
}
int main() {
std::ifstream fin("MNIST/train-images.idx3-ubyte", std::ios::binary);
if (!fin) {
std::cerr << "Could not open file\n";
return -1;
}
// delcare function;
int magicNumber = readFlippedInteger(fin);
int numImages = readFlippedInteger(fin);
int numRows = readFlippedInteger(fin);
int numCols = readFlippedInteger(fin);
std::cout << magicNumber << std::endl // 2051
<< numImages << std::endl // 60000
<< numRows << std::endl // 28
<< numCols << std::endl; // 28
}
An implementation that uses a user-defined stream manipulator is left as an exercise for the reader.
Related
I want to use clEnqueueReadBufferRect in OpenCL. To do it, I need to define the region as one of its passing arguement. But there is a inconsistency between references of OpenCL
In online reference, it is mention that
The (width, height, depth) in bytes of the 2D or 3D rectangle being read or written. For a 2D rectangle copy, the depth value given by region [2] should be 1.
but in the reference book, page 77, it is mentioned that
region defines the (width in bytes, height in rows, depth in slices) of the 2D or 3D rectangle being read or written. For a 2D rectangle copy, the depth value given by region [2] should be 1. The values in region cannot be 0
but unfortunately, none of those guides worked for me and I should provide region in (width in columns, height in rows, depth in slices), otherwise, when I defined them as byte not rows/columns, I got the error CL_INVALID_VALUE. Now which one is correct?
#define WGX 16
#define WGY 16
#include "misc.hpp"
int main(int argc, char** argv)
{
int i;
int n = 1000;
int filterWidth = 3;
int filterRadius = (int) filterWidth/2;
int padding = filterRadius * 2;
double h = 1.0 / n;
int width_x[2];
int height_x[2];
int deviceWidth[2];
int deviceHeight[2];
int deviceDataSize[2];
for (i = 0; i < 2; ++i)
{
set_domain_length(n, n, height_x[i], width_x[i], i);
}
float* x = new float [height_x[0] * width_x[0]];
init_unknown(x, height_x[0], width_x[0], 0);
set_bndryCond(x, width_x[0], h);
std::vector<cl::Platform> platforms;
cl::Platform::get(&platforms);
assert(platforms.size() > 0);
cl::Platform myPlatform = platforms[0];
std::vector<cl::Device> devices;
myPlatform.getDevices(CL_DEVICE_TYPE_GPU, &devices);
assert(devices.size() > 0);
cl::Device myDevice = devices[0];
cl_display_info(myPlatform, myDevice);
cl::Context context(myDevice);
std::ifstream kernelFile("iterative_scheme.cl");
std::string src(std::istreambuf_iterator<char>(kernelFile), (std::istreambuf_iterator<char>()));
cl::Program::Sources sources(1,std::make_pair(src.c_str(),src.length() + 1));
cl::Program program(context, sources);
cl::CommandQueue queue(context, myDevice);
deviceWidth[0] = roundUp(width_x[0], WGX);
deviceHeight[0] = height_x[0];
deviceDataSize[0] = deviceWidth[0] * deviceHeight[0] * sizeof(float);
cl::Buffer buffer_x;
try
{
buffer_x = cl::Buffer(context, CL_MEM_READ_WRITE, deviceDataSize[0]);
} catch (cl::Error& error)
{
std::cout << " ---> Problem in creating buffer(s) " << std::endl;
std::cout << " ---> " << getErrorString(error) << std::endl;
exit(0);
}
cl::size_t<3> buffer_origin;
buffer_origin[0] = 0;
buffer_origin[1] = 0;
buffer_origin[2] = 0;
cl::size_t<3> host_origin;
host_origin[0] = 0;
host_origin[1] = 0;
host_origin[2] = 0;
cl::size_t<3> region;
region[0] = (size_t)(deviceWidth[0] * sizeof(float));
region[1] = (size_t)(height_x[0]);
region[2] = 1;
std::cout << "===> Start writing data to device" << std::endl;
try
{
queue.enqueueWriteBufferRect(buffer_x, CL_TRUE, buffer_origin, host_origin, region,
deviceWidth[0] * sizeof(float), 0, width_x[0] * sizeof(float), 0, x);
} catch (cl::Error& error)
{
std::cout << " ---> Problem in writing data from Host to Device: " << std::endl;
std::cout << " ---> " << getErrorString(error) << std::endl;
exit(0);
}
// Build the program
std::cout << "===> Start building program" << std::endl;
try
{
program.build("-cl-std=CL2.0");
std::cout << " ---> Build Successfully " << std::endl;
} catch(cl::Error& error)
{
std::cout << " ---> Problem in building program " << std::endl;
std::cout << " ---> " << getErrorString(error) << std::endl;
std::cout << " ---> " << program.getBuildInfo<CL_PROGRAM_BUILD_LOG>(myDevice) << std::endl;
exit(0);
}
std::cout << "===> Start reading data from device" << std::endl;
// read result y and residual from the device
buffer_origin[0] = (size_t)(filterRadius * sizeof(float));
buffer_origin[1] = (size_t)filterRadius;
buffer_origin[2] = 0;
host_origin[0] = (size_t)(filterRadius * sizeof(float));
host_origin[1] = (size_t)filterRadius;
host_origin[2] = 0;
// region of x
region[0] = (size_t)((width_x[0] - padding) * sizeof(float));
region[1] = (size_t)(height_x[0] - padding);
region[2] = 1;
try
{
queue.enqueueReadBufferRect(buffer_x, CL_TRUE, buffer_origin, host_origin,
region, deviceWidth[0] * sizeof(float), 0, deviceWidth[0] * sizeof(float), 0, x);
} catch (cl::Error& error)
{
std::cout << " ---> Problem reading buffer in device: " << std::endl;
std::cout << " ---> " << getErrorString(error) << std::endl;
exit(0);
}
delete[] (x);
return 0;
}
The online reference link you provided says:
region
The (width in bytes, height in rows, depth in slices) of the 2D or 3D rectangle being read or written. For a 2D rectangle copy, the depth value given by region[2] should be 1. The values in region cannot be 0.
This is consistent with what you quoted later as "reference book". That's because your first link points to OpenCL 2.0 while the second link to 1.2.
The inconsistency you mention exist between online manual of 1.2 and the PDF of 1.2, but the online manual of 2.0 is consistent with the PDF. So i assume it was a bug in 1.2 online manual which was fixed in 2.0
otherwise, when I defined them as byte not rows/columns
What's a "column", and how is it different from bytes ?
The "elements" of buffer rect copy are always bytes. If you're reading/writing a 1D rect from a buffer, it simply transfers region[0] bytes. The reason why the API has "rows" and "slices" is because if using 2D/3D regions, you can have padding between data; but you can't have padding between elements in a 1D region.
I found out what is the reason of the problem, that's according to the online reference
CL_INVALID_VALUE if host_row_pitch is not 0 and is less than region[0].
so enqueueWriteBufferRect should change as follow:
queue.enqueueWriteBufferRect(buffer_x, CL_TRUE, buffer_origin, host_origin, region,
deviceWidth[0] * sizeof(float), 0, deviceWidth[0] * sizeof(float), 0, x);
which means host_row_pitch = deviceWidth[0] * sizeof(float) instead of host_row_pitch = width_x[0] * sizeof(float).
I'm running into a problem, trying to perform a template matching using OpenCV on Ubuntu 18.04LTS
#include "opencv2/highgui/highgui.hpp"
#include "opencv2/imgproc/imgproc.hpp"
#include <iostream>
#include <stdio.h>
using namespace std;
using namespace cv;
int main( int argc, char** argv )
{
int match_method =5;
string image_window = "Source Image";
string result_window = "Result window";
Mat img, templ, result;
/// Load image and template
img = imread("./RI2.jpg", IMREAD_GRAYSCALE );
templ = imread("./Pump2.jpg", IMREAD_GRAYSCALE );
/// Create windows
//namedWindow( image_window, WINDOW_AUTOSIZE );
//namedWindow( result_window, WINDOW_AUTOSIZE );
/// Source image to display
Mat img_display;
img.copyTo( img_display );
/// Create the result matrix
int result_cols = img.cols - templ.cols + 1;
int result_rows = img.rows - templ.rows + 1;
result.create( result_rows, result_cols, CV_32FC1 );
/// Do the Matching and Normalize
matchTemplate( img, templ, result, match_method );
normalize( result, result, 0, 1, NORM_MINMAX, -1, Mat() );
Mat resultgrey(result_rows, result_cols, CV_8UC1);
cout << "resultgrey.size().width: " << resultgrey.size().width << endl;
cout << "resultgrey.size().height: " << resultgrey.size().height << endl;
cout << "result.size().width: " << result.size().width << endl;
cout << "result.size().height: " << result.size().height << endl;
if( match_method == 0 || match_method == 1 )
{
for (int i=0; i<result.size().width; i++)
{
for (int j=0; j<result.size().height; j++)
{
if (result.at<float>(i,j)>=0.1)
{
resultgrey.at<int>(i,j)=0;
}
else
{
resultgrey.at<int>(i,j)=1;
}
}
}
}
else
{
for (int i=0; i<result.size().width; i++)
{
for (int j=0; j<result.size().height; j++)
{
if (result.at<float>(i,j)<=0.98)
{
resultgrey.at<int>(i,j)=0;
//cout << "0" << endl;
}
else
{
resultgrey.at<int>(i,j)=1;
//cout << "1" << endl;
}
}
}
}
cout << "3" << endl;
/// Localizing the objects
vector<Point> matchLoclist;
//cout << resultgrey << endl;
findNonZero(resultgrey, matchLoclist);
cout << "4" << endl;
if (matchLoclist.size() == 0)
{
cout << "no matches found" << endl;
return 0;
}
///Draw Rectangles on Pumps found in the scene
for (int i=0; i<matchLoclist.size(); i++)
{
//cout << "matchLoclist[i].x: "<<matchLoclist[i].x << endl << "matchLoclist[i].y: " << matchLoclist[i].y << endl;
rectangle( img_display, matchLoclist[i], Point( matchLoclist[i].x + templ.cols, matchLoclist[i].y + templ.rows ), Scalar::all(0), 2, 8, 0 );
rectangle( result, matchLoclist[i], Point( matchLoclist[i].x + templ.cols, matchLoclist[i].y + templ.rows ), Scalar::all(0), 2, 8, 0 );
}
imshow( image_window, img_display );
imshow( result_window, result );
waitKey(0);
return 0;
}
as an output i get:
xxx#ubuntu:~/Projects/Template_matching$ ./template_matching
resultgrey.size().width: 1216
resultgrey.size().height: 723
result.size().width: 1216
result.size().height: 723
Segmentation fault (core dumped)
This happens during the double for-loop where either a 1 or a 0 gets written into "resultrgrey" as I never get the "3" as an output from the cout below
if I take different input pictures (espacially smaller ones) the programm tends to run without this error.
I appreciate any help or suggestions!
Alex
You write outside of the allocated buffer because of (1) incorrectly specified data types and (2) swapped arguments to .at, as #rafix07 has noted.
You create 8-bit matrix (8 in CV_8UC1):
Mat resultgrey(result_rows, result_cols, CV_8UC1);
but try to assign 32-bit values to its elements in double-for loop:
resultgrey.at<int>(i,j)=0;
Template method cv::Mat::at calculates address of the (i,j)-th element in memory, based on:
data type, specified in template instantiation,
pointer to data start, stored in the cv::Mat instance,
and data stride (distance in bytes between leftmost pixels of two consecutive lines), also stored in the cv::Mat instance.
Then it returns reference to it. No checks is performed, for speed, therefore it's your responsibility to submit correct arguments.
Size of int is 32 bits on most modern platforms, but can be differrent.
Generally, it is safer to use types from stdint.h header, that have explicit length and sign in their names: uint8_t, int32_t, etc
Look at reference about Mat::at method
const _Tp& cv::Mat::at ( int i0, int i1 ) const
Parameters
i0 Index along the dimension 0
i1 Index along the dimension 1
the first dimenstion is number of rows, the second dim is number of columns, so you should change all lines in your code with at
resultgrey.at<int>(i,j) // i means col, j means row
to
resultgrey.at<int>(j,i)
In libc++ i have found that basic_string destructor does not gets called , once string goes out of the scope the memory is freed by calling delete operator rather than calling its destructor and then calling the delete operator from destructor, why so?
Can some one explain this?
see the sample program
void * operator new ( size_t len ) throw ( std::bad_alloc )
{
void * mem = malloc( len );
if ( (mem == 0) && (len != 0) )
throw std::bad_alloc();
return mem;
}
void operator delete ( void * ptr ) throw()
{
if ( ptr != 0 )
free( ptr );
}
int main(int argc, const char * argv[])
{
std::string mystr("testing very very very big string for string class");
std::string mystr2(mystr1.begin(),mystr.end());
}
Put break point on new and delete and then check the call stack.
new operator gets called from basic_string class while the delete gets called from the end of main, while ideally basic_string destructor should have called first and then the delete operator should be called via deallocate call of allocator, this is valid for 2nd string creation.
I'm seeing the same thing in the debugger that you are; I don't know for sure, but I suspect that stuff is getting inlined. The destructor for basic_string is very small; a single test (for the small string optimization), and then a call to the allocator's deallocate function (through allocate_traits). std::allocators allocate function is also quite small, just a wrapper around operator delete.
You could test this by writing your own allocator. (Later: see below)
More stuff that I generated while investigating this question; read on if you're interested.
[Note: there's a bug in your code - in the second line you wrote: mystr1.begin(),mystr.end()) - where is mystr1 declared?]
Assuming that's a typo, I tried some slightly different code:
#include <string>
#include <new>
#include <iostream>
int news = 0;
int dels = 0;
void * operator new ( size_t len ) throw ( std::bad_alloc )
{
void * mem = malloc( len );
if ( (mem == 0) && (len != 0) )
throw std::bad_alloc();
++news;
return mem;
}
void operator delete ( void * ptr ) throw()
{
++dels;
if ( ptr != 0 )
free( ptr );
}
int main(int argc, const char * argv[])
{
{
std::string mystr("testing very very very big string for string class");
std::string mystr2(mystr.begin(),mystr.end());
std::cout << "News = " << news << "; Dels = " << dels << std::endl;
}
std::cout << "News = " << news << "; Dels = " << dels << std::endl;
}
If you run this code, it prints (at least for me):
News = 2; Dels = 0
News = 2; Dels = 2
which is exactly what it should.
If I toss the code into compiler explorer, then I see both the calls to basic_string::~basic_string(), exactly as I expect. (Well, I see three of them, but one of them is in an exception handling block, which ends with a call to _Unwind_resume).
Later - this code:
#include <string>
#include <new>
#include <iostream>
int news = 0;
int dels = 0;
template <class T>
class MyAllocator
{
public:
typedef T value_type;
MyAllocator() noexcept {}
template <class U>
MyAllocator(MyAllocator<U>) noexcept {}
T* allocate(std::size_t n)
{
++news;
return static_cast<T*>(::operator new(n*sizeof(T)));
}
void deallocate(T* p, std::size_t)
{
++dels;
return ::operator delete(static_cast<void*>(p));
}
friend bool operator==(MyAllocator, MyAllocator) {return true;}
friend bool operator!=(MyAllocator, MyAllocator) {return false;}
};
int main(int argc, const char * argv[])
{
{
typedef std::basic_string<char, std::char_traits<char>, MyAllocator<char>> S;
S mystr("testing very very very big string for string class");
S mystr2(mystr.begin(),mystr.end());
std::cout << "Allocator News = " << news << "; Allocator Dels = " << dels << std::endl;
}
std::cout << "Allocator News = " << news << "; Allocator Dels = " << dels << std::endl;
}
prints:
Allocator News = 2; Allocator Dels = 0
Allocator News = 2; Allocator Dels = 2
which confirms that the allocator is getting called.
I am a newbie to OpenCV, so pls bear with me.. I am trying to dump the histogram Mat object for the given image.. It fails with the below error - Any help appreciated...
The first cout in the below program i.e of the loaded image prints successfully - While the second cout of the hist of the image fails with the below error
OpenCV Error: Assertion failed (m.dims <= 2) in FormattedImpl, file /mycode/ws/opencv/opencv-3.0.0-beta/modules/core/src/out.cpp, line 86
libc++abi.dylib: terminating with uncaught exception of type cv::Exception: /mycode/ws/opencv/opencv-3.0.0-beta/modules/core/src/out.cpp:86: error: (-215) m.dims <= 2 in function FormattedImpl
Here is the complete code
#include <stdio.h>
#include <string>
#include <opencv2/opencv.hpp>
using namespace std;
using namespace cv;
int main(int argc, char** argv) {
if (argc != 2) {
printf("usage: opencv.out <Image_Path>\n");
return -1;
}
string imagePath = (argv[1]);
cout << "loading image..." << imagePath << endl;
Mat image = imread(imagePath, 1);
Mat hist;
int imgCount = 1;
int dims = 3;
const int histSizes[] = {4, 4, 4};
const int channels[] = {0, 1, 2};
float rRange[] = {0, 256};
float gRange[] = {0, 256};
float bRange[] = {0, 256};
const float *ranges[] = {rRange, gRange, bRange};
Mat mask = Mat();
calcHist(&image, imgCount, channels, mask, hist, dims, histSizes, ranges);
cout << image << "Loaded image..." << endl;
cout << "Hist of image..." << hist;
return 0;
}
Based on the OpenCV 2.4.9 source code:
static inline std::ostream& operator << (std::ostream& out, const Mat& mtx)
{
Formatter::get()->write(out, mtx);
return out;
}
Is the function you are calling when using << operator. Formatter::get() returns appropriate
formatter class based on the programming language you are using.
write() function basicly calls:
static void writeMat(std::ostream& out, const Mat& m, char rowsep, char elembrace, bool singleLine)
{
CV_Assert(m.dims <= 2);
int type = m.type();
char crowbrace = getCloseBrace(rowsep);
char orowbrace = crowbrace ? rowsep : '\0';
if( orowbrace || isspace(rowsep) )
rowsep = '\0';
for( int i = 0; i < m.rows; i++ )
{
if(orowbrace)
out << orowbrace;
if( m.data )
writeElems(out, m.ptr(i), m.cols, type, elembrace);
if(orowbrace)
out << crowbrace << (i+1 < m.rows ? ", " : "");
if(i+1 < m.rows)
{
if(rowsep)
out << rowsep << (singleLine ? " " : "");
if(!singleLine)
out << "\n ";
}
}
}
As you can see if your Mat dimensionality is greater than 2 assertion will be thrown like in your code (CV_Assert(m.dims<=2)).
calcHist() with the parameters you gave produces 3-dimentional Mat and thus it cannot be displayed using << operator
By calling calcHist() function that way you are getting 3-dimentional histogram and I don't see a simple solution to visualize that in OpenCV (which doesn't mean it can't be done). If it's something you must do I would suggest to look into OpenGL for 3D data visualization. If not you could simply call this function for each channel seperatly - you will get 3 one-dimenational histograms which you can print using << operator.
I want to copy the rows 0, 2 and 4 of my matrix A into B, in this order.
Let A = [a0, a1, a2, a3, a4]^T , with a_i being row-vectors,
then B should be: [a0, a2, a4]^T.
The code below does what I want but I wonder whether there is a prettier solution (maybe using Eigen)?
#include <iostream>
#include <vector>
#include <opencv/cv.h>
int main(int argc, char **argv) {
const int num_points = 5;
const int vec_length = 3;
cv::Mat A(num_points, vec_length, CV_32FC1);
cv::RNG rng(0); // Fill A with random values
rng.fill(A, cv::RNG::UNIFORM, 0, 1);
// HACK Ugly way to fill that matrix .
cv::Mat B = cv::Mat(3,vec_length, CV_32FC1);
cv::Mat tmp0 = B(cv::Rect(0,0,vec_length,1));
cv::Mat tmp1 = B(cv::Rect(0,1,vec_length,1));
cv::Mat tmp2 = B(cv::Rect(0,2,vec_length,1));
A.row(0).copyTo(tmp0);
A.row(2).copyTo(tmp1);
A.row(4).copyTo(tmp2);
std::cout << "A: " << A << std::endl;
std::cout << "B: " << B << std::endl;
return 0;
}
The way in OpenCV 2.4.1 is:
A.row(0).copyTo(B.row(0));
A.row(2).copyTo(B.row(1));
A.row(4).copyTo(B.row(2));
I found push_back.
Create B with size 0 x vec_length and then use push_back to add the selected rows from A:
#include <iostream>
#include <vector>
#include <opencv/cv.h>
int main(int argc, char **argv) {
const int num_points = 5;
const int vec_length = 3;
cv::Mat A(num_points, vec_length, CV_32FC1);
cv::RNG rng(0); // Fill A with random values
rng.fill(A, cv::RNG::UNIFORM, 0, 1);
cv::Mat B = cv::Mat(0,vec_length, CV_32FC1);
B.push_back(A.row(0));
B.push_back(A.row(2));
B.push_back(A.row(4));
std::cout << "A: " << A << std::endl;
std::cout << "B: " << B << std::endl;
return 0;
}
Since they are non-contiguous I don't think there's any shortcut. In this particular case you could make the code cleaner with a loop:
for (int i=0; i<3; i++) {
cv::Mat tmp = B(cv::Rect(0,i,vec_length,1));
A.row(i * 2).copyTo(tmp);
}
Mat row = old.row(0);
old.row(0).copyTo(row);
new.push_back(row);