How to identify Bimodal Histograms automatically? - image-processing
I have a problem where I receive a lot of histograms from some database images. Those histograms are represented as vectors (0...255), and I have to identify and work with the bimodal histograms.
Is there a formula to automatically identify which histograms are bimodal and which aren't? Since they are numeric vectors, I could use a programming language (Java/C#) to work with it.
Is there a criterion on literature to identify bimodal histograms by software?
Here are 3 examples of histograms and format inputs I'm working with. Each histogram is a vector with 256 (0...255) positions.
Histogram 1
8029, 41, 82, 177, 135, 255, 315, 591, 949, 456, 499, 688, 446, 733, 712, 1595, 2633, 3945, 6134, 9755, 9236, 11911, 11888, 9450, 13119, 8819, 5991, 4399, 6745, 2017, 3747, 1777, 2946, 1623, 2151, 454, 3015, 3176, 2211, 1080, 391, 580, 750, 473, 10424, 334, 559, 621, 340, 2794, 1094, 5274, 2822, 204, 389, 728, 268, 15, 1060, 58, 113, 2728, 52, 3166, 11, 103, 522, 107, 351, 97, 66, 565, 315, 444, 3305, 245, 647, 306, 147, 112, 103, 672, 69, 317, 61, 224, 71, 52, 479, 62, 106, 166, 215, 132, 137, 321, 998, 427, 846, 787, 542, 1054, 1429, 615, 697, 580, 642, 768, 1244, 462, 4107, 1701, 2394, 4954, 4869, 1841, 1807, 1032, 3075, 331, 488, 627, 1281, 233, 1010, 1178, 727, 830, 1619, 728, 1428, 1849, 4826, 351, 745, 320, 888, 335, 741, 1151, 734, 689, 2143, 1130, 2482, 3609, 4779, 5678, 4186, 2654, 1668, 1290, 702, 1093, 476, 438, 445, 271, 98, 368, 226, 90, 75, 26, 33, 62, 16, 824, 21, 37, 34, 24, 54, 42, 101, 112, 18, 24, 17, 15, 3, 50, 7, 6, 54, 3, 58, 9, 10, 66, 12, 11, 10, 6, 25, 11, 7, 172, 13, 18, 21, 9, 8, 9, 42, 16, 15, 6, 12, 17, 7, 591, 6, 7, 14, 24, 7, 7, 19, 87, 18, 8, 9, 9, 35, 55, 4, 17, 10, 18, 22, 46, 8, 852, 15, 14, 12, 11, 9, 3, 50, 163, 12, 4, 18, 129, 6, 35, 47, 14, 18, 150, 21, 46, 24, 0
Histogram 2
8082, 4857, 1494, 2530, 1604, 1636, 1651, 1681, 1630, 1667, 1636, 1649, 1934, 1775, 1701, 1691, 1478, 1649, 1449, 1449, 1503, 1475, 1497, 1398, 1509, 1747, 1301, 1539, 1575, 1496, 1754, 1432, 1759, 1786, 1679, 1816, 2435, 1174, 1780, 1344, 1749, 2026, 1779, 1742, 1722, 1835, 2306, 1662, 1965, 1885, 2212, 2139, 1930, 2306, 2707, 2289, 2307, 2082, 2360, 2216, 2480, 2243, 2222, 1824, 4555, 1918, 2116, 2275, 2615, 2240, 2703, 2481, 2626, 2708, 3008, 2696, 2561, 2906, 3625, 2419, 3137, 2793, 2747, 2861, 2774, 4124, 3155, 3243, 3523, 3432, 3277, 3456, 2984, 2902, 2819, 2778, 3158, 2997, 2591, 2717, 2553, 2464, 3657, 2296, 2352, 2046, 2124, 1965, 2014, 2096, 1664, 1373, 1607, 1322, 1272, 1113, 1156, 1055, 924, 881, 1019, 669, 929, 636, 590, 463, 524, 177, 1267, 378, 409, 413, 415, 435, 385, 379, 267, 413, 266, 282, 499, 194, 360, 199, 337, 92, 986, 183, 160, 230, 124, 213, 188, 334, 164, 159, 130, 143, 135, 331, 25, 118, 114, 98, 74, 301, 92, 119, 94, 72, 192, 38, 64, 100, 138, 30, 98, 65, 226, 23, 46, 78, 78, 61, 55, 234, 26, 36, 95, 31, 49, 214, 25, 34, 58, 37, 101, 20, 41, 34, 150, 16, 50, 25, 53, 18, 30, 67, 27, 36, 42, 23, 60, 12, 21, 36, 12, 45, 21, 58, 53, 18, 51, 16, 25, 9, 24, 15, 18, 30, 33, 20, 19, 12, 23, 16, 14, 21, 14, 10, 20, 13, 12, 9, 6, 9, 7, 10, 7, 2, 0, 0, 0, 0, 0, 2087
Histogram 3
50, 226, 857, 2018, 1810, 1795, 1840, 1929, 1942, 1693, 1699, 1547, 1564, 1556, 1451, 1439, 1448, 1357, 1428, 1419, 1383, 1705, 1670, 1777, 1826, 1865, 1897, 1924, 2003, 1973, 1813, 1801, 1827, 1696, 1717, 1654, 1678, 1705, 1621, 1523, 1494, 1559, 1434, 1370, 1358, 1385, 1348, 1380, 1368, 1367, 1389, 1445, 1514, 1471, 1465, 1461, 1475, 1484, 1390, 1403, 1324, 1339, 1426, 1432, 1487, 1460, 1469, 1460, 1546, 1504, 1425, 1373, 1391, 1391, 1382, 1311, 1368, 1354, 1325, 1323, 1263, 1325, 1363, 1357, 1325, 1322, 1429, 1419, 1412, 1371, 1266, 1179, 1166, 1076, 1100, 1083, 1103, 1053, 1116, 1080, 1071, 1025, 1088, 1060, 1011, 984, 958, 959, 954, 937, 982, 950, 1001, 963, 965, 875, 1010, 954, 990, 894, 959, 972, 963, 1101, 971, 1042, 1064, 1075, 1029, 1088, 1090, 1068, 1073, 1058, 1102, 1105, 1009, 1062, 1005, 1048, 973, 998, 1034, 1013, 961, 1006, 983, 948, 1031, 972, 952, 1013, 954, 964, 970, 881, 887, 967, 941, 928, 994, 1019, 1106, 1056, 1113, 1071, 1158, 1108, 1178, 1071, 1080, 1074, 1050, 1076, 1106, 1048, 973, 1042, 997, 1034, 934, 863, 935, 845, 839, 803, 764, 782, 787, 771, 766, 751, 745, 804, 789, 765, 681, 658, 690, 672, 650, 635, 695, 619, 572, 499, 535, 565, 564, 520, 516, 568, 530, 479, 507, 424, 446, 455, 380, 395, 371, 360, 391, 373, 351, 388, 426, 349, 417, 421, 400, 443, 470, 485, 456, 495, 452, 484, 457, 518, 519, 631, 652, 693, 762, 771, 807, 906, 991, 1138, 1433, 1545, 2467, 4907, 6743, 1921
smooth histogram
this filter out small local min max and noise. Use symmetric smoothing to avoid shifting to one side. I smooth from left then from the right which lower the shifting a lot.
find/count the local max peaks
Count only big enough peaks (by some treshold). If peak count is not 2 then it is not a bimodal histogram unless you have different definition of bimodal like:
noise count too
no matter how many peaks but single gap must be present
It depends on what for the histograms are used
Here is some code in C++ I busted for this:
void histograms(Graphics::TBitmap *bmp,int xs,int ys,int **pyx)
{
// clear buffer
bmp->Canvas->Brush->Color=clBlack;
bmp->Canvas->FillRect(TRect(0,0,xs,ys));
bmp->Canvas->Font->Color=clAqua;
bmp->Canvas->TextOutA( 5,5,"Raw histogram");
bmp->Canvas->TextOutA(285,5,"Smoothed histogram");
int his1[256]={ 8029, 41, 82, 177, 135, 255, 315, 591, 949, 456, 499, 688, 446, 733, 712, 1595, 2633, 3945, 6134, 9755, 9236, 11911, 11888, 9450, 13119, 8819, 5991, 4399, 6745, 2017, 3747, 1777, 2946, 1623, 2151, 454, 3015, 3176, 2211, 1080, 391, 580, 750, 473, 10424, 334, 559, 621, 340, 2794, 1094, 5274, 2822, 204, 389, 728, 268, 15, 1060, 58, 113, 2728, 52, 3166, 11, 103, 522, 107, 351, 97, 66, 565, 315, 444, 3305, 245, 647, 306, 147, 112, 103, 672, 69, 317, 61, 224, 71, 52, 479, 62, 106, 166, 215, 132, 137, 321, 998, 427, 846, 787, 542, 1054, 1429, 615, 697, 580, 642, 768, 1244, 462, 4107, 1701, 2394, 4954, 4869, 1841, 1807, 1032, 3075, 331, 488, 627, 1281, 233, 1010, 1178, 727, 830, 1619, 728, 1428, 1849, 4826, 351, 745, 320, 888, 335, 741, 1151, 734, 689, 2143, 1130, 2482, 3609, 4779, 5678, 4186, 2654, 1668, 1290, 702, 1093, 476, 438, 445, 271, 98, 368, 226, 90, 75, 26, 33, 62, 16, 824, 21, 37, 34, 24, 54, 42, 101, 112, 18, 24, 17, 15, 3, 50, 7, 6, 54, 3, 58, 9, 10, 66, 12, 11, 10, 6, 25, 11, 7, 172, 13, 18, 21, 9, 8, 9, 42, 16, 15, 6, 12, 17, 7, 591, 6, 7, 14, 24, 7, 7, 19, 87, 18, 8, 9, 9, 35, 55, 4, 17, 10, 18, 22, 46, 8, 852, 15, 14, 12, 11, 9, 3, 50, 163, 12, 4, 18, 129, 6, 35, 47, 14, 18, 150, 21, 46, 24, 0 };
int his2[256]={ 8082, 4857, 1494, 2530, 1604, 1636, 1651, 1681, 1630, 1667, 1636, 1649, 1934, 1775, 1701, 1691, 1478, 1649, 1449, 1449, 1503, 1475, 1497, 1398, 1509, 1747, 1301, 1539, 1575, 1496, 1754, 1432, 1759, 1786, 1679, 1816, 2435, 1174, 1780, 1344, 1749, 2026, 1779, 1742, 1722, 1835, 2306, 1662, 1965, 1885, 2212, 2139, 1930, 2306, 2707, 2289, 2307, 2082, 2360, 2216, 2480, 2243, 2222, 1824, 4555, 1918, 2116, 2275, 2615, 2240, 2703, 2481, 2626, 2708, 3008, 2696, 2561, 2906, 3625, 2419, 3137, 2793, 2747, 2861, 2774, 4124, 3155, 3243, 3523, 3432, 3277, 3456, 2984, 2902, 2819, 2778, 3158, 2997, 2591, 2717, 2553, 2464, 3657, 2296, 2352, 2046, 2124, 1965, 2014, 2096, 1664, 1373, 1607, 1322, 1272, 1113, 1156, 1055, 924, 881, 1019, 669, 929, 636, 590, 463, 524, 177, 1267, 378, 409, 413, 415, 435, 385, 379, 267, 413, 266, 282, 499, 194, 360, 199, 337, 92, 986, 183, 160, 230, 124, 213, 188, 334, 164, 159, 130, 143, 135, 331, 25, 118, 114, 98, 74, 301, 92, 119, 94, 72, 192, 38, 64, 100, 138, 30, 98, 65, 226, 23, 46, 78, 78, 61, 55, 234, 26, 36, 95, 31, 49, 214, 25, 34, 58, 37, 101, 20, 41, 34, 150, 16, 50, 25, 53, 18, 30, 67, 27, 36, 42, 23, 60, 12, 21, 36, 12, 45, 21, 58, 53, 18, 51, 16, 25, 9, 24, 15, 18, 30, 33, 20, 19, 12, 23, 16, 14, 21, 14, 10, 20, 13, 12, 9, 6, 9, 7, 10, 7, 2, 0, 0, 0, 0, 0, 2087 };
int his3[256]={ 50, 226, 857, 2018, 1810, 1795, 1840, 1929, 1942, 1693, 1699, 1547, 1564, 1556, 1451, 1439, 1448, 1357, 1428, 1419, 1383, 1705, 1670, 1777, 1826, 1865, 1897, 1924, 2003, 1973, 1813, 1801, 1827, 1696, 1717, 1654, 1678, 1705, 1621, 1523, 1494, 1559, 1434, 1370, 1358, 1385, 1348, 1380, 1368, 1367, 1389, 1445, 1514, 1471, 1465, 1461, 1475, 1484, 1390, 1403, 1324, 1339, 1426, 1432, 1487, 1460, 1469, 1460, 1546, 1504, 1425, 1373, 1391, 1391, 1382, 1311, 1368, 1354, 1325, 1323, 1263, 1325, 1363, 1357, 1325, 1322, 1429, 1419, 1412, 1371, 1266, 1179, 1166, 1076, 1100, 1083, 1103, 1053, 1116, 1080, 1071, 1025, 1088, 1060, 1011, 984, 958, 959, 954, 937, 982, 950, 1001, 963, 965, 875, 1010, 954, 990, 894, 959, 972, 963, 1101, 971, 1042, 1064, 1075, 1029, 1088, 1090, 1068, 1073, 1058, 1102, 1105, 1009, 1062, 1005, 1048, 973, 998, 1034, 1013, 961, 1006, 983, 948, 1031, 972, 952, 1013, 954, 964, 970, 881, 887, 967, 941, 928, 994, 1019, 1106, 1056, 1113, 1071, 1158, 1108, 1178, 1071, 1080, 1074, 1050, 1076, 1106, 1048, 973, 1042, 997, 1034, 934, 863, 935, 845, 839, 803, 764, 782, 787, 771, 766, 751, 745, 804, 789, 765, 681, 658, 690, 672, 650, 635, 695, 619, 572, 499, 535, 565, 564, 520, 516, 568, 530, 479, 507, 424, 446, 455, 380, 395, 371, 360, 391, 373, 351, 388, 426, 349, 417, 421, 400, 443, 470, 485, 456, 495, 452, 484, 457, 518, 519, 631, 652, 693, 762, 771, 807, 906, 991, 1138, 1433, 1545, 2467, 4907, 6743, 1921 };
int *his,tmp[256],a,x0,y0,x,y,h,tr=12,sm=10,peak[256],peaks;
// loop through histograms
for (y0=20,h=0;;h++)
{
x0=5;if (h==0) his=his1;
else if (h==1) his=his2;
else if (h==2) his=his2;
else break;
// rescale his <0,?> to tmp <0-100>
for (y=his[0],x=0;x<256;x++) if (y<his[x]) y=his[x]; // y=max
for ( x=0;x<256;x++) tmp[x]=(his[x]*100)/y;
// draw tmp
for (x=0;x<256;x++) for (pyx[y0+100][x0+x]=0x00004040,y=0;y<tmp[x];y++) pyx[y0+100-y][x0+x]=(40+x)*0x00010101;
x0+=280;
// smooth tmp few times
for (y=0;y<sm;y++)
{
// from both directions to avoid shifting to one side
for (x=0;x<255;x++) tmp[x]=((90*tmp[x])+(10*tmp[x+1]))/100;
for (x=255;x>0;x--) tmp[x]=((90*tmp[x])+(10*tmp[x-1]))/100;
}
// find (count) peaks
for (peaks=0,a=0,y=0,x=0;x<255;x++)
{
if (tmp[x]<tmp[x+1]){ if ((y< 0)&&(a-tmp[x]>tr)){ a=tmp[x]; } y=+1; }
else if (tmp[x]>tmp[x+1]){ if ((y>=0)&&(tmp[x]-a>tr)){ a=tmp[x]; peak[peaks]=x; peaks++; } y=-1; }
}
// draw tmp
for (x=0;x<256;x++) for (pyx[y0+100][x0+x]=0x00004040,y=0;y<tmp[x];y++) pyx[y0+100-y][x0+x]=(40+x)*0x00010101;
// draw peaks
bmp->Canvas->Pen->Color=clAqua;
bmp->Canvas->Brush->Color=clAqua;
for (a=0;a<peaks;a++)
{
x=x0+peak[a];
y=y0+100-tmp[peak[a]];
bmp->Canvas->Ellipse(x-5,y-5,x+5,y+5);
}
// draw cross for not bimodal histograms
if (peaks!=2)
{
bmp->Canvas->Pen->Color=clRed;
bmp->Canvas->MoveTo(x0 ,y0 );
bmp->Canvas->LineTo(x0+256,y0+100);
bmp->Canvas->MoveTo(x0+256,y0 );
bmp->Canvas->LineTo(x0 ,y0+100);
}
y0+=128;
}
}
you can ignore the pyx[y][x] and bmp-> stuff it is just rendering
pyx[y][x] is direct 32bit pixel access of bitmap bmp
and bmp->Canvas is VCL encapsulated Windows GDI interface of bitmap bmp
xs,ys is bitmap resolution
set treshold tr and smooth sm to suite your needs best
If you have too much different types of histogram then you need to apply dynamic tresholding or different approach for peak finding this is how it looks like for your histograms:
Where Histogram 1 is the top one. Hope the code is clear enough if not comment me... if you rescale to power of 2 instead of 100 then you can change the multiplications and divisions to bit shifts to speed this a bit. I choose 100 for more clear selection of tresholds and smoothing coefficients...
I don't think there is a simple and straightforward solution for that.
If you think each peak in your histogram as a cluster, you can try to implement some sort of clustering algorithm that is able to automatically detect the number of peaks in your histogram.
You can start looking at here:
Determining the number of clusters in a data set
Kmeans without knowing the number of clusters?
AUTOMATIC DETERMINATION OF THE NUMBER OF CLUSTERS USING SPECTRAL
ALGORITHMS
Related
how to convert 8-bit unsigned int data to signed?
I am getting List of 8-bit unsigned int from a mic source for each sample rate which looks like this [61, 251, 199, 251, 56, 252, 138, 252, 211, 252, 18, 253, 91, 253, 194, 253, 25, 254, 54, 254, 19, 254, 190, 253, 80, 253, 249, 252, 233, 252, 46, 253, 180, 253, 54, 254, 136, 254, 157, 254, 110, 254, 38, 254, 208, 253, 117, 253, 68, 253, 57, 253, 83, 253, 163, 253, 20, 254, 151, 254, 51, 255, 215, 255, 105, 0, 207, 0, 246, 0, 249, 0, 10, 1, 64, 1, 162, 1, 4, 2, 64, 2, 97, 2, 111, 2, 110, 2, 89, 2, 40, 2, 241, 1, 199, 1, 178, 1, 192, 1, 241, 1, 45, 2, 77, 2, 70, 2, 45, 2, 36, 2, 83, 2, 176, 2, 21, 3, 121, 3, 229, 3, 87, 4, 185, 4, 225, 4, 197, 4, 129, 4, 26, 4, 150, 3, 7, 3, 128, 2, 55, 2, 65, 2, 134, 2, 223, 2, 25, 3, 41, 3, 28, 3, 255, 2, 234, 2, 240, 2, 25, 3, 62, 3, 92, 3, 146, 3, 219, 3, 65, 4, 149, 4, 164, 4, 130, 4, 51, 4, 195, 3, 69, 3, 164, 2, 244, 1, 75, 1, 187, 0, 81, 0, 240, 255, 135, 255, 19, 255, 155, 254, 64, 254, 22, 254, 58, 254, 146, 254, 217, 254, 248, 254, 215, 254, 144, 254, 92, 254, 84, 254, 141, 254, 229, 254, 39, 255, 96, 255, 170, 255, 248, 255, 69, 0, 117, 0, 128, 0, 137, 0, 131, 0, so how can I convert this into signed decimal value or someone can guide me to the right path
That depends on what the bytes mean. Looking at the bytes, every other byte is either very low or very high. That suggests to me that the bytes are really little-endian signed 16-bit values. In that case, you just need to view them as such. If we assume that the platform is little-endian (most are), you can just do: List<int> list = ...; Uint8List bytes = Uint8List.fromList(list); // Int16List words = Int16List.sublistView(bytes); Then the words list contains signed 16-bit numbers. (If the list is already a Uint8List, you can skip the first step.) If that's not what the bytes mean, you'll have to figure out what they do mean.
Dart int type provide a method to convert from signed to unsigned and from unsigned to signed. For example: int a = 16; int b = 239; print(a.toSigned(5).toString()); // Print -16 print(b.toSigned(5).toString()); // Print 15 the toSigned method parameter indicate the bit order of the sign bit. You can get more information here: https://api.flutter.dev/flutter/dart-core/int/toSigned.html A toUnsigned method exixts too: https://api.flutter.dev/flutter/dart-core/int/toUnsigned.html
Upgrade to OTP 18 breaks usage of public_key library
Building a pem file in Elixir requires several steps, including building an entity. In OTP 17, the following works: {public, private} = :crypto.generate_key(:ecdh, :secp256k1) ec_entity = {:ECPrivateKey, 1, :binary.bin_to_list(private), {:namedCurve, {1, 3, 132, 0, 10}}, {0, public}} der_encoded = :public_key.der_encode(:ECPrivateKey, ec_entity) pem = public_key.pem_encode([{:ECPrivateKey, der_encoded, :not_encrypted}]) But using OTP 18, the following error occurs: {public, private} = :crypto.generate_key(:ecdh, :secp256k1) ec_entity = {:ECPrivateKey, 1, :binary.bin_to_list(private), {:namedCurve, {1, 3, 132, 0, 10}}, {0, public}} der_encoded = :public_key.der_encode(:ECPrivateKey, ec_entity) ** (MatchError) no match of right hand side value: {:error, {:asn1, :badarg}} public_key.erl:253: :public_key.der_encode/2 What is the source of this error?
The source of the error is a change in the way that the public_key entity is constructed between OTP 17 and OTP 18. If we reverse the process, starting with a pem file, we can see the difference. OTP 17: iex(6)> pem = "-----BEGIN EC PRIVATE KEY-----\nMHQCAQEEIJniJF4vtTqE4wS5AkhmMZsHIbil0l3XfRButkw5IJYFoAcGBSuBBAAK\noUQDQgAEtxm+jijBB0JxZTceHnCHE0HpMXJp1ScVUZ5McvDUVsS/Dek8IdAsMOPz\nnnVALflZzXtH/wU9p2LrFdJeuXwL8g==\n-----END EC PRIVATE KEY-----\n\n" "-----BEGIN EC PRIVATE KEY-----\nMHQCAQEEIJniJF4vtTqE4wS5AkhmMZsHIbil0l3XfRButkw5IJYFoAcGBSuBBAAK\noUQDQgAEtxm+jijBB0JxZTceHnCHE0HpMXJp1ScVUZ5McvDUVsS/Dek8IdAsMOPz\nnnVALflZzXtH/wU9p2LrFdJeuXwL8g==\n-----END EC PRIVATE KEY-----\n\n" iex(7)> [{type, decoded, _}] = :public_key.pem_decode(pem) [{:ECPrivateKey, <<48, 116, 2, 1, 1, 4, 32, 153, 226, 36, 94, 47, 181, 58, 132, 227, 4, 185, 2, 72, 102, 49, 155, 7, 33, 184, 165, 210, 93, 215, 125, 16, 110, 182, 76, 57, 32, 150, 5, 160, 7, 6, 5, 43, 129, 4, 0, 10, ...>>, :not_encrypted}] iex(8)> :public_key.der_decode(type, decoded) {:ECPrivateKey, 1, [153, 226, 36, 94, 47, 181, 58, 132, 227, 4, 185, 2, 72, 102, 49, 155, 7, 33, 184, 165, 210, 93, 215, 125, 16, 110, 182, 76, 57, 32, 150, 5], {:namedCurve, {1, 3, 132, 0, 10}}, {0, <<4, 183, 25, 190, 142, 40, 193, 7, 66, 113, 101, 55, 30, 30, 112, 135, 19, 65, 233, 49, 114, 105, 213, 39, 21, 81, 158, 76, 114, 240, 212, 86, 196, 191, 13, 233, 60, 33, 208, 44, 48, 227, 243, 158, 117, ...>>}} OTP 18: iex(5)> [{type, decoded, _}] = :public_key.pem_decode(pem) [{:ECPrivateKey, <<48, 116, 2, 1, 1, 4, 32, 153, 226, 36, 94, 47, 181, 58, 132, 227, 4, 185, 2, 72, 102, 49, 155, 7, 33, 184, 165, 210, 93, 215, 125, 16, 110, 182, 76, 57, 32, 150, 5, 160, 7, 6, 5, 43, 129, 4, 0, 10, ...>>, :not_encrypted}] iex(6)> entity = :public_key.der_decode(type, decoded) {:ECPrivateKey, 1, <<153, 226, 36, 94, 47, 181, 58, 132, 227, 4, 185, 2, 72, 102, 49, 155, 7, 33, 184, 165, 210, 93, 215, 125, 16, 110, 182, 76, 57, 32, 150, 5>>, {:namedCurve, {1, 3, 132, 0, 10}}, <<4, 183, 25, 190, 142, 40, 193, 7, 66, 113, 101, 55, 30, 30, 112, 135, 19, 65, 233, 49, 114, 105, 213, 39, 21, 81, 158, 76, 114, 240, 212, 86, 196, 191, 13, 233, 60, 33, 208, 44, 48, 227, 243, 158, 117, 64, ...>>} The difference is in how the public and private keys are represented. The signature of an ECPrivateKey Record is: ECPrivateKey'{ version, privateKey, parameters, publicKey} In Erlang 18, both values are represented at plain binaries, in 17, the private key is a list and the public key is part of a tuple, {0, binary}. So in order to build the pem file correctly, the entity representation has to change. {public, private} = :crypto.generate_key(:ecdh, :secp256k1) entity = {:ECPrivateKey, 1, private, {:namedCurve, {1, 3, 132, 0, 10}}, public} Using the new representation of the record will solve the problem.
I didn't really check why your version works on some versions, but I've got some code that works on all these erlang versions: 19.0, 18.2.1, 18.1, 18.0, 17.5, R16B03 (running on travis). -include_lib("public_key/include/public_key.hrl"). genPEMKey() -> CurveId = secp256k1, {PubKey, PrivKey} = crypto:generate_key(ecdh, CurveId), Key = #'ECPrivateKey'{version = 1, privateKey = PrivKey, parameters = { namedCurve, pubkey_cert_records:namedCurves(CurveId)}, publicKey = PubKey}, DERKey = public_key:der_encode('ECPrivateKey', Key), public_key:pem_encode([{'ECPrivateKey', DERKey, not_encrypted}]). This piece of code was based on the examples found in the OTP codebase: https://github.com/erlang/otp/blob/master/lib/public_key/test/erl_make_certs.erl#L407
Query caching in controller
Rails 3.2.18 ruby 1.9.3 redis checking caching in development environment. development.rb: S2yd::Application.configure do # Settings specified here will take precedence over those in config/application.rb # In the development environment your application's code is reloaded on # every request. This slows down response time but is perfect for development # since you don't have to restart the webserver when you make code changes. config.cache_classes = false #config.assets.enabled = true # Log error messages when you accidentally call methods on nil. config.whiny_nils = true # Show full error reports and disable caching config.consider_all_requests_local = true config.action_controller.perform_caching = false # Don't care if the mailer can't send config.action_mailer.raise_delivery_errors = false # Print deprecation notices to the Rails logger config.active_support.deprecation = :log # Only use best-standards-support built into browsers config.action_dispatch.best_standards_support = :builtin config.cache_store = :redis_store, 'redis://localhost:6379/0/cache', { expires_in: 90.minutes, namespace: 'production' } controller #restaurants = Rails.cache.fetch("api/restaurants/#{#thirdparty.name}/1", expires_in: 90.minutes) do Restaurant .joins(:thirdparties, :franchise) .includes(:menu_sections, :menus) .where("`thirdparties`.`id` = ? and `restaurants`.`active` = ? and lower(`restaurants`.`name`) <> ?", #thirdparty.id, true, 'drinks') end log Processing by Api::V1Controller#restaurants as JSON Parameters: {"token"=>"eatstreet"} (0.5ms) SELECT COUNT(*) FROM `domains` Domain Load (0.6ms) SELECT `domains`.* FROM `domains` Domain Load (0.5ms) SELECT `domains`.* FROM `domains` LIMIT 1 Domain: 127.0.0.1 -> S2YD User Load (0.6ms) SELECT `users`.* FROM `users` WHERE `users`.`id` = 93626 LIMIT 1 Thirdparty Load (0.5ms) SELECT `thirdparties`.* FROM `thirdparties` WHERE `thirdparties`.`access_token` = 'eatstreet' LIMIT 1 Thirdparty: eatstreet Restaurant Load (21.7ms) SELECT `restaurants`.* FROM `restaurants` INNER JOIN `restaurants_thirdparties` ON `restaurants_thirdparties`.`restaurant_id` = `restaurants`.`id` INNER JOIN `thirdparties` ON `thirdparties`.`id` = `restaurants_thirdparties`.`thirdparty_id` INNER JOIN `franchises` ON `franchises`.`id` = `restaurants`.`franchise_id` WHERE (`thirdparties`.`id` = 3 and `restaurants`.`active` = 1 and lower(`restaurants`.`name`) <> 'drinks') Menu Load (10.2ms) SELECT `menus`.* FROM `menus` WHERE `menus`.`restaurant_id` IN (4, 6, 7, 8, 10, 12, 14, 286, 287, 288, 322, 352, 353, 355, 394, 467, 480, 503, 523, 552, 553, 555, 615, 641, 650, 744, 785, 790, 793, 797, 799, 800, 801, 802, 807, 834, 835, 863, 865, 882, 954, 1012, 1014, 1110, 1111, 1114, 1128, 1139, 1188, 1191, 1196, 1198, 1202, 1206, 1225, 1297, 1325, 1358, 1379, 1387, 1404, 1436, 1509, 1510, 1519, 18, 39, 46, 49, 62, 66, 67, 86, 112, 129, 199, 203, 205, 209, 228, 296, 305, 306, 365, 368, 381, 382, 384, 390, 395, 496, 501, 510, 561, 565, 584, 607, 694, 767, 825, 839, 857, 926, 946, 947, 970, 996, 1000, 1028, 1040, 1055, 1071, 1094, 1109, 1160, 1216, 1244, 1339, 1355, 1356, 1364, 1375, 1418, 1438, 1448, 1508, 1518, 1527, 1564, 131, 132, 134, 135, 139, 142, 218, 249, 308, 309, 310, 324, 339, 350, 351, 361, 363, 367, 369, 370, 392, 396, 400, 490, 581, 585, 597, 836, 849, 993, 1118, 1119, 1122, 1140, 1173, 1218, 1226, 1245, 1246, 1247, 1248, 1249, 1254, 1255, 1259, 1260, 1262, 1266, 1272, 1301, 1308, 1316, 1318, 1335, 1351, 1405, 1408, 1451, 1513, 146, 149, 152, 155, 160, 161, 163, 165, 167, 190, 193, 194, 283, 291, 294, 334, 345, 346, 347, 360, 388, 426, 427, 519, 536, 577, 588, 606, 712, 798, 921, 1043, 1155, 1238, 1258, 1287, 1376, 1381, 1385, 1386, 1413, 1417, 1430, 1433, 1440, 1441, 1479, 1480, 1481, 1482, 1558, 255, 256, 259, 276, 446, 453, 456, 458, 460, 461, 462, 464, 465, 466, 468, 470, 471, 474, 491, 494, 514, 558, 626, 683, 697, 730, 748, 780, 786, 842, 858, 1015, 1161, 1197, 1357, 1371, 1373, 1444, 1453, 374, 377, 378, 401, 498, 499, 528, 564, 573, 636, 809, 909, 933, 944, 407, 409, 417, 455, 476, 479, 517, 560, 591, 604, 709, 754, 771, 833, 1072, 1073, 1074, 1099, 1185, 419, 420, 436, 437, 440, 441, 492, 507, 508, 605, 634, 648, 651, 655, 710, 729, 824, 860, 1181, 1207, 1227, 1239, 1253, 1269, 1270, 1271, 1286, 1310, 1311, 1312, 1313, 1321, 1322, 1343, 1344, 1345, 1346, 1382, 1383, 1416, 488, 543, 568, 571, 598, 622, 625, 724, 1022, 1235, 1288, 1298, 1299, 1302, 1320, 1323, 1334, 1349, 1412, 1429, 1439, 1449, 1450, 1454, 1486, 1515, 1516, 1517, 1520, 1523, 529, 537, 546, 590, 608, 609, 610, 624, 680, 681, 685, 716, 831, 1150, 1151, 1153, 1178, 1186, 1363, 1512, 1514, 638, 695, 715, 717, 718, 719, 753, 781, 837, 845, 848, 850, 960, 1027, 1129, 1233, 1250, 1303, 1367, 1423, 1434, 1442, 1468, 629, 631, 656, 658, 659, 665, 669, 741, 760, 769, 778, 908, 910, 915, 916, 917, 927, 937, 938, 941, 942, 943, 955, 957, 959, 972, 973, 974, 975, 976, 977, 979, 980, 992, 994, 999, 1005, 1023, 1134, 1157, 1174, 1208, 1210, 1224, 1229, 1242, 1304, 1314, 1315, 1378, 1415, 1424, 1435, 1445, 1492, 675, 1041, 1042, 1045, 1046, 1060, 1061, 1064, 1067, 1069, 1080, 1081, 1091, 1101, 1102, 1103, 1123, 1156, 1166, 1222, 1230, 1353, 1365, 1366, 1403, 1410, 1414, 1431, 1443, 1500, 1521, 722, 761, 764, 766, 768, 770, 805, 821, 822, 859, 928, 929, 930, 931, 932, 1059, 1106, 1115, 1234, 1352, 1447, 728, 731, 732, 734, 737, 738, 750, 777, 782, 794, 795, 818, 829, 841, 1029, 1063, 1068, 1105, 1182, 1183, 1256, 1275, 1278, 1348, 1372, 1406, 1566, 854, 861, 862, 867, 874, 880, 881, 890, 893, 894, 895, 905, 936, 956, 1006, 1011, 1013, 1024, 1026, 1030, 1033, 1038, 1044, 1116, 1117, 1121, 1165, 1192, 1193, 1209, 1309, 876, 877, 878, 879, 899, 902, 907, 913, 1004, 1050, 1078, 1079, 1086, 1107, 1108, 1199, 1243, 1292, 1411, 1457, 1458, 1459, 1460, 1461, 1465, 1466, 1467, 1470, 1472, 1474, 1477, 1526, 1538, 1568, 1569, 1570, 1572, 1573, 1574, 1576, 1577, 1578, 1582, 1583, 1584) MenuSection Load (44.5ms) SELECT `menu_sections`.* FROM `menu_sections` WHERE `menu_sections`.`menu_id` IN (4, 160, 1885, 7, 8, 121, 10, 319, 12, 14, 130, 358, 2225, 2226, 251, 253, 305, 367, 368, 374, 447, 536, 2242, 554, 591, 618, 652, 663, 1812, 1826, 2235, 2257, 745, 1857, 804, 896, 971, 1798, 1000, 2455, 1816, 2418, 1015, 1005, 1010, 1009, 2389, 1019, 1082, 1072, 1903, 1099, 1104, 1113, 2477, 1246, 1346, 1357, 1518, 1519, 2390, 1525, 1539, 2196, 1555, 1643, 1659, 1664, 1668, 1917, 2557, 2559, 1843, 1900, 2292, 2088, 2112, 2116, 2173, 2198, 2180, 2191, 2272, 2406, 2412, 2490, 2408, 2444, 2445, 2446, 2447, 2448, 2449, 171, 172, 175, 2574, 21, 1092, 1889, 1976, 2011, 22, 669, 24, 25, 26, 686, 27, 2558, 351, 352, 31, 1018, 33, 129, 242, 1735, 1762, 1763, 95, 98, 1787, 1792, 1871, 2159, 2161, 2162, 2496, 2497, 103, 478, 1027, 148, 256, 1733, 1770, 2245, 1723, 1883, 505, 2526, 392, 399, 400, 1064, 1066, 1067, 414, 415, 416, 435, 1197, 1200, 2139, 1811, 1872, 1873, 1876, 1878, 1898, 587, 588, 590, 1326, 1979, 1987, 612, 615, 672, 680, 1841, 735, 1925, 865, 2499, 2500, 2502, 2503, 2505, 1058, 1942, 1943, 1190, 1191, 1606, 2293, 1097, 1217, 1241, 1349, 1273, 1275, 1276, 1277, 1278, 1279, 1327, 1649, 1650, 1651, 1652, 1653, 2106, 2107, 2280, 2281, 2309, 2310, 1387, 1395, 1425, 1426, 1436, 1457, 1495, 1523, 1617, 1619, 1793, 1794, 1780, 1781, 1782, 1888, 1918, 2097, 2127, 2108, 2115, 2125, 2128, 2163, 2437, 2239, 2284, 2338, 2433, 2393, 2399, 2410, 2421, 2462, 2584, 35, 265, 1116, 2054, 2113, 1696, 1751, 43, 46, 132, 2114, 2211, 2212, 2214, 2436, 2440, 287, 671, 673, 1542, 1703, 292, 2080, 309, 344, 346, 364, 2228, 366, 381, 384, 2146, 391, 1802, 393, 394, 1710, 443, 448, 2419, 568, 1186, 711, 2602, 2603, 702, 1372, 1075, 1129, 1757, 1321, 1529, 1766, 1884, 1567, 1724, 1725, 2650, 2651, 1796, 1797, 1836, 1837, 1926, 1928, 1931, 1929, 1932, 1934, 2342, 2343, 1980, 1946, 2523, 2010, 1950, 1951, 2219, 2060, 2018, 2035, 2034, 2079, 2575, 2164, 2197, 2531, 2203, 2344, 2405, 50, 395, 2150, 52, 1879, 2008, 55, 58, 62, 63, 65, 67, 159, 167, 398, 1702, 86, 89, 90, 238, 239, 240, 247, 252, 337, 2474, 1803, 2613, 545, 550, 1823, 1824, 1587, 1902, 2611, 480, 481, 630, 637, 2493, 2264, 2612, 701, 734, 1867, 870, 1448, 1325, 1455, 2000, 2279, 1579, 1863, 2546, 1936, 1940, 1974, 2223, 2176, 2190, 2555, 2643, 2179, 2216, 2222, 2178, 2215, 2217, 2675, 2232, 2256, 2261, 2295, 2582, 2321, 2352, 2353, 2396, 2397, 2364, 2371, 2372, 2560, 1127, 198, 2271, 2274, 230, 499, 1984, 1985, 519, 2204, 1989, 527, 528, 529, 1765, 533, 534, 539, 541, 542, 548, 569, 570, 2265, 2266, 2004, 2006, 1817, 766, 854, 863, 864, 894, 1905, 2529, 962, 978, 1079, 1916, 2556, 1359, 1604, 2282, 1821, 2109, 2111, 2154, 2155, 2480, 2313, 2341, 2348, 423, 425, 2233, 2247, 421, 2134, 2135, 2136, 420, 450, 452, 581, 2335, 583, 584, 678, 799, 1400, 683, 687, 708, 1418, 771, 822, 1033, 1175, 2165, 1239, 1259, 1265, 458, 2573, 460, 472, 518, 565, 551, 1865, 2229, 553, 613, 676, 715, 2206, 2498, 856, 2409, 872, 918, 953, 954, 958, 959, 960, 963, 2562, 2563, 2564, 2565, 2566, 1070, 1458, 1459, 1755, 1501, 1736, 1636, 470, 1911, 490, 1071, 2607, 1963, 1964, 494, 506, 1324, 2463, 2600, 2601, 600, 1743, 751, 1679, 1691, 800, 1795, 2147, 2646, 814, 1699, 817, 818, 868, 1632, 1052, 1103, 1778, 1640, 1708, 1838, 1868, 1870, 1933, 1956, 1960, 2609, 1959, 2608, 2007, 2019, 2028, 2645, 2020, 2610, 2021, 2022, 2059, 2057, 2102, 2103, 2089, 2090, 2181, 2182, 2236, 1734, 2071, 2101, 2083, 1832, 1464, 1808, 1927, 724, 760, 770, 782, 885, 2407, 1969, 1970, 1971, 1861, 2156, 2031, 2030, 2032, 2033, 2046, 2055, 2056, 2048, 2063, 2314, 2218, 2385, 2294, 2328, 2339, 2350, 2382, 2383, 2359, 2411, 2414, 2415, 2413, 2452, 2416, 2438, 2430, 632, 653, 2401, 655, 679, 2244, 2604, 2539, 2540, 750, 1534, 1839, 1840, 769, 883, 2606, 943, 2605, 866, 874, 1068, 1086, 1576, 1575, 2395, 1578, 1625, 1638, 1827, 2124, 2402, 2441, 2521, 907, 903, 873, 2188, 875, 2289, 880, 1470, 2423, 2442, 891, 2479, 2519, 916, 966, 2095, 2595, 1083, 1968, 1685, 2243, 1088, 2476, 1263, 1272, 1383, 2495, 1541, 1784, 1999, 2596, 1935, 2494, 1996, 2543, 2213, 2520, 2250, 2262, 2434, 2327, 2422, 2384, 798, 2062, 2210, 2532, 831, 823, 908, 829, 1845, 834, 1858, 895, 1666, 932, 933, 945, 946, 969, 1767, 1901, 1586, 1834, 2224, 1180, 1288, 1163, 1164, 1166, 1165, 1201, 1223, 1222, 1225, 1227, 1228, 1371, 1247, 1690, 1552, 1257, 1282, 1375, 1378, 2142, 2143, 1304, 2398, 1290, 1362, 1629, 1631, 1941, 1785, 1789, 1801, 2185, 1286, 1295, 1420, 1421, 1292, 1294, 2337, 1340, 1726, 2283, 1374, 1558, 1623, 1682, 1639, 1709, 1768, 1769, 1831, 1848, 1904, 1998, 2220, 2027, 2029, 2172, 2199, 2227, 2249, 2269, 2325, 2332, 2367, 1403, 1396, 1507, 1509, 1510, 1397, 1566, 2622, 2623, 1404, 1405, 1428, 2168, 1438, 2486, 1443, 2567, 2590, 1453, 1472, 1481, 2588, 2589, 1475, 1496, 1503, 2585, 1505, 2591, 1506, 1532, 1591, 2315, 2316, 2317, 2318, 2319, 2320, 1605, 1819, 1846, 2096, 2133, 2131, 2192, 2202, 2221, 2268, 2312, 2365, 2400, 2417, 2427, 2428, 884, 935, 941, 1687, 1850, 944, 2465, 947, 1017, 1353, 2470, 1049, 1047, 1048, 1105, 1777, 1237, 2580, 2581, 1236, 1230, 1234, 2597, 2598, 1231, 1232, 2578, 1882, 1521, 1783, 1528, 1860, 2149, 2336, 920, 2471, 1669, 2002, 2267, 2647, 928, 1497, 2511, 2512, 2513, 2514, 2515, 997, 936, 919, 956, 1323, 1986, 2148, 2472, 1694, 2484, 996, 998, 1040, 1063, 1078, 1386, 1389, 1437, 1444, 2370, 1513, 1514, 1633, 1634, 1938, 1952, 1966, 2545, 2158, 2193, 2644, 1145, 1188, 2458, 1154, 1572, 1158, 1159, 1658, 2259, 2530, 1148, 1176, 1150, 1160, 1169, 2340, 1173, 1181, 1194, 1195, 1172, 1206, 2522, 1177, 1179, 1252, 2119, 1262, 1351, 2349, 2594, 1360, 1367, 1385, 1381, 2288, 2291, 2329, 2330, 1410, 1467, 1540, 1527, 1551, 1624, 2461, 1663, 1660, 2049, 2052, 2024, 2025, 1107, 1149, 1132, 1168, 1114, 1135, 1143, 1967, 1973, 1174, 1184, 1334, 1434, 1465, 1531, 1490, 1491, 2047, 1516, 1515, 1522, 1707, 1912, 2085, 2205, 2376, 2379, 2380, 2387, 2391, 2386, 2392, 2355, 2394, 2425, 2381, 2354, 2360, 2361, 2374, 2454, 2488, 2616, 2627, 2619, 2624, 2625, 2639, 2626, 2629, 2630, 2631, 2634, 2635, 2636) ORDER BY `index` ASC Completed 200 OK in 3321.9ms (Views: 3235.1ms | ActiveRecord: 79.0ms) Seems Rails.cache.fetch doesn't work and still touches the database.
I think you should change the config. config.action_controller.perform_caching = false Try to change false to true.
How to programmatically monitor if a docker container exited?
I am running multiple named docker containers (200+) on my VM Host. I have a manager script/code that is supposed to manage the containers from the host. I would like to know if there is any event-based mechanism to get notified when a container stops/fails. So that I can restart the stopped container. One solution I could think of is doing a periodic docker inspect and looking at State.Pid or State.Running to confirm the status. But,instead of periodic polling, it would be better if the manager is notified with pid/name when a container fails so that, the particular container alone can be restarted. On a general note, are there ways to programmatically monitor the status of a process from a different process that is not the parent ?
Look at docker events - there is an event for container 'die'. There is also an http interface to get the same information programmatically - see here You may want to do a web search for 'docker orchestration' - many projects springing up to manage multiple containers in the way you describe.
If you just want to restart the containers why don't you use a restart policy? docker run --restart=always IMAGE
psutil seems to do what you want http://pypi.python.org/pypi/psutil From Python import psutil psutil.pids() [1, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 65, 66, 67, 69, 70, 71, 72, 73, 74, 76, 77, 78, 79, 80, 81, 82, 94, 97, 98, 117, 118, 137, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 200, 201, 210, 211, 213, 214, 229, 230, 411, 416, 419, 526, 527, 542, 543, 544, 545, 555, 569, 625, 709, 714, 756, 781, 782, 796, 862, 863, 864, 869, 914, 944, 945, 948, 954, 996, 1052, 1061, 1064, 1067, 1170, 1174, 1179, 1180, 1183, 1234, 1240, 1241, 1245, 1323, 1328, 1340, 1351, 1354, 1390, 1408, 1457, 1507, 1531, 1631, 1662, 1933, 1972, 1981, 1987, 1989, 1993, 2346, 2348, 2413, 2422, 2429, 2442, 2445, 2449, 2451, 2457, 2461, 2471, 2489, 2490, 2491, 2493, 2497, 2501, 2505, 2509, 2513, 2524, 2546, 2549, 2551, 2554, 2563, 2567, 2572, 2573, 2576, 2578, 2586, 2595, 2598, 2624, 2644, 2655, 2665, 2667, 2687, 2689, 2693, 2699, 2744, 2752, 2785, 2789, 2794, 2798, 2804, 2817, 2820, 2830, 2838, 2856, 2862, 2864, 2886, 2903, 2935, 2972, 2985, 2986, 3138, 3164, 3211, 3368, 3371, 3557, 4125, 4352, 4443, 4444, 4743, 4818, 4819, 4840, 4841, 4844, 4845, 4866, 4876, 6142, 6363, 6366, 6372, 6378, 6385, 6391, 6452, 6518, 6524, 6531, 6555, 6558, 6601] p = psutil.Process(2862) p.status() 'sleeping'
How to make a list of a mapping of IDs
I have a long list of group_ids like so: #group_ids = #groups.map(&:group_id) Rails.logger.info #group_ids [182, 122, 181, 173, 167, 58, 13, 11, 180, 40, 71, 1, 29, 47, 142, 52, 174, 7, 168, 171, 156, 120, 79, 72, 54, 26, 65] How can I take all those group_ids and output: group_id:11 OR group_id:22 etc
Just do: #group_ids * " OR " (Ruby is great)
#group_ids = [182, 122, 181, 173, 167, 58, 13, 11, 180, 40, 71, 1, 29, 47, 142, 52, 174, 7, 168, 171, 156, 120, 79, 72, 54, 26, 65] #group_ids.map{|id| "group_id:#{id}"}.join(" OR ") #=> "group_id:182 OR group_id:122 OR group_id:181 OR group_id:173 OR group_id:167 OR group_id:58 OR group_id:13 OR group_id:11 OR group_id:180 OR group_id:40 OR group_id:71 OR group_id:1 OR group_id:29 OR group_id:47 OR group_id:142 OR group_id:52 OR group_id:174 OR group_id:7 OR group_id:168 OR group_id:171 OR group_id:156 OR group_id:120 OR group_id:79 OR group_id:72 OR group_id:54 OR group_id:26 OR group_id:65"