Related
I'm trying to train an SVC classifier for image data. Yet, when I run this code:
classifier = svm.SVC(gamma=0.001)
classifier.fit(train_set, train_set_labels)
I get this error:
ValueError: setting an array element with a sequence.
I produced the images into an array with Matplotlib: plt.imread(image).
The error seems like it's not in an array, yet when I check the types of the data and the labels they're both lists (I manually add to a list for the labels data):
print(type(train_set))
print(type(train_set_labels))
<class 'list'>
<class 'list'>
If I do a plt.imshow(items[0]) then the image shows correctly in the output.
I also called train_test_split from scikit-learn:
train_set, test_set = train_test_split(items, test_size=0.2, random_state=42)
Example input:
train_set[0]
array([[[212, 134, 34],
[221, 140, 48],
[240, 154, 71],
...,
[245, 182, 51],
[235, 175, 43],
[242, 182, 50]],
[[230, 152, 51],
[222, 139, 47],
[236, 147, 65],
...,
[246, 184, 49],
[238, 179, 43],
[245, 186, 50]],
[[229, 150, 47],
[205, 122, 28],
[220, 129, 46],
...,
[232, 171, 28],
[237, 179, 35],
[244, 188, 43]],
...,
[[115, 112, 103],
[112, 109, 102],
[ 80, 77, 72],
...,
[ 34, 25, 28],
[ 55, 46, 49],
[ 80, 71, 74]],
[[ 59, 56, 47],
[ 66, 63, 56],
[ 48, 45, 40],
...,
[ 32, 23, 26],
[ 56, 47, 50],
[ 82, 73, 76]],
[[ 29, 26, 17],
[ 41, 38, 31],
[ 32, 29, 24],
...,
[ 56, 47, 50],
[ 59, 50, 53],
[ 84, 75, 78]]], dtype=uint8)
Example label:
train_set_labels[0]
'Picasso'
I'm not sure what step I'm missing to get the data in the form that the classifier needs in order to train it. Can anyone see what may be needed?
The error message you are receiving:
ValueError: setting an array element with a sequence,
normally results when you are trying to put a list somewhere that a single value is required. This would suggest to me that your train_set is made up of a list of multidimensional elements, although you do state that your inputs are lists. Would you be able to post an example of your inputs and labels?
UPDATE
Yes, it's as I thought. The first element of your training data, train_set[0], corresponds to a long list (I can't tell how long), each element of which consists of a list of 3 elements. You are therefore calling the classifier on a list of lists of lists, when the classifier requires a list of lists (m rows corresponding to the number of training examples with each row made up of a list of n features). What else is in your train_set array? Is the full data set in train_set[0]? If so, you would need to create a new array with each element corresponding to each of the subelements of train_set[0], and then I believe your code should run, although I am not too familiar with that classifier. Alternatively you could try running the classifier with train_set[0].
UPDATE 2
I don't have experience with scikit-learn.svc so I wouldn't be able to tell you what the best way of preprocessing the data in order for it to be acceptable to the algorithm, but one method would be to do as I said previously and for each element of train_set, which is composed of lists of lists, would be to recurse through and place all the elements of sublist into the list above. For example
new_train_set = []
for i in range(len(train_set)):
for j in range(len(train_set[i]):
new_train_set.append([train_set[i,j])
I would then train with new_train_set and the training labels.
Rails 3.2.18
ruby 1.9.3
redis
checking caching in development environment.
development.rb:
S2yd::Application.configure do
# Settings specified here will take precedence over those in config/application.rb
# In the development environment your application's code is reloaded on
# every request. This slows down response time but is perfect for development
# since you don't have to restart the webserver when you make code changes.
config.cache_classes = false
#config.assets.enabled = true
# Log error messages when you accidentally call methods on nil.
config.whiny_nils = true
# Show full error reports and disable caching
config.consider_all_requests_local = true
config.action_controller.perform_caching = false
# Don't care if the mailer can't send
config.action_mailer.raise_delivery_errors = false
# Print deprecation notices to the Rails logger
config.active_support.deprecation = :log
# Only use best-standards-support built into browsers
config.action_dispatch.best_standards_support = :builtin
config.cache_store = :redis_store, 'redis://localhost:6379/0/cache', { expires_in: 90.minutes, namespace: 'production' }
controller
#restaurants = Rails.cache.fetch("api/restaurants/#{#thirdparty.name}/1", expires_in: 90.minutes) do
Restaurant
.joins(:thirdparties, :franchise)
.includes(:menu_sections, :menus)
.where("`thirdparties`.`id` = ? and `restaurants`.`active` = ? and lower(`restaurants`.`name`) <> ?", #thirdparty.id, true, 'drinks')
end
log
Processing by Api::V1Controller#restaurants as JSON
Parameters: {"token"=>"eatstreet"}
(0.5ms) SELECT COUNT(*) FROM `domains`
Domain Load (0.6ms) SELECT `domains`.* FROM `domains`
Domain Load (0.5ms) SELECT `domains`.* FROM `domains` LIMIT 1
Domain: 127.0.0.1 -> S2YD
User Load (0.6ms) SELECT `users`.* FROM `users` WHERE `users`.`id` = 93626 LIMIT 1
Thirdparty Load (0.5ms) SELECT `thirdparties`.* FROM `thirdparties` WHERE `thirdparties`.`access_token` = 'eatstreet' LIMIT 1
Thirdparty: eatstreet
Restaurant Load (21.7ms) SELECT `restaurants`.* FROM `restaurants` INNER JOIN `restaurants_thirdparties` ON `restaurants_thirdparties`.`restaurant_id` = `restaurants`.`id` INNER JOIN `thirdparties` ON `thirdparties`.`id` = `restaurants_thirdparties`.`thirdparty_id` INNER JOIN `franchises` ON `franchises`.`id` = `restaurants`.`franchise_id` WHERE (`thirdparties`.`id` = 3 and `restaurants`.`active` = 1 and lower(`restaurants`.`name`) <> 'drinks')
Menu Load (10.2ms) SELECT `menus`.* FROM `menus` WHERE `menus`.`restaurant_id` IN (4, 6, 7, 8, 10, 12, 14, 286, 287, 288, 322, 352, 353, 355, 394, 467, 480, 503, 523, 552, 553, 555, 615, 641, 650, 744, 785, 790, 793, 797, 799, 800, 801, 802, 807, 834, 835, 863, 865, 882, 954, 1012, 1014, 1110, 1111, 1114, 1128, 1139, 1188, 1191, 1196, 1198, 1202, 1206, 1225, 1297, 1325, 1358, 1379, 1387, 1404, 1436, 1509, 1510, 1519, 18, 39, 46, 49, 62, 66, 67, 86, 112, 129, 199, 203, 205, 209, 228, 296, 305, 306, 365, 368, 381, 382, 384, 390, 395, 496, 501, 510, 561, 565, 584, 607, 694, 767, 825, 839, 857, 926, 946, 947, 970, 996, 1000, 1028, 1040, 1055, 1071, 1094, 1109, 1160, 1216, 1244, 1339, 1355, 1356, 1364, 1375, 1418, 1438, 1448, 1508, 1518, 1527, 1564, 131, 132, 134, 135, 139, 142, 218, 249, 308, 309, 310, 324, 339, 350, 351, 361, 363, 367, 369, 370, 392, 396, 400, 490, 581, 585, 597, 836, 849, 993, 1118, 1119, 1122, 1140, 1173, 1218, 1226, 1245, 1246, 1247, 1248, 1249, 1254, 1255, 1259, 1260, 1262, 1266, 1272, 1301, 1308, 1316, 1318, 1335, 1351, 1405, 1408, 1451, 1513, 146, 149, 152, 155, 160, 161, 163, 165, 167, 190, 193, 194, 283, 291, 294, 334, 345, 346, 347, 360, 388, 426, 427, 519, 536, 577, 588, 606, 712, 798, 921, 1043, 1155, 1238, 1258, 1287, 1376, 1381, 1385, 1386, 1413, 1417, 1430, 1433, 1440, 1441, 1479, 1480, 1481, 1482, 1558, 255, 256, 259, 276, 446, 453, 456, 458, 460, 461, 462, 464, 465, 466, 468, 470, 471, 474, 491, 494, 514, 558, 626, 683, 697, 730, 748, 780, 786, 842, 858, 1015, 1161, 1197, 1357, 1371, 1373, 1444, 1453, 374, 377, 378, 401, 498, 499, 528, 564, 573, 636, 809, 909, 933, 944, 407, 409, 417, 455, 476, 479, 517, 560, 591, 604, 709, 754, 771, 833, 1072, 1073, 1074, 1099, 1185, 419, 420, 436, 437, 440, 441, 492, 507, 508, 605, 634, 648, 651, 655, 710, 729, 824, 860, 1181, 1207, 1227, 1239, 1253, 1269, 1270, 1271, 1286, 1310, 1311, 1312, 1313, 1321, 1322, 1343, 1344, 1345, 1346, 1382, 1383, 1416, 488, 543, 568, 571, 598, 622, 625, 724, 1022, 1235, 1288, 1298, 1299, 1302, 1320, 1323, 1334, 1349, 1412, 1429, 1439, 1449, 1450, 1454, 1486, 1515, 1516, 1517, 1520, 1523, 529, 537, 546, 590, 608, 609, 610, 624, 680, 681, 685, 716, 831, 1150, 1151, 1153, 1178, 1186, 1363, 1512, 1514, 638, 695, 715, 717, 718, 719, 753, 781, 837, 845, 848, 850, 960, 1027, 1129, 1233, 1250, 1303, 1367, 1423, 1434, 1442, 1468, 629, 631, 656, 658, 659, 665, 669, 741, 760, 769, 778, 908, 910, 915, 916, 917, 927, 937, 938, 941, 942, 943, 955, 957, 959, 972, 973, 974, 975, 976, 977, 979, 980, 992, 994, 999, 1005, 1023, 1134, 1157, 1174, 1208, 1210, 1224, 1229, 1242, 1304, 1314, 1315, 1378, 1415, 1424, 1435, 1445, 1492, 675, 1041, 1042, 1045, 1046, 1060, 1061, 1064, 1067, 1069, 1080, 1081, 1091, 1101, 1102, 1103, 1123, 1156, 1166, 1222, 1230, 1353, 1365, 1366, 1403, 1410, 1414, 1431, 1443, 1500, 1521, 722, 761, 764, 766, 768, 770, 805, 821, 822, 859, 928, 929, 930, 931, 932, 1059, 1106, 1115, 1234, 1352, 1447, 728, 731, 732, 734, 737, 738, 750, 777, 782, 794, 795, 818, 829, 841, 1029, 1063, 1068, 1105, 1182, 1183, 1256, 1275, 1278, 1348, 1372, 1406, 1566, 854, 861, 862, 867, 874, 880, 881, 890, 893, 894, 895, 905, 936, 956, 1006, 1011, 1013, 1024, 1026, 1030, 1033, 1038, 1044, 1116, 1117, 1121, 1165, 1192, 1193, 1209, 1309, 876, 877, 878, 879, 899, 902, 907, 913, 1004, 1050, 1078, 1079, 1086, 1107, 1108, 1199, 1243, 1292, 1411, 1457, 1458, 1459, 1460, 1461, 1465, 1466, 1467, 1470, 1472, 1474, 1477, 1526, 1538, 1568, 1569, 1570, 1572, 1573, 1574, 1576, 1577, 1578, 1582, 1583, 1584)
MenuSection Load (44.5ms) SELECT `menu_sections`.* FROM `menu_sections` WHERE `menu_sections`.`menu_id` IN (4, 160, 1885, 7, 8, 121, 10, 319, 12, 14, 130, 358, 2225, 2226, 251, 253, 305, 367, 368, 374, 447, 536, 2242, 554, 591, 618, 652, 663, 1812, 1826, 2235, 2257, 745, 1857, 804, 896, 971, 1798, 1000, 2455, 1816, 2418, 1015, 1005, 1010, 1009, 2389, 1019, 1082, 1072, 1903, 1099, 1104, 1113, 2477, 1246, 1346, 1357, 1518, 1519, 2390, 1525, 1539, 2196, 1555, 1643, 1659, 1664, 1668, 1917, 2557, 2559, 1843, 1900, 2292, 2088, 2112, 2116, 2173, 2198, 2180, 2191, 2272, 2406, 2412, 2490, 2408, 2444, 2445, 2446, 2447, 2448, 2449, 171, 172, 175, 2574, 21, 1092, 1889, 1976, 2011, 22, 669, 24, 25, 26, 686, 27, 2558, 351, 352, 31, 1018, 33, 129, 242, 1735, 1762, 1763, 95, 98, 1787, 1792, 1871, 2159, 2161, 2162, 2496, 2497, 103, 478, 1027, 148, 256, 1733, 1770, 2245, 1723, 1883, 505, 2526, 392, 399, 400, 1064, 1066, 1067, 414, 415, 416, 435, 1197, 1200, 2139, 1811, 1872, 1873, 1876, 1878, 1898, 587, 588, 590, 1326, 1979, 1987, 612, 615, 672, 680, 1841, 735, 1925, 865, 2499, 2500, 2502, 2503, 2505, 1058, 1942, 1943, 1190, 1191, 1606, 2293, 1097, 1217, 1241, 1349, 1273, 1275, 1276, 1277, 1278, 1279, 1327, 1649, 1650, 1651, 1652, 1653, 2106, 2107, 2280, 2281, 2309, 2310, 1387, 1395, 1425, 1426, 1436, 1457, 1495, 1523, 1617, 1619, 1793, 1794, 1780, 1781, 1782, 1888, 1918, 2097, 2127, 2108, 2115, 2125, 2128, 2163, 2437, 2239, 2284, 2338, 2433, 2393, 2399, 2410, 2421, 2462, 2584, 35, 265, 1116, 2054, 2113, 1696, 1751, 43, 46, 132, 2114, 2211, 2212, 2214, 2436, 2440, 287, 671, 673, 1542, 1703, 292, 2080, 309, 344, 346, 364, 2228, 366, 381, 384, 2146, 391, 1802, 393, 394, 1710, 443, 448, 2419, 568, 1186, 711, 2602, 2603, 702, 1372, 1075, 1129, 1757, 1321, 1529, 1766, 1884, 1567, 1724, 1725, 2650, 2651, 1796, 1797, 1836, 1837, 1926, 1928, 1931, 1929, 1932, 1934, 2342, 2343, 1980, 1946, 2523, 2010, 1950, 1951, 2219, 2060, 2018, 2035, 2034, 2079, 2575, 2164, 2197, 2531, 2203, 2344, 2405, 50, 395, 2150, 52, 1879, 2008, 55, 58, 62, 63, 65, 67, 159, 167, 398, 1702, 86, 89, 90, 238, 239, 240, 247, 252, 337, 2474, 1803, 2613, 545, 550, 1823, 1824, 1587, 1902, 2611, 480, 481, 630, 637, 2493, 2264, 2612, 701, 734, 1867, 870, 1448, 1325, 1455, 2000, 2279, 1579, 1863, 2546, 1936, 1940, 1974, 2223, 2176, 2190, 2555, 2643, 2179, 2216, 2222, 2178, 2215, 2217, 2675, 2232, 2256, 2261, 2295, 2582, 2321, 2352, 2353, 2396, 2397, 2364, 2371, 2372, 2560, 1127, 198, 2271, 2274, 230, 499, 1984, 1985, 519, 2204, 1989, 527, 528, 529, 1765, 533, 534, 539, 541, 542, 548, 569, 570, 2265, 2266, 2004, 2006, 1817, 766, 854, 863, 864, 894, 1905, 2529, 962, 978, 1079, 1916, 2556, 1359, 1604, 2282, 1821, 2109, 2111, 2154, 2155, 2480, 2313, 2341, 2348, 423, 425, 2233, 2247, 421, 2134, 2135, 2136, 420, 450, 452, 581, 2335, 583, 584, 678, 799, 1400, 683, 687, 708, 1418, 771, 822, 1033, 1175, 2165, 1239, 1259, 1265, 458, 2573, 460, 472, 518, 565, 551, 1865, 2229, 553, 613, 676, 715, 2206, 2498, 856, 2409, 872, 918, 953, 954, 958, 959, 960, 963, 2562, 2563, 2564, 2565, 2566, 1070, 1458, 1459, 1755, 1501, 1736, 1636, 470, 1911, 490, 1071, 2607, 1963, 1964, 494, 506, 1324, 2463, 2600, 2601, 600, 1743, 751, 1679, 1691, 800, 1795, 2147, 2646, 814, 1699, 817, 818, 868, 1632, 1052, 1103, 1778, 1640, 1708, 1838, 1868, 1870, 1933, 1956, 1960, 2609, 1959, 2608, 2007, 2019, 2028, 2645, 2020, 2610, 2021, 2022, 2059, 2057, 2102, 2103, 2089, 2090, 2181, 2182, 2236, 1734, 2071, 2101, 2083, 1832, 1464, 1808, 1927, 724, 760, 770, 782, 885, 2407, 1969, 1970, 1971, 1861, 2156, 2031, 2030, 2032, 2033, 2046, 2055, 2056, 2048, 2063, 2314, 2218, 2385, 2294, 2328, 2339, 2350, 2382, 2383, 2359, 2411, 2414, 2415, 2413, 2452, 2416, 2438, 2430, 632, 653, 2401, 655, 679, 2244, 2604, 2539, 2540, 750, 1534, 1839, 1840, 769, 883, 2606, 943, 2605, 866, 874, 1068, 1086, 1576, 1575, 2395, 1578, 1625, 1638, 1827, 2124, 2402, 2441, 2521, 907, 903, 873, 2188, 875, 2289, 880, 1470, 2423, 2442, 891, 2479, 2519, 916, 966, 2095, 2595, 1083, 1968, 1685, 2243, 1088, 2476, 1263, 1272, 1383, 2495, 1541, 1784, 1999, 2596, 1935, 2494, 1996, 2543, 2213, 2520, 2250, 2262, 2434, 2327, 2422, 2384, 798, 2062, 2210, 2532, 831, 823, 908, 829, 1845, 834, 1858, 895, 1666, 932, 933, 945, 946, 969, 1767, 1901, 1586, 1834, 2224, 1180, 1288, 1163, 1164, 1166, 1165, 1201, 1223, 1222, 1225, 1227, 1228, 1371, 1247, 1690, 1552, 1257, 1282, 1375, 1378, 2142, 2143, 1304, 2398, 1290, 1362, 1629, 1631, 1941, 1785, 1789, 1801, 2185, 1286, 1295, 1420, 1421, 1292, 1294, 2337, 1340, 1726, 2283, 1374, 1558, 1623, 1682, 1639, 1709, 1768, 1769, 1831, 1848, 1904, 1998, 2220, 2027, 2029, 2172, 2199, 2227, 2249, 2269, 2325, 2332, 2367, 1403, 1396, 1507, 1509, 1510, 1397, 1566, 2622, 2623, 1404, 1405, 1428, 2168, 1438, 2486, 1443, 2567, 2590, 1453, 1472, 1481, 2588, 2589, 1475, 1496, 1503, 2585, 1505, 2591, 1506, 1532, 1591, 2315, 2316, 2317, 2318, 2319, 2320, 1605, 1819, 1846, 2096, 2133, 2131, 2192, 2202, 2221, 2268, 2312, 2365, 2400, 2417, 2427, 2428, 884, 935, 941, 1687, 1850, 944, 2465, 947, 1017, 1353, 2470, 1049, 1047, 1048, 1105, 1777, 1237, 2580, 2581, 1236, 1230, 1234, 2597, 2598, 1231, 1232, 2578, 1882, 1521, 1783, 1528, 1860, 2149, 2336, 920, 2471, 1669, 2002, 2267, 2647, 928, 1497, 2511, 2512, 2513, 2514, 2515, 997, 936, 919, 956, 1323, 1986, 2148, 2472, 1694, 2484, 996, 998, 1040, 1063, 1078, 1386, 1389, 1437, 1444, 2370, 1513, 1514, 1633, 1634, 1938, 1952, 1966, 2545, 2158, 2193, 2644, 1145, 1188, 2458, 1154, 1572, 1158, 1159, 1658, 2259, 2530, 1148, 1176, 1150, 1160, 1169, 2340, 1173, 1181, 1194, 1195, 1172, 1206, 2522, 1177, 1179, 1252, 2119, 1262, 1351, 2349, 2594, 1360, 1367, 1385, 1381, 2288, 2291, 2329, 2330, 1410, 1467, 1540, 1527, 1551, 1624, 2461, 1663, 1660, 2049, 2052, 2024, 2025, 1107, 1149, 1132, 1168, 1114, 1135, 1143, 1967, 1973, 1174, 1184, 1334, 1434, 1465, 1531, 1490, 1491, 2047, 1516, 1515, 1522, 1707, 1912, 2085, 2205, 2376, 2379, 2380, 2387, 2391, 2386, 2392, 2355, 2394, 2425, 2381, 2354, 2360, 2361, 2374, 2454, 2488, 2616, 2627, 2619, 2624, 2625, 2639, 2626, 2629, 2630, 2631, 2634, 2635, 2636) ORDER BY `index` ASC
Completed 200 OK in 3321.9ms (Views: 3235.1ms | ActiveRecord: 79.0ms)
Seems Rails.cache.fetch doesn't work and still touches the database.
I think you should change the config.
config.action_controller.perform_caching = false
Try to change false to true.
I have a problem where I receive a lot of histograms from some database images. Those histograms are represented as vectors (0...255), and I have to identify and work with the bimodal histograms.
Is there a formula to automatically identify which histograms are bimodal and which aren't? Since they are numeric vectors, I could use a programming language (Java/C#) to work with it.
Is there a criterion on literature to identify bimodal histograms by software?
Here are 3 examples of histograms and format inputs I'm working with. Each histogram is a vector with 256 (0...255) positions.
Histogram 1
8029, 41, 82, 177, 135, 255, 315, 591, 949, 456, 499, 688, 446, 733, 712, 1595, 2633, 3945, 6134, 9755, 9236, 11911, 11888, 9450, 13119, 8819, 5991, 4399, 6745, 2017, 3747, 1777, 2946, 1623, 2151, 454, 3015, 3176, 2211, 1080, 391, 580, 750, 473, 10424, 334, 559, 621, 340, 2794, 1094, 5274, 2822, 204, 389, 728, 268, 15, 1060, 58, 113, 2728, 52, 3166, 11, 103, 522, 107, 351, 97, 66, 565, 315, 444, 3305, 245, 647, 306, 147, 112, 103, 672, 69, 317, 61, 224, 71, 52, 479, 62, 106, 166, 215, 132, 137, 321, 998, 427, 846, 787, 542, 1054, 1429, 615, 697, 580, 642, 768, 1244, 462, 4107, 1701, 2394, 4954, 4869, 1841, 1807, 1032, 3075, 331, 488, 627, 1281, 233, 1010, 1178, 727, 830, 1619, 728, 1428, 1849, 4826, 351, 745, 320, 888, 335, 741, 1151, 734, 689, 2143, 1130, 2482, 3609, 4779, 5678, 4186, 2654, 1668, 1290, 702, 1093, 476, 438, 445, 271, 98, 368, 226, 90, 75, 26, 33, 62, 16, 824, 21, 37, 34, 24, 54, 42, 101, 112, 18, 24, 17, 15, 3, 50, 7, 6, 54, 3, 58, 9, 10, 66, 12, 11, 10, 6, 25, 11, 7, 172, 13, 18, 21, 9, 8, 9, 42, 16, 15, 6, 12, 17, 7, 591, 6, 7, 14, 24, 7, 7, 19, 87, 18, 8, 9, 9, 35, 55, 4, 17, 10, 18, 22, 46, 8, 852, 15, 14, 12, 11, 9, 3, 50, 163, 12, 4, 18, 129, 6, 35, 47, 14, 18, 150, 21, 46, 24, 0
Histogram 2
8082, 4857, 1494, 2530, 1604, 1636, 1651, 1681, 1630, 1667, 1636, 1649, 1934, 1775, 1701, 1691, 1478, 1649, 1449, 1449, 1503, 1475, 1497, 1398, 1509, 1747, 1301, 1539, 1575, 1496, 1754, 1432, 1759, 1786, 1679, 1816, 2435, 1174, 1780, 1344, 1749, 2026, 1779, 1742, 1722, 1835, 2306, 1662, 1965, 1885, 2212, 2139, 1930, 2306, 2707, 2289, 2307, 2082, 2360, 2216, 2480, 2243, 2222, 1824, 4555, 1918, 2116, 2275, 2615, 2240, 2703, 2481, 2626, 2708, 3008, 2696, 2561, 2906, 3625, 2419, 3137, 2793, 2747, 2861, 2774, 4124, 3155, 3243, 3523, 3432, 3277, 3456, 2984, 2902, 2819, 2778, 3158, 2997, 2591, 2717, 2553, 2464, 3657, 2296, 2352, 2046, 2124, 1965, 2014, 2096, 1664, 1373, 1607, 1322, 1272, 1113, 1156, 1055, 924, 881, 1019, 669, 929, 636, 590, 463, 524, 177, 1267, 378, 409, 413, 415, 435, 385, 379, 267, 413, 266, 282, 499, 194, 360, 199, 337, 92, 986, 183, 160, 230, 124, 213, 188, 334, 164, 159, 130, 143, 135, 331, 25, 118, 114, 98, 74, 301, 92, 119, 94, 72, 192, 38, 64, 100, 138, 30, 98, 65, 226, 23, 46, 78, 78, 61, 55, 234, 26, 36, 95, 31, 49, 214, 25, 34, 58, 37, 101, 20, 41, 34, 150, 16, 50, 25, 53, 18, 30, 67, 27, 36, 42, 23, 60, 12, 21, 36, 12, 45, 21, 58, 53, 18, 51, 16, 25, 9, 24, 15, 18, 30, 33, 20, 19, 12, 23, 16, 14, 21, 14, 10, 20, 13, 12, 9, 6, 9, 7, 10, 7, 2, 0, 0, 0, 0, 0, 2087
Histogram 3
50, 226, 857, 2018, 1810, 1795, 1840, 1929, 1942, 1693, 1699, 1547, 1564, 1556, 1451, 1439, 1448, 1357, 1428, 1419, 1383, 1705, 1670, 1777, 1826, 1865, 1897, 1924, 2003, 1973, 1813, 1801, 1827, 1696, 1717, 1654, 1678, 1705, 1621, 1523, 1494, 1559, 1434, 1370, 1358, 1385, 1348, 1380, 1368, 1367, 1389, 1445, 1514, 1471, 1465, 1461, 1475, 1484, 1390, 1403, 1324, 1339, 1426, 1432, 1487, 1460, 1469, 1460, 1546, 1504, 1425, 1373, 1391, 1391, 1382, 1311, 1368, 1354, 1325, 1323, 1263, 1325, 1363, 1357, 1325, 1322, 1429, 1419, 1412, 1371, 1266, 1179, 1166, 1076, 1100, 1083, 1103, 1053, 1116, 1080, 1071, 1025, 1088, 1060, 1011, 984, 958, 959, 954, 937, 982, 950, 1001, 963, 965, 875, 1010, 954, 990, 894, 959, 972, 963, 1101, 971, 1042, 1064, 1075, 1029, 1088, 1090, 1068, 1073, 1058, 1102, 1105, 1009, 1062, 1005, 1048, 973, 998, 1034, 1013, 961, 1006, 983, 948, 1031, 972, 952, 1013, 954, 964, 970, 881, 887, 967, 941, 928, 994, 1019, 1106, 1056, 1113, 1071, 1158, 1108, 1178, 1071, 1080, 1074, 1050, 1076, 1106, 1048, 973, 1042, 997, 1034, 934, 863, 935, 845, 839, 803, 764, 782, 787, 771, 766, 751, 745, 804, 789, 765, 681, 658, 690, 672, 650, 635, 695, 619, 572, 499, 535, 565, 564, 520, 516, 568, 530, 479, 507, 424, 446, 455, 380, 395, 371, 360, 391, 373, 351, 388, 426, 349, 417, 421, 400, 443, 470, 485, 456, 495, 452, 484, 457, 518, 519, 631, 652, 693, 762, 771, 807, 906, 991, 1138, 1433, 1545, 2467, 4907, 6743, 1921
smooth histogram
this filter out small local min max and noise. Use symmetric smoothing to avoid shifting to one side. I smooth from left then from the right which lower the shifting a lot.
find/count the local max peaks
Count only big enough peaks (by some treshold). If peak count is not 2 then it is not a bimodal histogram unless you have different definition of bimodal like:
noise count too
no matter how many peaks but single gap must be present
It depends on what for the histograms are used
Here is some code in C++ I busted for this:
void histograms(Graphics::TBitmap *bmp,int xs,int ys,int **pyx)
{
// clear buffer
bmp->Canvas->Brush->Color=clBlack;
bmp->Canvas->FillRect(TRect(0,0,xs,ys));
bmp->Canvas->Font->Color=clAqua;
bmp->Canvas->TextOutA( 5,5,"Raw histogram");
bmp->Canvas->TextOutA(285,5,"Smoothed histogram");
int his1[256]={ 8029, 41, 82, 177, 135, 255, 315, 591, 949, 456, 499, 688, 446, 733, 712, 1595, 2633, 3945, 6134, 9755, 9236, 11911, 11888, 9450, 13119, 8819, 5991, 4399, 6745, 2017, 3747, 1777, 2946, 1623, 2151, 454, 3015, 3176, 2211, 1080, 391, 580, 750, 473, 10424, 334, 559, 621, 340, 2794, 1094, 5274, 2822, 204, 389, 728, 268, 15, 1060, 58, 113, 2728, 52, 3166, 11, 103, 522, 107, 351, 97, 66, 565, 315, 444, 3305, 245, 647, 306, 147, 112, 103, 672, 69, 317, 61, 224, 71, 52, 479, 62, 106, 166, 215, 132, 137, 321, 998, 427, 846, 787, 542, 1054, 1429, 615, 697, 580, 642, 768, 1244, 462, 4107, 1701, 2394, 4954, 4869, 1841, 1807, 1032, 3075, 331, 488, 627, 1281, 233, 1010, 1178, 727, 830, 1619, 728, 1428, 1849, 4826, 351, 745, 320, 888, 335, 741, 1151, 734, 689, 2143, 1130, 2482, 3609, 4779, 5678, 4186, 2654, 1668, 1290, 702, 1093, 476, 438, 445, 271, 98, 368, 226, 90, 75, 26, 33, 62, 16, 824, 21, 37, 34, 24, 54, 42, 101, 112, 18, 24, 17, 15, 3, 50, 7, 6, 54, 3, 58, 9, 10, 66, 12, 11, 10, 6, 25, 11, 7, 172, 13, 18, 21, 9, 8, 9, 42, 16, 15, 6, 12, 17, 7, 591, 6, 7, 14, 24, 7, 7, 19, 87, 18, 8, 9, 9, 35, 55, 4, 17, 10, 18, 22, 46, 8, 852, 15, 14, 12, 11, 9, 3, 50, 163, 12, 4, 18, 129, 6, 35, 47, 14, 18, 150, 21, 46, 24, 0 };
int his2[256]={ 8082, 4857, 1494, 2530, 1604, 1636, 1651, 1681, 1630, 1667, 1636, 1649, 1934, 1775, 1701, 1691, 1478, 1649, 1449, 1449, 1503, 1475, 1497, 1398, 1509, 1747, 1301, 1539, 1575, 1496, 1754, 1432, 1759, 1786, 1679, 1816, 2435, 1174, 1780, 1344, 1749, 2026, 1779, 1742, 1722, 1835, 2306, 1662, 1965, 1885, 2212, 2139, 1930, 2306, 2707, 2289, 2307, 2082, 2360, 2216, 2480, 2243, 2222, 1824, 4555, 1918, 2116, 2275, 2615, 2240, 2703, 2481, 2626, 2708, 3008, 2696, 2561, 2906, 3625, 2419, 3137, 2793, 2747, 2861, 2774, 4124, 3155, 3243, 3523, 3432, 3277, 3456, 2984, 2902, 2819, 2778, 3158, 2997, 2591, 2717, 2553, 2464, 3657, 2296, 2352, 2046, 2124, 1965, 2014, 2096, 1664, 1373, 1607, 1322, 1272, 1113, 1156, 1055, 924, 881, 1019, 669, 929, 636, 590, 463, 524, 177, 1267, 378, 409, 413, 415, 435, 385, 379, 267, 413, 266, 282, 499, 194, 360, 199, 337, 92, 986, 183, 160, 230, 124, 213, 188, 334, 164, 159, 130, 143, 135, 331, 25, 118, 114, 98, 74, 301, 92, 119, 94, 72, 192, 38, 64, 100, 138, 30, 98, 65, 226, 23, 46, 78, 78, 61, 55, 234, 26, 36, 95, 31, 49, 214, 25, 34, 58, 37, 101, 20, 41, 34, 150, 16, 50, 25, 53, 18, 30, 67, 27, 36, 42, 23, 60, 12, 21, 36, 12, 45, 21, 58, 53, 18, 51, 16, 25, 9, 24, 15, 18, 30, 33, 20, 19, 12, 23, 16, 14, 21, 14, 10, 20, 13, 12, 9, 6, 9, 7, 10, 7, 2, 0, 0, 0, 0, 0, 2087 };
int his3[256]={ 50, 226, 857, 2018, 1810, 1795, 1840, 1929, 1942, 1693, 1699, 1547, 1564, 1556, 1451, 1439, 1448, 1357, 1428, 1419, 1383, 1705, 1670, 1777, 1826, 1865, 1897, 1924, 2003, 1973, 1813, 1801, 1827, 1696, 1717, 1654, 1678, 1705, 1621, 1523, 1494, 1559, 1434, 1370, 1358, 1385, 1348, 1380, 1368, 1367, 1389, 1445, 1514, 1471, 1465, 1461, 1475, 1484, 1390, 1403, 1324, 1339, 1426, 1432, 1487, 1460, 1469, 1460, 1546, 1504, 1425, 1373, 1391, 1391, 1382, 1311, 1368, 1354, 1325, 1323, 1263, 1325, 1363, 1357, 1325, 1322, 1429, 1419, 1412, 1371, 1266, 1179, 1166, 1076, 1100, 1083, 1103, 1053, 1116, 1080, 1071, 1025, 1088, 1060, 1011, 984, 958, 959, 954, 937, 982, 950, 1001, 963, 965, 875, 1010, 954, 990, 894, 959, 972, 963, 1101, 971, 1042, 1064, 1075, 1029, 1088, 1090, 1068, 1073, 1058, 1102, 1105, 1009, 1062, 1005, 1048, 973, 998, 1034, 1013, 961, 1006, 983, 948, 1031, 972, 952, 1013, 954, 964, 970, 881, 887, 967, 941, 928, 994, 1019, 1106, 1056, 1113, 1071, 1158, 1108, 1178, 1071, 1080, 1074, 1050, 1076, 1106, 1048, 973, 1042, 997, 1034, 934, 863, 935, 845, 839, 803, 764, 782, 787, 771, 766, 751, 745, 804, 789, 765, 681, 658, 690, 672, 650, 635, 695, 619, 572, 499, 535, 565, 564, 520, 516, 568, 530, 479, 507, 424, 446, 455, 380, 395, 371, 360, 391, 373, 351, 388, 426, 349, 417, 421, 400, 443, 470, 485, 456, 495, 452, 484, 457, 518, 519, 631, 652, 693, 762, 771, 807, 906, 991, 1138, 1433, 1545, 2467, 4907, 6743, 1921 };
int *his,tmp[256],a,x0,y0,x,y,h,tr=12,sm=10,peak[256],peaks;
// loop through histograms
for (y0=20,h=0;;h++)
{
x0=5;if (h==0) his=his1;
else if (h==1) his=his2;
else if (h==2) his=his2;
else break;
// rescale his <0,?> to tmp <0-100>
for (y=his[0],x=0;x<256;x++) if (y<his[x]) y=his[x]; // y=max
for ( x=0;x<256;x++) tmp[x]=(his[x]*100)/y;
// draw tmp
for (x=0;x<256;x++) for (pyx[y0+100][x0+x]=0x00004040,y=0;y<tmp[x];y++) pyx[y0+100-y][x0+x]=(40+x)*0x00010101;
x0+=280;
// smooth tmp few times
for (y=0;y<sm;y++)
{
// from both directions to avoid shifting to one side
for (x=0;x<255;x++) tmp[x]=((90*tmp[x])+(10*tmp[x+1]))/100;
for (x=255;x>0;x--) tmp[x]=((90*tmp[x])+(10*tmp[x-1]))/100;
}
// find (count) peaks
for (peaks=0,a=0,y=0,x=0;x<255;x++)
{
if (tmp[x]<tmp[x+1]){ if ((y< 0)&&(a-tmp[x]>tr)){ a=tmp[x]; } y=+1; }
else if (tmp[x]>tmp[x+1]){ if ((y>=0)&&(tmp[x]-a>tr)){ a=tmp[x]; peak[peaks]=x; peaks++; } y=-1; }
}
// draw tmp
for (x=0;x<256;x++) for (pyx[y0+100][x0+x]=0x00004040,y=0;y<tmp[x];y++) pyx[y0+100-y][x0+x]=(40+x)*0x00010101;
// draw peaks
bmp->Canvas->Pen->Color=clAqua;
bmp->Canvas->Brush->Color=clAqua;
for (a=0;a<peaks;a++)
{
x=x0+peak[a];
y=y0+100-tmp[peak[a]];
bmp->Canvas->Ellipse(x-5,y-5,x+5,y+5);
}
// draw cross for not bimodal histograms
if (peaks!=2)
{
bmp->Canvas->Pen->Color=clRed;
bmp->Canvas->MoveTo(x0 ,y0 );
bmp->Canvas->LineTo(x0+256,y0+100);
bmp->Canvas->MoveTo(x0+256,y0 );
bmp->Canvas->LineTo(x0 ,y0+100);
}
y0+=128;
}
}
you can ignore the pyx[y][x] and bmp-> stuff it is just rendering
pyx[y][x] is direct 32bit pixel access of bitmap bmp
and bmp->Canvas is VCL encapsulated Windows GDI interface of bitmap bmp
xs,ys is bitmap resolution
set treshold tr and smooth sm to suite your needs best
If you have too much different types of histogram then you need to apply dynamic tresholding or different approach for peak finding this is how it looks like for your histograms:
Where Histogram 1 is the top one. Hope the code is clear enough if not comment me... if you rescale to power of 2 instead of 100 then you can change the multiplications and divisions to bit shifts to speed this a bit. I choose 100 for more clear selection of tresholds and smoothing coefficients...
I don't think there is a simple and straightforward solution for that.
If you think each peak in your histogram as a cluster, you can try to implement some sort of clustering algorithm that is able to automatically detect the number of peaks in your histogram.
You can start looking at here:
Determining the number of clusters in a data set
Kmeans without knowing the number of clusters?
AUTOMATIC DETERMINATION OF THE NUMBER OF CLUSTERS USING SPECTRAL
ALGORITHMS
I am running multiple named docker containers (200+) on my VM Host.
I have a manager script/code that is supposed to manage the containers from the host.
I would like to know if there is any event-based mechanism to get notified when a container stops/fails. So that I can restart the stopped container.
One solution I could think of is doing a periodic docker inspect and looking at State.Pid or State.Running to confirm the status.
But,instead of periodic polling, it would be better if the manager is notified with pid/name when a container fails so that, the particular container alone can be restarted.
On a general note, are there ways to programmatically monitor the status of a process from a different process that is not the parent ?
Look at docker events - there is an event for container 'die'.
There is also an http interface to get the same information programmatically - see here
You may want to do a web search for 'docker orchestration' - many projects springing up to manage multiple containers in the way you describe.
If you just want to restart the containers why don't you use a restart policy?
docker run --restart=always IMAGE
psutil seems to do what you want
http://pypi.python.org/pypi/psutil
From Python
import psutil
psutil.pids()
[1, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 65, 66, 67, 69, 70, 71, 72, 73, 74, 76, 77, 78, 79, 80, 81, 82, 94, 97, 98, 117, 118, 137, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 200, 201, 210, 211, 213, 214, 229, 230, 411, 416, 419, 526, 527, 542, 543, 544, 545, 555, 569, 625, 709, 714, 756, 781, 782, 796, 862, 863, 864, 869, 914, 944, 945, 948, 954, 996, 1052, 1061, 1064, 1067, 1170, 1174, 1179, 1180, 1183, 1234, 1240, 1241, 1245, 1323, 1328, 1340, 1351, 1354, 1390, 1408, 1457, 1507, 1531, 1631, 1662, 1933, 1972, 1981, 1987, 1989, 1993, 2346, 2348, 2413, 2422, 2429, 2442, 2445, 2449, 2451, 2457, 2461, 2471, 2489, 2490, 2491, 2493, 2497, 2501, 2505, 2509, 2513, 2524, 2546, 2549, 2551, 2554, 2563, 2567, 2572, 2573, 2576, 2578, 2586, 2595, 2598, 2624, 2644, 2655, 2665, 2667, 2687, 2689, 2693, 2699, 2744, 2752, 2785, 2789, 2794, 2798, 2804, 2817, 2820, 2830, 2838, 2856, 2862, 2864, 2886, 2903, 2935, 2972, 2985, 2986, 3138, 3164, 3211, 3368, 3371, 3557, 4125, 4352, 4443, 4444, 4743, 4818, 4819, 4840, 4841, 4844, 4845, 4866, 4876, 6142, 6363, 6366, 6372, 6378, 6385, 6391, 6452, 6518, 6524, 6531, 6555, 6558, 6601]
p = psutil.Process(2862)
p.status()
'sleeping'
I have two Models: AssetItem and Title.
An AssetItem belongs to a Title / Title has many asset_items.
A Title can have a parent Title (thereby creating a hierarchy); this parent/child relationship within Titles is managed by the ancestry gem.
I've got a controller that is fetching a bunch of AssetItems. Those AssetItems should come with their Title, that Title's parent, and that Title's parent and so on. In the end, the controller presents this object graph in JSON. I am trying to find a way to eager-load all of the title hierarchy for a given asset item.
I am doing this in my controller:
def show
asset_items = AssetItem.includes(:title).where(:id => list_of_ids)
eager_load_titles_for(asset_items)
respond_with asset_items, serializer: AssetItemSerializer
end
private
def eager_load_titles_for(asset_items)
title_ids = asset_items.map { |a| a.title.ancestor_ids }.flatten
Title.find(title_ids) # these titles are not really used, this is just to eager load them
end
The idea is, that a collection of asset items are fetched and then the title ids associated with those asset items are simply loaded (but not used), so that when the serializer does something like object.parent.parent.parent.some_attribute, it gets that Title from the cache, not the db.
However, the results in the log do not match my expectations:
10:19:27 web.1 | AssetItem Load (1.3ms) SELECT "asset_items".* FROM "asset_items" WHERE "asset_items"."id" IN (291, 311, 316, 309, 323, 304, 296, 289, 328, 284, 373, 342, 330, 335, 361, 366, 354, 347, 359, 380, 400, 405, 392, 378, 397, 417, 412, 385, 436, 429, 450, 448, 431, 455, 462, 424, 443, 481, 498, 467, 501, 493, 474, 506, 486, 479, 532, 525, 520, 513)
10:19:27 web.1 | Title Load (0.6ms) SELECT "titles".* FROM "titles" WHERE "titles"."id" IN (167, 182, 188, 203, 227, 242, 248, 263, 284, 299, 305, 320, 341, 356, 377, 392, 398, 413, 434, 449, 455, 470, 491, 506, 515, 530, 551, 566, 587, 602, 608, 623, 644, 659, 665, 680, 701, 716, 737, 752, 758, 773, 794, 809, 818, 833, 854, 875, 890, 911)
10:19:27 web.1 | Title Load (0.9ms) SELECT "titles".* FROM "titles" WHERE "titles"."id" IN (186, 187, 246, 247, 261, 262, 240, 241, 282, 283, 225, 226, 201, 202, 180, 181, 297, 298, 165, 166, 432, 433, 339, 340, 303, 304, 318, 319, 396, 397, 411, 412, 375, 376, 354, 355, 390, 391, 453, 454, 513, 514, 528, 529, 489, 490, 447, 448, 504, 505, 564, 565, 549, 550, 468, 469, 621, 622, 600, 601, 663, 664, 657, 658, 606, 607, 678, 679, 699, 700, 585, 586, 642, 643, 756, 757, 807, 808, 714, 715, 816, 817, 792, 793, 735, 736, 831, 832, 771, 772, 750, 751, 909, 910, 888, 889, 873, 874, 852, 853)
10:19:27 web.1 | CACHE (0.0ms) SELECT "asset_items".* FROM "asset_items" WHERE "asset_items"."id" IN (291, 311, 316, 309, 323, 304, 296, 289, 328, 284, 373, 342, 330, 335, 361, 366, 354, 347, 359, 380, 400, 405, 392, 378, 397, 417, 412, 385, 436, 429, 450, 448, 431, 455, 462, 424, 443, 481, 498, 467, 501, 493, 474, 506, 486, 479, 532, 525, 520, 513)
10:19:27 web.1 | CACHE (0.0ms) SELECT "titles".* FROM "titles" WHERE "titles"."id" IN (167, 182, 188, 203, 227, 242, 248, 263, 284, 299, 305, 320, 341, 356, 377, 392, 398, 413, 434, 449, 455, 470, 491, 506, 515, 530, 551, 566, 587, 602, 608, 623, 644, 659, 665, 680, 701, 716, 737, 752, 758, 773, 794, 809, 818, 833, 854, 875, 890, 911)
10:19:27 web.1 | CACHE (0.0ms) SELECT "titles".* FROM "titles" WHERE "titles"."id" IN (186, 187, 246, 247, 261, 262, 240, 241, 282, 283, 225, 226, 201, 202, 180, 181, 297, 298, 165, 166, 432, 433, 339, 340, 303, 304, 318, 319, 396, 397, 411, 412, 375, 376, 354, 355, 390, 391, 453, 454, 513, 514, 528, 529, 489, 490, 447, 448, 504, 505, 564, 565, 549, 550, 468, 469, 621, 622, 600, 601, 663, 664, 657, 658, 606, 607, 678, 679, 699, 700, 585, 586, 642, 643, 756, 757, 807, 808, 714, 715, 816, 817, 792, 793, 735, 736, 831, 832, 771, 772, 750, 751, 909, 910, 888, 889, 873, 874, 852, 853)
10:19:27 web.1 | Title Load (0.4ms) SELECT "titles".* FROM "titles" WHERE "titles"."id" = $1 LIMIT 1 [["id", 186]]
10:19:27 web.1 | Title Load (0.2ms) SELECT "titles".* FROM "titles" WHERE "titles"."id" = $1 LIMIT 1 [["id", 187]]
10:19:27 web.1 | CACHE (0.0ms) SELECT "titles".* FROM "titles" WHERE "titles"."id" = $1 LIMIT 1 [["id", 187]]
10:19:27 web.1 | CACHE (0.0ms) SELECT "titles".* FROM "titles" WHERE "titles"."id" = $1 LIMIT 1 [["id", 187]]
10:19:27 web.1 | CACHE (0.0ms) SELECT "titles".* FROM "titles" WHERE "titles"."id" = $1 LIMIT 1 [["id", 187]]
10:19:27 web.1 | CACHE (0.0ms) SELECT "titles".* FROM "titles" WHERE "titles"."id" = $1 LIMIT 1 [["id", 187]]
10:19:27 web.1 | CACHE (0.0ms) SELECT "titles".* FROM "titles" WHERE "titles"."id" = $1 LIMIT 1 [["id", 187]]
As you can see, initially title ID 186, and 187 are loaded in the where clause here:
10:19:27 web.1 | Title Load (0.9ms) SELECT "titles".* FROM "titles" WHERE "titles"."id" IN (186, 187, 246, 247, 261, 262, 240, 241, 282, 283, 225, 226, 201, 202, 180, 181, 297, 298, 165, 166, 432, 433, 339, 340, 303, 304, 318, 319, 396, 397, 411, 412, 375, 376, 354, 355, 390, 391, 453, 454, 513, 514, 528, 529, 489, 490, 447, 448, 504, 505, 564, 565, 549, 550, 468, 469, 621, 622, 600, 601, 663, 664, 657, 658, 606, 607, 678, 679, 699, 700, 585, 586, 642, 643, 756, 757, 807, 808, 714, 715, 816, 817, 792, 793, 735, 736, 831, 832, 771, 772, 750, 751, 909, 910, 888, 889, 873, 874, 852, 853)
and then below that, we see again:
10:19:27 web.1 | Title Load (0.4ms) SELECT "titles".* FROM "titles" WHERE "titles"."id" = $1 LIMIT 1 [["id", 186]]
10:19:27 web.1 | Title Load (0.2ms) SELECT "titles".* FROM "titles" WHERE "titles"."id" = $1 LIMIT 1 [["id", 187]]
And that is why I am wondering what I am doing wrong. Any help would be greatly appreciated.