Related
I am trying to implement a weighted sampler for a very imbalanced data set. There are 182 different classes. Here is an array of the bin counts per class:
array([69487, 5770, 5753, 138, 4308, 10, 1161, 29, 5611,
350, 7, 183, 218, 4, 3, 3872, 5, 950,
33, 3, 443, 16, 20, 330, 4353, 186, 19,
122, 546, 6, 44, 6, 3561, 2186, 3, 48,
8440, 338, 9, 610, 74, 236, 160, 449, 72,
6, 37, 1729, 2255, 1392, 12, 1, 3426, 513,
44, 3, 28, 12, 9, 27, 5, 75, 15,
3, 21, 549, 7, 25, 871, 240, 128, 28,
253, 62, 55, 12, 8, 57, 16, 99, 6,
5, 150, 7, 110, 8, 2, 1296, 70, 1927,
470, 1, 1, 511, 2, 620, 946, 36, 19,
21, 39, 6, 101, 15, 7, 1, 90, 29,
40, 14, 1, 4, 330, 1099, 1248, 1146, 7414,
934, 156, 80, 755, 3, 6, 6, 9, 21,
70, 219, 3, 3, 15, 15, 12, 69, 21,
15, 3, 101, 9, 9, 11, 6, 32, 6,
32, 4422, 16282, 12408, 2959, 3352, 146, 1329, 1300,
3795, 90, 1109, 120, 48, 23, 9, 1, 6,
2, 1, 11, 5, 27, 3, 7, 1, 3,
70, 1598, 254, 90, 20, 120, 380, 230, 180,
10, 10])
In some classes, instances are as low as 1. I am trying to implement a Weighted random sampler from torch for this dataset. However, as the class imbalance is so large, when I calculate weights using
count_occr = np.bincount(dataset.y)
lbl_weights = 1. / count_occr
weights = np.array(lbl_weights)
weights = torch.from_numpy(weights)
sampler = WeightedRandomSampler(weights.type('torch.DoubleTensor'), len(weights*2))
I get two error messages:
RuntimeWarning: divide by zero encountered in true_divide
and
RuntimeError: invalid multinomial distribution (encountering probability entry = infinity or NaN)
Does anyone have a work around for this ? I was considering multiplying the lbl_weights by some scalar however I am not sure if this is a viable option.
I am getting List of 8-bit unsigned int from a mic source for each sample rate which looks like this
[61, 251, 199, 251, 56, 252, 138, 252, 211, 252, 18, 253, 91, 253, 194, 253, 25, 254, 54, 254, 19, 254, 190, 253, 80, 253, 249, 252, 233, 252, 46, 253, 180, 253, 54, 254, 136, 254, 157, 254, 110, 254, 38, 254, 208, 253, 117, 253, 68, 253, 57, 253, 83, 253, 163, 253, 20, 254, 151, 254, 51, 255, 215, 255, 105, 0, 207, 0, 246, 0, 249, 0, 10, 1, 64, 1, 162, 1, 4, 2, 64, 2, 97, 2, 111, 2, 110, 2, 89, 2, 40, 2, 241, 1, 199, 1, 178, 1, 192, 1, 241, 1, 45, 2, 77, 2, 70, 2, 45, 2, 36, 2, 83, 2, 176, 2, 21, 3, 121, 3, 229, 3, 87, 4, 185, 4, 225, 4, 197, 4, 129, 4, 26, 4, 150, 3, 7, 3, 128, 2, 55, 2, 65, 2, 134, 2, 223, 2, 25, 3, 41, 3, 28, 3, 255, 2, 234, 2, 240, 2, 25, 3, 62, 3, 92, 3, 146, 3, 219, 3, 65, 4, 149, 4, 164, 4, 130, 4, 51, 4, 195, 3, 69, 3, 164, 2, 244, 1, 75, 1, 187, 0, 81, 0, 240, 255, 135, 255, 19, 255, 155, 254, 64, 254, 22, 254, 58, 254, 146, 254, 217, 254, 248, 254, 215, 254, 144, 254, 92, 254, 84, 254, 141, 254, 229, 254, 39, 255, 96, 255, 170, 255, 248, 255, 69, 0, 117, 0, 128, 0, 137, 0, 131, 0,
so how can I convert this into signed decimal value or someone can guide me to the right path
That depends on what the bytes mean.
Looking at the bytes, every other byte is either very low or very high. That suggests to me that the bytes are really little-endian signed 16-bit values.
In that case, you just need to view them as such. If we assume that the platform is little-endian (most are), you can just do:
List<int> list = ...;
Uint8List bytes = Uint8List.fromList(list); //
Int16List words = Int16List.sublistView(bytes);
Then the words list contains signed 16-bit numbers.
(If the list is already a Uint8List, you can skip the first step.)
If that's not what the bytes mean, you'll have to figure out what they do mean.
Dart int type provide a method to convert from signed to unsigned and from unsigned to signed.
For example:
int a = 16;
int b = 239;
print(a.toSigned(5).toString()); // Print -16
print(b.toSigned(5).toString()); // Print 15
the toSigned method parameter indicate the bit order of the sign bit.
You can get more information here: https://api.flutter.dev/flutter/dart-core/int/toSigned.html
A toUnsigned method exixts too: https://api.flutter.dev/flutter/dart-core/int/toUnsigned.html
In the README.md of darknet repo https://github.com/AlexeyAB/darknet we have this sentence about anchor boxes:
But you should change indexes of anchors masks= for each [yolo]-layer, so for YOLOv4 the 1st-[yolo]-layer has anchors smaller than 30x30, 2nd smaller than 60x60, 3rd remaining.
It looks like the default anchor boxes for yolov4-sam-mish.cfg are
12, 16, 19, 36, 40, 28, 36, 75, 76, 55, 72, 146, 142, 110, 192, 243, 459, 401
and the first yolo layer has config:
mask = 0,1,2
Do I understand correctly that this will use those anchors?
(12, 16), (19, 36), (40, 28)
If yes it seems to contradict with the statement or do I understand it incorrectly.
I'm asking because for my dataset and my image sizes (256, 96) I got those anchors from calc_anchors in darknet
15, 56, 22, 52, 28, 48, 23, 62, 26, 59, 39, 43, 31, 57, 29, 66, 37, 64
and trying to figure out how should I set the masks.
Looks good to me.
12, 16,
19, 36,
40, 28,
36, 75,
76, 55,
72, 146,
142, 110,
192, 243,
459, 401
You may leave the masks as are. She current config you show will yield higher MaP; supporting documentation here:
https://github.com/WongKinYiu/PartialResidualNetworks/issues/2
Building a pem file in Elixir requires several steps, including building an entity. In OTP 17, the following works:
{public, private} = :crypto.generate_key(:ecdh, :secp256k1)
ec_entity = {:ECPrivateKey,
1,
:binary.bin_to_list(private),
{:namedCurve, {1, 3, 132, 0, 10}},
{0, public}}
der_encoded = :public_key.der_encode(:ECPrivateKey, ec_entity)
pem = public_key.pem_encode([{:ECPrivateKey, der_encoded, :not_encrypted}])
But using OTP 18, the following error occurs:
{public, private} = :crypto.generate_key(:ecdh, :secp256k1)
ec_entity = {:ECPrivateKey,
1,
:binary.bin_to_list(private),
{:namedCurve, {1, 3, 132, 0, 10}},
{0, public}}
der_encoded = :public_key.der_encode(:ECPrivateKey, ec_entity)
** (MatchError) no match of right hand side value: {:error, {:asn1, :badarg}}
public_key.erl:253: :public_key.der_encode/2
What is the source of this error?
The source of the error is a change in the way that the public_key entity is constructed between OTP 17 and OTP 18. If we reverse the process, starting with a pem file, we can see the difference.
OTP 17:
iex(6)> pem = "-----BEGIN EC PRIVATE KEY-----\nMHQCAQEEIJniJF4vtTqE4wS5AkhmMZsHIbil0l3XfRButkw5IJYFoAcGBSuBBAAK\noUQDQgAEtxm+jijBB0JxZTceHnCHE0HpMXJp1ScVUZ5McvDUVsS/Dek8IdAsMOPz\nnnVALflZzXtH/wU9p2LrFdJeuXwL8g==\n-----END EC PRIVATE KEY-----\n\n"
"-----BEGIN EC PRIVATE KEY-----\nMHQCAQEEIJniJF4vtTqE4wS5AkhmMZsHIbil0l3XfRButkw5IJYFoAcGBSuBBAAK\noUQDQgAEtxm+jijBB0JxZTceHnCHE0HpMXJp1ScVUZ5McvDUVsS/Dek8IdAsMOPz\nnnVALflZzXtH/wU9p2LrFdJeuXwL8g==\n-----END EC PRIVATE KEY-----\n\n"
iex(7)> [{type, decoded, _}] = :public_key.pem_decode(pem)
[{:ECPrivateKey,
<<48, 116, 2, 1, 1, 4, 32, 153, 226, 36, 94, 47, 181, 58, 132, 227, 4, 185, 2, 72, 102, 49, 155, 7, 33, 184, 165, 210, 93, 215, 125, 16, 110, 182, 76, 57, 32, 150, 5, 160, 7, 6, 5, 43, 129, 4, 0, 10, ...>>,
:not_encrypted}]
iex(8)> :public_key.der_decode(type, decoded)
{:ECPrivateKey, 1,
[153, 226, 36, 94, 47, 181, 58, 132, 227, 4, 185, 2, 72, 102, 49, 155, 7, 33,
184, 165, 210, 93, 215, 125, 16, 110, 182, 76, 57, 32, 150, 5],
{:namedCurve, {1, 3, 132, 0, 10}},
{0,
<<4, 183, 25, 190, 142, 40, 193, 7, 66, 113, 101, 55, 30, 30, 112, 135, 19, 65, 233, 49, 114, 105, 213, 39, 21, 81, 158, 76, 114, 240, 212, 86, 196, 191, 13, 233, 60, 33, 208, 44, 48, 227, 243, 158, 117, ...>>}}
OTP 18:
iex(5)> [{type, decoded, _}] = :public_key.pem_decode(pem)
[{:ECPrivateKey,
<<48, 116, 2, 1, 1, 4, 32, 153, 226, 36, 94, 47, 181, 58, 132, 227, 4, 185, 2, 72, 102, 49, 155, 7, 33, 184, 165, 210, 93, 215, 125, 16, 110, 182, 76, 57, 32, 150, 5, 160, 7, 6, 5, 43, 129, 4, 0, 10, ...>>,
:not_encrypted}]
iex(6)> entity = :public_key.der_decode(type, decoded)
{:ECPrivateKey, 1,
<<153, 226, 36, 94, 47, 181, 58, 132, 227, 4, 185, 2, 72, 102, 49, 155, 7, 33, 184, 165, 210, 93, 215, 125, 16, 110, 182, 76, 57, 32, 150, 5>>,
{:namedCurve, {1, 3, 132, 0, 10}},
<<4, 183, 25, 190, 142, 40, 193, 7, 66, 113, 101, 55, 30, 30, 112, 135, 19, 65, 233, 49, 114, 105, 213, 39, 21, 81, 158, 76, 114, 240, 212, 86, 196, 191, 13, 233, 60, 33, 208, 44, 48, 227, 243, 158, 117, 64, ...>>}
The difference is in how the public and private keys are represented.
The signature of an ECPrivateKey Record is:
ECPrivateKey'{ version, privateKey, parameters, publicKey}
In Erlang 18, both values are represented at plain binaries, in 17, the private key is a list and the public key is part of a tuple, {0, binary}.
So in order to build the pem file correctly, the entity representation has to change.
{public, private} = :crypto.generate_key(:ecdh, :secp256k1)
entity = {:ECPrivateKey,
1,
private,
{:namedCurve, {1, 3, 132, 0, 10}},
public}
Using the new representation of the record will solve the problem.
I didn't really check why your version works on some versions, but I've got some code that works on all these erlang versions: 19.0, 18.2.1, 18.1, 18.0, 17.5, R16B03 (running on travis).
-include_lib("public_key/include/public_key.hrl").
genPEMKey() ->
CurveId = secp256k1,
{PubKey, PrivKey} = crypto:generate_key(ecdh, CurveId),
Key = #'ECPrivateKey'{version = 1,
privateKey = PrivKey,
parameters = {
namedCurve,
pubkey_cert_records:namedCurves(CurveId)},
publicKey = PubKey},
DERKey = public_key:der_encode('ECPrivateKey', Key),
public_key:pem_encode([{'ECPrivateKey', DERKey, not_encrypted}]).
This piece of code was based on the examples found in the OTP codebase:
https://github.com/erlang/otp/blob/master/lib/public_key/test/erl_make_certs.erl#L407
I wrote a simple module in the repl:
iex(26)> defmodule Adder do
...(26)> def add(a \\ 0, b \\ 0) do
...(26)> a+b
...(26)> end
...(26)> end
{:module, Adder,
<<70, 79, 82, 49, 0, 0, 4, 224, 66, 69, 65, 77, 65, 116, 111, 109, 0, 0, 0, 106, 0, 0, 0, 12, 12, 69, 108, 105, 120, 105, 114, 46, 65, 100, 100, 101, 114, 8, 95, 95, 105, 110, 102, 111, 95, 95, 4, 100, 111, 99, ...>>,
{:add, 2}}
What is the binary printed out after the module definition?
That's the BEAM bytecode output as a raw binary. It's used by the compiler to persist the compiled module to a Elixir.Adder.beam file when compiling via mix. Modules defined in iex aren't ever persisted to disk, but you can think of it as confirmation that your module was successfully compiled in memory.