In my application I need to convert integer to some term; for example:
1 → :red
2 → :green
3 → :blue
The table is static, it is known during compilation time and its indices range from <1, n>. There is around 60 of them.
In which way should the table be represented, so the lookup is the fastest? Dict, HashDict, tuple (with kernel.elem()), ets, function with pattern matching...?
As Julius Beckmann suggested in this case functions with pattern matching should be sufficient as they are according to my measurements over 5 times faster than accessing a map.
Below you can find an implementation of what you are looking for (benchmark code included at the bottom):
defmodule TermLookUpByInteger do
#term_by_integer %{
1 => :aa, 11 => :ba, 21 => :ca, 31 => :da, 41 => :ea, 51 => :fa, 61 => :ga,
2 => :ab, 12 => :bb, 22 => :cb, 32 => :db, 42 => :eb, 52 => :fb, 62 => :gb,
3 => :ac, 13 => :bc, 23 => :cc, 33 => :dc, 43 => :ec, 53 => :fc, 63 => :gc,
4 => :ad, 14 => :bd, 24 => :cd, 34 => :dd, 44 => :ed, 54 => :fd, 64 => :gd,
5 => :ae, 15 => :be, 25 => :ce, 35 => :de, 45 => :ee, 55 => :fe, 65 => :ge,
6 => :af, 16 => :bf, 26 => :cf, 36 => :df, 46 => :ef, 56 => :ff, 66 => :gf,
7 => :ag, 17 => :bg, 27 => :cg, 37 => :dg, 47 => :eg, 57 => :fg, 67 => :gg,
8 => :ah, 18 => :bh, 28 => :ch, 38 => :dh, 48 => :eh, 58 => :fh, 68 => :gh,
9 => :ai, 19 => :bi, 29 => :ci, 39 => :di, 49 => :ei, 59 => :fi, 69 => :gi,
0 => :aj, 10 => :bj, 20 => :cj, 30 => :dj, 40 => :ej, 50 => :fj, 60 => :gj,
}
#doc """
iex> TermLookUpByInteger.lookup_pmf(2)
:ab
"""
def lookup_pmf(int), do: do_lookup(int)
for {int, term} <- #term_by_integer do
defp do_lookup(unquote(int)), do: unquote(term)
end
#doc """
iex> TermLookUpByInteger.lookup_m(3)
:ac
"""
def lookup_m(int), do: #term_by_integer[int]
end
# Benchmark:
n = 1_000_000
range = 1..(n)
measure = fn fun -> :timer.tc(fn -> for _ <- range, do: fun.() end) end
{time_pmf, _result} = measure.(fn -> TermLookUpByInteger.lookup_pmf(:random.uniform(60)) end)
{time_m, _result} = measure.(fn -> TermLookUpByInteger.lookup_m(:random.uniform(60)) end)
IO.puts " Sample size: #{n}"
IO.puts "Pattern matching functions lookup: #{time_pmf/1000} ms"
IO.puts " Map lookup: #{time_m/1000} ms"
IO.puts " Absolute Difference: #{(time_pmf-time_m)/1000} ms"
IO.puts " Relative Difference: #{round((time_pmf-time_m)/time_m*100)}%"
IO.puts " Faster: x #{Float.round(time_m/time_pmf, 2)} times"
Results:
Sample size: 1000000
Pattern matching functions lookup: 447.6 ms
Map lookup: 2423.517 ms
Absolute Difference: -1975.917 ms
Relative Difference: -82%
Faster: x 5.41 times
I hope that will be useful.
If the map is completely static and will not change, you can use generated pattern matching. This will be the fastest way to integrate that lookup in your application.
Some example code, reading these mappings from a external file: https://github.com/h4cc/slugger/blob/master/lib/slugger.ex#L69-72
Instead of using a external file, your source map data can be held in a #attribute.
Even if new mappings will be needed on runtime, these could be handled with a catchall-pattern match doing a lookup in a HashDict.
If you rely on fast access from many processes, then mochiglobal may be the answer. It's tricky constant pool, which keeps keys and values as functions in module. Every time you put/2 something, module is recompiled (it's slow but doesn't matter in your case). With this approach value is not copied into process heap as it is function call. Here is better explanation.
Related
I would like (without using a gem) to create a level and experience point system for my users.
When creating his account, he starts level 0 with 0 experience points. To reach level 1 he will need 10 additional points, for a total of 10 points. For level 3 he will need 30 additional points, for a total of 60 experience points. Each additional level requires 10 more experience points. (see example below).
Level | Total XP | XP for next level|
----------------------------------------|
0 | 0 | 10 |
1 | 10 | 20 |
2 | 30 | 30 |
3 | 60 | 40 |
4 | 100 | 50 |
5 | 150 | 60 |
etc...
I would like an xp column in my user table that would represent the total experience of a user.
In my view I would like to display its level. But what is the formula for calculating this?
Let's imagine that my user has 157 experience points in total. Which corresponds to a level 5. How to calculate a level only via the total experience points and how to calculate this regardless of its total experience point whether it is 38 like 369 or 4393.
I'm not sure if this is exactly what you wanted. It's more of an algorithm than a formula, but here is a method that will return the proper level based on the experience given. Think of the bar variable like the bar you have to get over to meet a new level, and think of the step variable like the amount of experience added between each level (as you describe in your right-most column).
def calculate_level(experience)
level = 0
bar = 10
step = 10
while experience >= bar
level += 1
step += 10
bar += step
end
level
end
Example output:
irb> calculate_level(0) # => 0
irb> calculate_level(9) # => 0
irb> calculate_level(10) # => 1
irb> calculate_level(11) # => 1
irb> calculate_level(29) # => 1
irb> calculate_level(30) # => 2
irb> calculate_level(31) # => 2
irb> calculate_level(59) # => 2
irb> calculate_level(60) # => 3
irb> calculate_level(61) # => 3
irb> calculate_level(99) # => 3
irb> calculate_level(100) # => 4
irb> calculate_level(101) # => 4
irb> calculate_level(149) # => 4
irb> calculate_level(150) # => 5
irb> calculate_level(151) # => 5
Calculate the inverse of my above comment and take the floor:
def calculate_level experience
(Math.sqrt(0.2 * experience + 0.25) - 0.5).floor
end
You can use such "formula"
def calculate_level(experience)
(1..).find { |i| (experience -= 10 * i).negative? } - 1
end
Just decrease experience on 10x steps every time and check if result is not negative
calculate_level(0)
# => 0
calculate_level(9)
# => 0
calculate_level(10)
# => 1
calculate_level(11)
# => 1
calculate_level(29)
# => 1
calculate_level(30)
# => 2
calculate_level(31)
# => 2
calculate_level(59)
# => 2
calculate_level(60)
# => 3
calculate_level(61)
# => 3
calculate_level(99)
# => 3
calculate_level(100)
# => 4
calculate_level(101)
# => 4
calculate_level(149)
# => 4
calculate_level(150)
# => 5
calculate_level(151)
# => 5
calculate_level(209)
# => 5
calculate_level(210)
# => 6
well.. This is my first attempt to ml modelling. Given data is like below..
// normalized stock price data with label
// is just EXAMPLE
// $labels = ['10%', '30%', '-30%', ..];
// $samples= [[1,0,45,29,100], [100,13,0,14,5], [3,2,0,10,100], ..];
Each sample has 31 value and all label has 7 types. I successfully trained Model itself but score doesn't add up. Every attempt make under 0.2 score.
array:4 [▼
"environment" => array:4 [▶]
"filename" => "MP-test-220425-010403.rbx"
"scores" => array:9 [▼
1 => 0.077863577863578
2 => 0.077863577863578
3 => 0.071805006587615
4 => 0.14278357892554
5 => 0.077863577863578
6 => 0.077863577863578
7 => 0.13096842384232
8 => 0.078355812459859
9 => 0.077863577863578
]
"losses" => array:9 [▼
1 => 0.2336202611478
2 => 0.22053903758692
3 => 0.22142868877431
4 => 0.21962296766134
5 => 0.21888143998952
6 => 0.21872846102315
7 => 0.21900067894143
8 => 0.21882642822037
9 => 0.21780065553406
]
]
This is what I got .. Scores keep getting same value. What does it mean? I can find examples about underfitting and overfitting but they all just explaining on somewhat high accuracy.. at least above 50% ;;;;
Is.. underfitting or overfitting? Do I need more training data? Data is actually cannot classifiable? ..
What can I do to improving result?
I noticed that often you get ERRNO errors, but have no clue what they mean. Here is some guide, that is helping me and might help others.
In console
110.times do |errno| #there are not more then 110 ERRNO errors
puts "#{ errno } => #{ SystemCallError.new(errno).inspect }"
end
Results. Enjoy your debuging and code digging
0 => #<Errno::NOERROR: Undefined error: 0>
1 => #<Errno::EPERM: Operation not permitted>
2 => #<Errno::ENOENT: No such file or directory>
3 => #<Errno::ESRCH: No such process>
4 => #<Errno::EINTR: Interrupted system call>
5 => #<Errno::EIO: Input/output error>
6 => #<Errno::ENXIO: Device not configured>
7 => #<Errno::E2BIG: Argument list too long>
8 => #<Errno::ENOEXEC: Exec format error>
9 => #<Errno::EBADF: Bad file descriptor>
10 => #<Errno::ECHILD: No child processes>
11 => #<Errno::EDEADLK: Resource deadlock avoided>
12 => #<Errno::ENOMEM: Cannot allocate memory>
13 => #<Errno::EACCES: Permission denied>
14 => #<Errno::EFAULT: Bad address>
15 => #<Errno::ENOTBLK: Block device required>
16 => #<Errno::EBUSY: Resource busy>
17 => #<Errno::EEXIST: File exists>
18 => #<Errno::EXDEV: Cross-device link>
19 => #<Errno::ENODEV: Operation not supported by device>
20 => #<Errno::ENOTDIR: Not a directory>
21 => #<Errno::EISDIR: Is a directory>
22 => #<Errno::EINVAL: Invalid argument>
23 => #<Errno::ENFILE: Too many open files in system>
24 => #<Errno::EMFILE: Too many open files>
25 => #<Errno::ENOTTY: Inappropriate ioctl for device>
26 => #<Errno::ETXTBSY: Text file busy>
27 => #<Errno::EFBIG: File too large>
28 => #<Errno::ENOSPC: No space left on device>
29 => #<Errno::ESPIPE: Illegal seek>
30 => #<Errno::EROFS: Read-only file system>
31 => #<Errno::EMLINK: Too many links>
32 => #<Errno::EPIPE: Broken pipe>
33 => #<Errno::EDOM: Numerical argument out of domain>
34 => #<Errno::ERANGE: Result too large>
35 => #<Errno::EAGAIN: Resource temporarily unavailable>
36 => #<Errno::EINPROGRESS: Operation now in progress>
37 => #<Errno::EALREADY: Operation already in progress>
38 => #<Errno::ENOTSOCK: Socket operation on non-socket>
39 => #<Errno::EDESTADDRREQ: Destination address required>
40 => #<Errno::EMSGSIZE: Message too long>
41 => #<Errno::EPROTOTYPE: Protocol wrong type for socket>
42 => #<Errno::ENOPROTOOPT: Protocol not available>
43 => #<Errno::EPROTONOSUPPORT: Protocol not supported>
44 => #<Errno::ESOCKTNOSUPPORT: Socket type not supported>
45 => #<Errno::ENOTSUP: Operation not supported>
46 => #<Errno::EPFNOSUPPORT: Protocol family not supported>
47 => #<Errno::EAFNOSUPPORT: Address family not supported by protocol family>
48 => #<Errno::EADDRINUSE: Address already in use>
49 => #<Errno::EADDRNOTAVAIL: Can't assign requested address>
50 => #<Errno::ENETDOWN: Network is down>
51 => #<Errno::ENETUNREACH: Network is unreachable>
52 => #<Errno::ENETRESET: Network dropped connection on reset>
53 => #<Errno::ECONNABORTED: Software caused connection abort>
54 => #<Errno::ECONNRESET: Connection reset by peer>
55 => #<Errno::ENOBUFS: No buffer space available>
56 => #<Errno::EISCONN: Socket is already connected>
57 => #<Errno::ENOTCONN: Socket is not connected>
58 => #<Errno::ESHUTDOWN: Can't send after socket shutdown>
59 => #<Errno::ETOOMANYREFS: Too many references: can't splice>
60 => #<Errno::ETIMEDOUT: Operation timed out>
61 => #<Errno::ECONNREFUSED: Connection refused>
62 => #<Errno::ELOOP: Too many levels of symbolic links>
63 => #<Errno::ENAMETOOLONG: File name too long>
64 => #<Errno::EHOSTDOWN: Host is down>
65 => #<Errno::EHOSTUNREACH: No route to host>
66 => #<Errno::ENOTEMPTY: Directory not empty>
67 => #<Errno::EPROCLIM: Too many processes>
68 => #<Errno::EUSERS: Too many users>
69 => #<Errno::EDQUOT: Disc quota exceeded>
70 => #<Errno::ESTALE: Stale NFS file handle>
71 => #<Errno::EREMOTE: Too many levels of remote in path>
72 => #<Errno::EBADRPC: RPC struct is bad>
73 => #<Errno::ERPCMISMATCH: RPC version wrong>
74 => #<Errno::EPROGUNAVAIL: RPC prog. not avail>
75 => #<Errno::EPROGMISMATCH: Program version wrong>
76 => #<Errno::EPROCUNAVAIL: Bad procedure for program>
77 => #<Errno::ENOLCK: No locks available>
78 => #<Errno::ENOSYS: Function not implemented>
79 => #<Errno::EFTYPE: Inappropriate file type or format>
80 => #<Errno::EAUTH: Authentication error>
81 => #<Errno::ENEEDAUTH: Need authenticator>
82 => #<SystemCallError: Device power is off>
83 => #<SystemCallError: Device error>
84 => #<Errno::EOVERFLOW: Value too large to be stored in data type>
85 => #<SystemCallError: Bad executable (or shared library)>
86 => #<SystemCallError: Bad CPU type in executable>
87 => #<SystemCallError: Shared library version mismatch>
88 => #<SystemCallError: Malformed Mach-o file>
89 => #<Errno::ECANCELED: Operation canceled>
90 => #<Errno::EIDRM: Identifier removed>
91 => #<Errno::ENOMSG: No message of desired type>
92 => #<Errno::EILSEQ: Illegal byte sequence>
93 => #<Errno::ENOATTR: Attribute not found>
94 => #<Errno::EBADMSG: Bad message>
95 => #<Errno::EMULTIHOP: EMULTIHOP (Reserved)>
96 => #<Errno::ENODATA: No message available on STREAM>
97 => #<Errno::ENOLINK: ENOLINK (Reserved)>
98 => #<Errno::ENOSR: No STREAM resources>
99 => #<Errno::ENOSTR: Not a STREAM>
100 => #<Errno::EPROTO: Protocol error>
101 => #<Errno::ETIME: STREAM ioctl timeout>
102 => #<Errno::EOPNOTSUPP: Operation not supported on socket>
103 => #<SystemCallError: Policy not found>
104 => #<Errno::ENOTRECOVERABLE: State not recoverable>
105 => #<Errno::EOWNERDEAD: Previous owner died>
106 => #<SystemCallError: Interface output queue is full>
107 => #<SystemCallError: Unknown error: 107>
108 => #<SystemCallError: Unknown error: 108>
109 => #<SystemCallError: Unknown error: 109>
110 => #<SystemCallError: Unknown error: 110>
I have an application developed on BlackBerry JE 4.6.1 that decrypts an information from WebServer using DES algorythm.
If I send encrypted information to the server, it is decrypted well. But in case if the server sends an encrypted data,
I do not get the correct value after decryption.
Key is supposed to be the same and crypted information is sent base64 encoded.
During debugging I have found out, that after DESKey is created it's inner data differs from the byte array passed to the constructor.
For example if I create the DESKey the next way
String keyStr = "2100000A";
DESKey desKey = new DESKey(keyStr.getBytes()); // pass the byte array {'2','1','0','0','0','0','0','A'}
the method desKey.getData() returns the byte array {'2','1','1','1','1','1','1','#'} that differs from the initial key bytes.
So is it possible for such behavior of the DESKey to be the reason why I can not decrypt data from server?
Thank you.
The desKey.getData() behaviour is expected.
The doc states:
DES operates on 64 bit blocks and has
an effective key length of 56 bits. In
reality, the key is 64 bits but there
are 8 bits of parity used which means
that the effective key length is only
56 bits. Every eighth bit is used for
parity and it is the least significant
bit that is used for parity.
Parity bit definition:
A parity bit, or check bit, is a bit that is added to ensure that the number of bits with the value one in a set of bits is even or odd. Parity bits are used as the simplest form of error detecting code.
So, this is how it happens:
'2' => 0x32 => 00110010 => 0011001 + (parity bit 0) => 00110010 => 0x32 => '2'
'1' => 0x31 => 00110001 => 0011000 + (parity bit 1) => 00110001 => 0x31 => '1'
'0' => 0x30 => 00110000 => 0011000 + (parity bit 1) => 00110001 => 0x31 => '1'
'0' => 0x30 => 00110000 => 0011000 + (parity bit 1) => 00110001 => 0x31 => '1'
'0' => 0x30 => 00110000 => 0011000 + (parity bit 1) => 00110001 => 0x31 => '1'
'0' => 0x30 => 00110000 => 0011000 + (parity bit 1) => 00110001 => 0x31 => '1'
'0' => 0x30 => 00110000 => 0011000 + (parity bit 1) => 00110001 => 0x31 => '1'
'A' => 0x41 => 01000001 => 0100000 + (parity bit 0) => 01000000 => 0x40 => '#'
I need to format an integer representation of bytes into something friendly, and I'm hoping that there's a utility function in Ruby or in Rails that will do that formatting for me (to perpetuate my laziness, of course.)
I'm looking for something that would look like:
format_bytes(1024) -> "1 KB"
format_bytes(1048576) -> "1 MB"
Looks like there's some stuff in ActiveSupport to do it the other way around, but I haven't found a way to do it in this direction.
If there isn't one that exists, does anyone have a particularly elegant solution?
Number to human size is what you're looking for.
require 'action_view'
include ActionView::Helpers::NumberHelper
number_to_human_size(123) # => 123 Bytes
number_to_human_size(1234) # => 1.2 KB
number_to_human_size(12345) # => 12.1 KB
number_to_human_size(1234567) # => 1.2 MB
number_to_human_size(1234567890) # => 1.1 GB
number_to_human_size(1234567890123) # => 1.1 TB
number_to_human_size(1234567, :precision => 2) # => 1.18 MB
number_to_human_size(483989, :precision => 0) # => 473 KB
number_to_human_size(1234567, :precision => 2, :separator => ',') # => 1,18 MB
Accepted answer still work, but requires actionpack instead of actionview in newer rails.
require 'actionpack'
Accepted answer it's perfect, but I didn't need the first two lines.
I only put:
number_to_human_size(123) # => 123 Bytes
number_to_human_size(1234) # => 1.2 KB
number_to_human_size(12345) # => 12.1 KB
number_to_human_size(1234567) # => 1.2 MB
number_to_human_size(1234567890) # => 1.1 GB
number_to_human_size(1234567890123) # => 1.1 TB
number_to_human_size(1234567, :precision => 2) # => 1.18 MB
number_to_human_size(483989, :precision => 0) # => 473 KB
number_to_human_size(1234567, :precision => 2, :separator => ',') # => 1,18 MB
and works like a charm.