I am new to .NET programming. Sorry if this question has been asked before.
I am currently learning F#. What are the differences between Dictionary, Hashtable and Map? When should I use each?
I also have another question that is not mentioned in the title. When should I use Async.RunSynchronously? It seems rather self-contradictory to me, so I am sure that I am missing something.
The choice between Dictionary, Hashtable and Map depends on the uses cases. You should however know the characteristics of each. This is not an exhaustive list but just some key differences you might want to start from :
Hashtable Represents a collection of key/value pairs that are organized based on the hash code of the key. This is a mutable collection from .NET BCL
Dictionary<> this is a generic implementation of the hashtable. Also a mutable collection from .NET BCL
Map This is the F# immutable type. It is implemented based on AVL trees which is an entirely different data structure with different performance characteristics and use cases.
If you are doing many writes, Hash tables collections have significantly better fill rate performance than AVL trees.
Retrieving a value from a Dictionary by using its key is very fast, close to O(1), because the Dictionary class is implemented as a hash table.
F# maps are implemented as immutable AVL trees, an efficient data structure which forms a self-balancing binary tree. AVL trees are well-known for their efficiency, in which they can search, insert, and delete elements in the tree in O(log n) time, where n is the number of elements in the tree.
As for the map uses case, if you’ve got a set of static data (such as configuration data that’s loaded when your application starts up) you need to look up by key frequently, a Map is as good a choice as any, its immutability in this case ensures that the static data cannot be modified by mistake and has little impact to performance as you never need to mutate it once initialized.
Async.RunSynchronously runs the provided asynchronous computation and awaits its result. You can use it in F# interactive window for example to test your asynchronous workflows.
Related
So I am trying to use the F# Set as a hash table. But my element type doesn't implement the IComparable interface (although it implements IEquatable). I got an error saying the construction is not allowed because of comparison constraint. And through some further read, I discovered that F# Set is implemented using binary tree, which makes insertion causes O(log(n)). This looks weird to me, why is the Set structure designed this way?
Edit: So I learned that Set in F# is actually a SortedSet. And I guess the question becomes, why is Sorted Set somehow more preferable than a general Hash Set as an immutable/functional data structure?
There are two important points that should help you understand how sets in F# (and in functional languages in general) work and how they are used:
Implementing immutable hashtables (like .NET HashSet) is hard - when you remove or add elements, you want to avoid copying everything in the data structure and (as far as I know) there is no general way of doing that (you would end up copying too much, so it would be inefficient).
For this reason, most functional sets are implemented as (some form of trees). Those require comparison to build a sorted tree. The nice property of balanced trees is that removing and adding elements does not have to copy everything in the tree, so even the worst case scenario is reasonably efficient (though mutable hashtable is still faster).
Now, F# is functional-first, which means that immutable structures are preferred, but it is perfectly fine to use mutable data structures (especially if you limit the usage to some well defined and restricted scope). For this reason, F# programmers often use Dictionary or HashSet, especially when this is only within the scope of a single function.
I need to maintain correspondence between strings and integers, then lookup the string value and return the integer. What's the best structure to store this info that meets the following requirements:
Speed and memory size are important, in that order.
I don't want to reinvent the wheel and write my own sorting routine. A call to Sort(CompareFunction) is fine of course.
Conditions:
The integers are not guaranteed to be sequential, neither is there a 'start value' like 0 or 1
Number of data pairs can vary from 100 to 100000
The data are all read in at the beginning, there's no subsequent additions/deletions/modifications
FWIW the strings are the hex entry ID's that Outlook (MAPI?) uses to identify entries. Example: 00000000FE42AA0A18C71A10E8850B651C24000003000000040000000000000018000000000000001E7FDF4152B0E944BA66DFBF2C6A6416E4F52000487F22
There's so many options (TStringList (with objects or name/value pairs), TObjectList, TDictionary, ...) that I'd better ask for advice first...
I have read How can I search faster for name/value pairs in a Delphi TStringList? which suggest TDictionary for string/string pairs, and Sorting multidimensional array in Delphi 2007 which suggest TStringlist objects for string/integer but where sorting is done on the integers.
The second link that you include in the question is not applicable. That is a question concerning sorting rather than efficient lookup. Although you discuss sorting a number of times in your question, you do not have a requirement to sort. Your requirement is simply a dictionary, also known as an associative array. Of course, you can implement that by sorting an array and using binary search for your lookup, but sorting is not a requirement. You simply need an efficient dictionary.
Out of the box, the most efficient and convenient data structure for your problem is TDictionary<string, Integer>. This has lookup complexity of O(1) and so scales well for large collections. For smaller collections a binary search based lookup with lookup complexity of O(log n) can be competitive and can indeed out-perform a dictionary.
Cosmin Prund wrote an excellent answer here on SO where he compared the performance of dictionary lookup against binary search based lookup. I recommend you have a read. I would say that for small containers, performance is probably not that big a problem for you. So even though binary search may be quicker, it probably does not matter because your performance is good either way. But performance probably becomes an issue for larger containers and that's where the dictionary is always stronger. For large enough containers, the performance of binary search may become unacceptable.
I'm sure that it is possible to produce more efficient implementations of dictionaries than the Embarcadero one, but I'd also say that the Embarcadero implementation is perfectly solid. It uses a decent hash function and does not have any glaring weaknesses.
In terms of memory complexity, there's little to choose between a dictionary and a sorted array. It's not possible to improve on a sorted array for memory use.
I suggest that you start with TDictionary<string, Integer> and only look beyond that if your performance requirements are not met.
It seems you are going to lookup long evenly distributed strings. One of the fastest data structures for this kind of problem is Trie.
But your dataset size is rather small, and ready-to-use Delphi solutions like THashedStringList or TDictionary (more convenient) would provide a fairly high speed.
With focus on read performance, I want to create a Term such as an Orddict or Proplist that contains a large number (100,000s) entries, each containing an ID and a Term value. This encapsulating Term should be able to return the a value stored under its key, just like an Orddict is able to do.
example:
K001 - Term001
K002 - Term002
K003 - Term003
The resulting Term containing the whole set needs to be passed from function to function, for several computing purposes without storing it on a persistence store to avoid disk I/O. I also chose not to use memory caching at this stage to avoid architectural complexity at this moment, therefore my focus is to have all of this to be simply key-searcheable.
Orddicts are key-sorted, which enhance the seek of a key, compared to a normal Dict. I am not aware of any other Erlang Module that can embed a more efficient indexing mechanism within its Term.
Any suggestions for an approach better than an Orddict ?
Actually, orddict is implemented as a sorted list (source), so it performs poorly both for insertion and lookup, especially when the keys are inserted in ascending order. Stay away from it; it won't work for your use case. dict is a hash-based data structure and offers solid insert/lookup performance. If the order of keys is important to you, consider using a tree-based map (such as gb_trees) as you can extract an ordered key sequence by taking the in-order tree walk.
If you want to share a large dataset between Erlang processes, you can try to use ETS.
It is fast in-memory key-value store, that only supports destructive updates.
I am able to understand immutability with python (surprisingly simple too). Let's say I assign a number to
x = 42
print(id(x))
print(id(42))
On both counts, the value I get is
505494448
My question is, does python interpreter allot ids to all the numbers, alphabets, True/False in the memory before the environment loads? If it doesn't, how are the ids kept track of? Or am I looking at this in the wrong way? Can someone explain it please?
What you're seeing is an implementation detail (an internal optimization) calling interning. This is a technique (used by implementations of a number of languages including Java and Lua) which aliases names or variables to be references to single object instances where that's possible or feasible.
You should not depend on this behavior. It's not part of the language's formal specification and there are no guarantees that separate literal references to a string or integer will be interned nor that a given set of operations (string or numeric) yielding a given object will be interned against otherwise identical objects.
I've heard that the C Python implementation does include a set of the first hundred or so integers as statically instantiated immutable objects. I suspect that other very high level language run-time libraries are likely to include similar optimizations: the first hundred integers are used very frequently by most non-trivial fragments of code.
In terms of how such things are implemented ... for strings and larger integers it would make sense for Python to maintain these as dictionaries. Thus any expression yielding an integer (and perhaps even floats) and strings (at least sufficiently short strings) would be hashed, looked up in the appropriate (internal) object dictionary, added if necessary and then returned as references to the resulting object.
You can do your own similar interning of any sorts of custom object you like by wrapping the instantiation in your own calls to your own class static dictionary.
If I want to make a time series type in F# to hold stock prices, which basic type should I use? We need
Select a subset based on time index,
Calculate basic statistics for a subset like mean, STD or for several subsets like correlations,
Append item for new data and fast update statistics or technical indicators,
Do linear regression between time series, etc
I have read that array has a better performance, seq has a smaller memory footnote, list is better for adding items and F# vector is easier for certain math calculation. To balance all the trade offs, how would you model a stock price time series in f#? Thanks.
As a concrete representation you can choose either array or list or some other .NET colllection type. A sequence seq<'T> is an abstract type and both array and list are automatically also sequences - this means that when you write some code that works with sequences, it will work with any concrete data type (array, list or any other .NET collection).
So, when writing data processing, you can use Seq by default (as it gives you great flexibility - it doesn't matter what concrete representation you use) and then optimize some operations to use the concrete representation (whatever that will be) if you need something to run faster.
Regarding the concrete representation - I think the crucial question is whether you want to add elements without changing original data structure (immutable list or array used in an immutable way) or whether you want to mutate the data structure (e.g. use some mutable .NET collection).
If you need to add new items freuqently then you can either use immutable list (which supports appending elements to front) or a mutable collection (array won't do as it cannot be resized).
If you're working on a more sophisticated system, I would recommend taking a look at ObservableCollection<T> (see MSDN). This is a collection that automatically notifies you when it is changed. In response to the notification, you could update your statistics (it also tells you which elements were added, so you don't need to recalculate everything). However, F# doesn't have any libraries for working with this type, so you'll need to write a lot of things yourself.
If you're adding data only rarely or adding them in larger groups, you could use array (and allocate new array each time you add items). If you have only relatively small number of items in the collection, you could use lists (where adding item is easy).
For numerical calculations, the F# PowerPack (and types like vector) offer only quite limitied set of features, so you may need to look at some thrid party libraries. Extreme optimizations is a commercial library with some F# examples and Math.NET is an open source alternative.
Otherwise, it is difficult to give any concrete advice - can you add some more details about your system? (e.g. how large the data set is, how many items need to be added how often etc...)