Related
I am new to google cloud data platform as well as to Apache beam api. I would like aggregate data based on multiple keys. In my requirement I will get a transaction feed having fields like customer id,customer name,transaction amount and transaction type. I would like to aggregate the data based on customer id & transaction type. Here is an example.
customer id,customer name,transction amount,transaction type
cust123,ravi,100,D
cust123,ravi,200,D
cust234,Srini,200,C
cust444,shaker,500,D
cust123,ravi,100,C
cust123,ravi,300,C
O/p should be
cust123,ravi,300,D
cust123,ravi,400,C
cust234,Srini,200,C
cust444,shaker,500,D
In google most of the examples are based on single key like group by single key. Can any please help me on how my PTransform look like in my requirement and how to produce aggregated data along with rest of the fields.
Regards,
Ravi.
Here is an easy way. I concatenated all the keys together to form a single key and then did the the sub and after than split the key to organize the output to a way you wanted. Please let me know if any question.
The code does not expect header in the CSV file. I just kept it short to show the main point you are asking.
import apache_beam as beam
import sys
class Split(beam.DoFn):
def process(self, element):
"""
Splits each row on commas and returns a tuple representing the row to process
"""
customer_id, customer_name, transction_amount, transaction_type = element.split(",")
return [
(customer_id +","+customer_name+","+transaction_type, float(transction_amount))
]
if __name__ == '__main__':
p = beam.Pipeline(argv=sys.argv)
input = 'aggregate.csv'
output_prefix = 'C:\\pythonVirtual\\Mycodes\\output'
(p
| 'ReadFile' >> beam.io.ReadFromText(input)
| 'parse' >> beam.ParDo(Split())
| 'sum' >> beam.CombinePerKey(sum)
| 'convertToString' >>beam.Map(lambda (combined_key, total_balance): '%s,%s,%s,%s' % (combined_key.split(",")[0], combined_key.split(",")[1],total_balance,combined_key.split(",")[2]))
| 'write' >> beam.io.WriteToText(output_prefix)
)
p.run().wait_until_finish()
it will produce output as below:
cust234,Srini,200.0,C
cust444,shaker,500.0,D
cust123,ravi,300.0,D
cust123,ravi,400.0,C
this might be a realy easy question but i think i am mentaly blind or something; How can i get a key by its index in The Dictionary class In Delphi (10.1). I mean the structure has a property called Count, so it must have a some sort of array or a list in it, why cant i get the keys by indices.
I also tried KeyCollection property in Dictionary class, but it also doesn't have anything useful. I need something like:
key: string;
key := dicTest.GetKey(keyIndex);
Thanks a lot.
The Delphi RTL generic dictionary is unordered. As a consequence of being unordered, items in the container do not possess a meaningful index.
The keys can be enumerated using the Keys property:
var
dict: TDictionary<string, Integer>;
key: string;
....
for key in dict.Keys do
Writeln(key);
Likewise the values can be enumerated using the Values property.
var
dict: TDictionary<string, Integer>;
value: Integer;
....
for value in dict.Values do
Writeln(value);
If you wish to enumerate key/value pairs then the dictionary itself provides an enumerator for that:
var
dict: TDictionary<string, Integer>;
item: TPair<string, Integer>;
....
for item in dict do
Writeln(item.Key, ', ', item.Value);
Note that for each of these enumerators, no guarantees are made about the order of the items. Simple acts like adding a new item to the dictionary can result in a change in the order of the items under enumeration.
To add to David's answer, the whole point of a dictionary or hash-table structure is to very efficiently store and retrieve key-value pairs in memory.
This is achieved as follows:
Items are placed into a predictable location within in large block of memory based on the key.
When trying to find the item, you know where it should be stored (based on the key), and can immediately go to that location to hopefully find it.
The following diagram illustrates:
+-------------+
| .. |
| .. |
Add |World (Data) | Find
Hello |Abc (Data) | Hello
| | .. | |
| | .. | |
+---> |Hello (Data) | <---+
| .. |
| .. |
|Xyz (Data) |
+-------------+
Note the following:
There is no ordering as to where items will be inserted.
The basic operations require knowing the key to look them up.
Although Delphi allows the items to be iterated, their positions are not guaranteed to be consistent.
The big advantage of this kind of structure is that it doesn't matter how large the collection grows, adding and looking up and deleting items has the same performance. This is referred to as having order of complexity O(1).
These structures tend to be inefficient in terms of memory requirements. You may have noted the "gaps" in the allocated memory in the illustration above.
There's no built in mechanism for knowing the number of items in the collection, but it's trivial track the number as items are added/removed in a single extra field. (Which is what Delphi's implementation does.) Knowing count does not imply you can access items by index.
In summary
If you're unable to keep track of your keys, then a Dictionary is not the right tool for you. You may be better with a List or array. Though I suggest you make an effort to understand the benefits and limitations of these structures to help decide which is the best tool for the job.
The easiest way to access the keys by index is to retrieve them as an array. The Keys property provides a function for that: ToArray
I am looking for a data structure that can mimic an Object or a struct. Really, just some compact way to pass around different types of variables. Currently I am using a tuple but referencing various parts of the tuple is less pleasant than I would like. Currently I've just created aliases that represent the various locations in the tuple:
alias AuxClass = tuple[str,str,list[int],list[int],Dec];
int ACLS = 0;
But I've had to restructure this tuple and thus had to change the indexing. Is there something I can use here that I've missed or perhaps a feature coming in the future?
Thanks!
Please take a look at the algebraic data types feature:
http://tutor.rascal-mpl.org/Rascal/Rascal.html#/Rascal/Declarations/AlgebraicDataType/AlgebraicDataType.html
You can create a constructor to represent the type of data that you are trying to define above, similar to what you would do with a struct, and give each element in the constructor a field name:
data AuxClass = auxClass(str f1, str f2, list[int] f3, list[int] f4, Dec f5)
You can then create new instances of this just using the constructor name and providing the data:
a = auxClass("Hello", "World", [1,2,3], [4,5,6], D1) (where D1 is a Dec).
Once you have an instance, you can access information using the field names:
a.f1 // which equals "Hello"
a.f3 // which equals [1,2,3]
size(a.f3) // which currently equals 3
and you can update information using the field names:
a.f2 = "Rascal"
a.f4 = a.f4 + 7 // f4 is now [4,5,6,7]
Algebraic data types are actually quite flexible, so there is a lot you can do with them beyond this. Feel free to look through the documentation and ask questions here.
I tried to display datas which is in Dictionary format. Below, three attempts are there. First attempt, output order is completely changed. Second attempt, output order is same as input. But, in third attempt, I declared variable as NSDictionary. Exact output I received. Why this changes in Dictionary? Kindly guide me. I searched for Swift's Dictionary tag. But I couldn't found out.
//First Attempt
var dict : Dictionary = ["name1" : "Loy", "name2" : "Roy"]
println(dict)
//output:
[name2: Roy, name1: Loy]
//Second Attempt
var dict : Dictionary = ["name2" : "Loy", "name1" : "Roy"]
println(dict)
//output:
[name2: Loy, name1: Roy]
-----------------------------------------------------------
//Third Attempt With NSDictionary
var dict : NSDictionary = ["name1" : "Loy", "name2" : "Roy"]
println(dict)
//output:
{
name1 = Loy;
name2 = Roy;
}
ANOTHER QUERY: I have used play ground to verify. My screen shot is below:
Here, In NSDictionary, I gave name5 as first, but in right side name2 is displaying, then, in println, it is displaying in ascending order. Why this is happening??
Here, In Dictionary, I gave name5 as first, but in right side name2 is displaying, then, in println, it is displaying, how it is taken on the Dictionary line. Why this is happening??
This is because of the definition of Dictionaries:
Dictionary
A dictionary stores associations between keys of the same type and values of the same type in an collection with no defined ordering.
There is no order, they might come out differently than they were put in. This is comparable to NSSet.
Edit:
NSDictionary
Dictionaries Collect Key-Value Pairs. Rather than simply maintaining an ordered or unordered collection of objects, an NSDictionary stores objects against given keys, which can then be used for retrieval.
There is also no order, however there is sorting on print for debugging purposes.
You can't sort a dictionary but you can sort its keys and loop through them as follow:
let myDictionary = ["name1" : "Loy", "name2" : "Roy", "name3" : "Tim", "name4" : "Steve"] // ["name1": "Loy", "name2": "Roy", "name3": "Tim", "name4": "Steve"]
let sorted = myDictionary.sorted {$0.key < $1.key} // or {$0.value < $1.value} to sort using the dictionary values
print(sorted) // "[(key: "name1", value: "Loy"), (key: "name2", value: "Roy"), (key: "name3", value: "Tim"), (key: "name4", value: "Steve")]\n"
for element in sorted {
print("Key = \(element.key) Value = \(element.value)" )
}
A little late for the party but if you want to maintain the order then use KeyValuePairs, the trade-off here is that if you use KeyValuePairs you lose the capability of maintaining unique elements in your list
var user: KeyValuePairs<String, String> {
return ["FirstName": "NSDumb",
"Address": "some address value here",
"Age":"30"]
}
prints
["FirstName": "NSDumb", "Address": "some address value", "Age": "30"]
Dictionaries, by nature, are not designed to be ordered, meaning that they're not supposed to be (although they can be!).
From the Dictionaries (Swift Standard Library documentation):
A dictionary is a type of hash table, providing fast access to the entries it contains. Each entry in the table is identified using its key, which is a hashable type such as a string or number. You use that key to retrieve the corresponding value, which can be any object. In other languages, similar data types are known as hashes or associated arrays.
This requires some basic knowledge of Data Structures, which I'll outline & oversimplify briefly.
Storing associated data without a dictionary
Consider for a minute if there was no Dictionary and you had to use an array of tuples instead, to store some information about different fruits and their colors, as another answer suggested:
let array = [
("Apple", "Red"),
("Banana", "Yellow"),
// ...
]
If you wanted to find the color of a fruit you'd have to loop through each element and check its value for the fruit, then return the color portion.
Dictionaries optimize their storage using hash functions to store their data using a unique hash that represents the key that is being stored. For swift this means turning our key—in this case a String—into an Int. Swift uses Int-based hashes, which we know because we all read the Hashable protocol documentation and we see that Hashable defines a hashValue property that returns an Int.
Storing associated data with a dictionary
The benefits of using a dictionary are that you get fast read access and fast write access to data; it makes "looking up" associated data easy and quick. Typically O(1) time complexity, although the apple docs don't specify, maybe because it depends on the key type's hash function implementation.
let dictionary = [
"Apple": "Red",
"Banana": "Yellow"
// ...
]
The trade off is that the order is typically not guaranteed to be preserved. Not guaranteed means that you might get lucky and it might be the same order, but it's not intended to be, so don't rely on it.
As an arbitrary example, maybe the string "Banana" gets hashed into the number 0, and "Apple" becomes 4. Since we now have an Int we could, under the hood, represent our dictionary as an array of size 5:
// what things *might* look like under, the hood, not our actual code
// comments represent the array index numbers
let privateArrayImplementationOfDictionary = [
"Yellow", // 0
nil, // 1
nil, // 2
nil, // 3
"Red", // 4
] // count = 5
You'll notice, we've converted our keys into array indices, and there are a bunch of blank spaces where we have nothing. Since we are using an array, we can insert data lightning fast, and retrieve it just as quickly.
Those nil spaces are reserved for more values that may come later, but this is also why when we try to get values out of a dictionary, they might be nil. So when we decide to add more values, something like:
dictionary["Lime"] = "Green" // pretend hashValue: 2
dictionary["Dragonfruit"] = "Pink" // pretend hashValue: 1
Our dictionary, under the hood, may look like this:
// what things *might* look like under, the hood, not our actual code
// comments represent the array index numbers
let privateArrayImplementationOfDictionary = [
"Yellow", // 0 ("Banana")
"Pink", // 1 ("Dragonfruit")
"Green", // 2 ("Lime")
nil, // 3 (unused space)
"Red", // 4 ("Apple")
] // count = 5
As you can see, the values are not stored at all in the order we entered them. In fact, the keys aren't even really there. This is because the hash function has change our keys into something else, a set of Int values that give us valid array indices for our actual storage mechanism, an array, which is hidden from the world.
I'm sure that was more information than you wanted and probably riddled with many inaccuracies, but it gives you the gist of how a dictionary works in practice and hopefully sounds better than, "that's just how it works."
When searching for the actual performance of Swift dictionaries, Is Swift dictionary ... indexed for performance? ... StackOverflow had some extra possible relevant details to offer.
If you're still interested to know more details about this, you can try implementing your own dictionary as an academic exercise. I'd also suggest picking up a book on Data Structures and Algorithms, there are many to choose from, unfortunately I don't have any suggestions for you.
The deeper you get into this topic the more you'll understand why you'll want to use one particular data structure over another.
Hope that helps!
✅ It is possible!
Although the Dictionary is not ordered, you can make it preserve the initial order by using the official OrderedDictionary from the original Swift Repo
The ordered collections currently contain:
Ordered Dictionary (That you are looking for)
Ordered Set
They said it is going to be merged in the Swift's source code soon (reference WWDC21)
Neither NSDictionary nor Swift::Dictionary orders its storage. The difference is that some NSDictionary objects sort their output when printing and Swift::Dictionary does not.
From the documentation of -[NSDictionary description]:
If each key in the dictionary is an NSString object, the entries are
listed in ascending order by key, otherwise the order in which the
entries are listed is undefined. This property is intended to produce
readable output for debugging purposes, not for serializing data.
From The Swift Programming Language:
A dictionary stores associations between keys of the same type and values of the same type in an collection with no defined ordering.
Basically, order of items as seen in output is arbitrary, dependant on internal implementation of data structure, and should not be relied on.
This is indeed an issue with dictionaries. However, there's a library available to make sure the order stays the way you initialised it.
OrderedDictionary is a lightweight implementation of an ordered dictionary data structure in Swift.
The OrderedDictionary structure is an immutable generic collection which combines the features of Dictionary and Array from the Swift standard library. Like Dictionary it stores key-value pairs and maps each key to a value. Like Array it stores those pairs sorted and accessible by a zero-based integer index.
Check it out here:
https://github.com/lukaskubanek/OrderedDictionary
I would like to access value with the given pattern from hash in redis with out providing key.
Example
HSET myKey va11 "Hello" val2 "Hi" Val3 "GooMorning" val4 "Good Evening"
HSET myKey2 va11 "one val2 "two" Val3 "three" val4 "four"
I have set of keys with their values as above. Is there any way to retrieve values without providing keys.
i just want to check is there any value with Good* something like that without providing key.
I saw that you're using the "lua" tag - if LUA is not a a must, please consider the following example using HVALS. I'l provide some redis-py code to go with:
import redis
# connect to local redis-server:6379
r = redis.Redis()
# initialize sample "myKey" hash
r.hmset("myKey", {'val1': "Hello", 'val2': "Hi", 'val3': "GoodMorning", 'val4': "Good Evening"})
# provide "starts with" pattern (this could obviously be a regex as well)
pattern = "Good"
# find all values starting with the pattern and return them
values_matching = filter(lambda x: x.startswith(pattern), r.hvals("myKey"))
# print values: ["GoodMorning", "Good Evening"]
print values_matching
You don't say that you need the keys for the matching values. If you do, then you should look into the HGETALL command.
EDIT after reading your comments: yes, you will need to loop over all key/values in the hash using HGETALL. An amended example following:
import redis
r = redis.Redis()
r.hmset("myKey", {'val1': "Hello", 'val2': "Hi", 'val3': "GoodMorning", 'val4': "Good Evening"})
pattern = "Good"
# use hgetall() to iterate over all key-value pairs and form values_matching by looking only at pairs where the value starts with "Good"
values_matching = dict([(k, v) for k, v in r.hgetall("myKey").iteritems() if v.startswith(pattern)])
# print values: {'val3': 'GoodMorning', 'val4': 'Good Evening'}
print values_matching
Redis doesn't provide this functionality out of the box. However, if I understand your question correctly, you can easily add it.
Keep a sorted set, for example call it allvals, with all the values from your hashes (i.e. do a ZADD to it on every hash update). Use ZRANGEBYLEX on it for suffix searching as means for existence checking:
ZRANGEBYLEX allvals "[Good" "[Good\xff"
EDIT: Given your clarification, this approach is not what you're looking for. Here's my take on the revised challenge.
So. What you're looking for is a way to fetch hash key names if they contain a specific value in one of their fields. One way would be to scan all you hashes and look for that value (as #tobiash suggested) but, depending on the size of your dataset, this may be quite expensive. Another way to go about this is to use sets as means for indexing your hashes. The basic premise is to take every value in the hash(es) and create sets for it. Each such set's members would be the actual key names of the hashes that contain that value.
Let's see how your example's myKey should be indexed (yes, we're building an index here). To begin with, you'll create the following sets out of it:
SADD idx:Hello myKey
SADD idx:Hi myKey
SADD idx:GooMorning myKey
SADD "idx:Good Evening" myKey
Now, by doing SMEMBERS on idx:Hello, you'll get all the key names that have Hello as values. However, since what you're looking for is doing suffix searching, for each value you'll actually need to maintain multiple sets, where each set indexes a substring of the value. For example, for the string Hello you'll actually need:
SADD idx:H myKey
SADD idx:He myKey
SADD idx:Hel myKey
SADD idx:Hell myKey
SADD idx:Hello myKey
Lua can help here if you'll create a script that you'll call to update your hashes. The script should be given the hash key name and the field names + values. Once called, the script will need not only to create/update the actual hash, but also to iterate over all field values and construct the index for them (and their respective substrings).
Let us know if you need help with writing that script ;)