complicated join in spark: rdd elements have many key-value pairs - join

Question

I'm new to spark and trying to find a way to integrate information from one rdd into another, but their structures don't lend themselves to a standard join function
I have on rdd of this format:
[{a:a1, b:b1, c:[1,2,3,4], d:d1},
{a:a2, b:b2, c:[5,6,7,8], d:d2}]
and another of this format:
[{1:x1},{2,x2},{3,x3},{4,x4},{5,x5},{6,x6},{7,x7},{8,x8}]
I want to match the values in the second rdd to their keys in the first rdd (which are in a list value in the c key). I know how to manipulate them once they're there, so I'm not too concerned about the final output, but I'd maybe like to see something like this:
[{a:a1, b:b1, c:[1,2,3,4],c0: [x1,x2,x3,x4], d:d1},
{a:a2, b:b2, c:[5,6,7,8],c0: [x5,x6,x7,x8], d:d2}]
or this:
[{a:a1, b:b1, c:[(1,x1),(2,x2),(3,x3),(4,x4)], d:d1},
{a:a2, b:b2, c:[(5,x5),(6,x6),(7,x7),(8,x8)], d:d2}]
or anything else that can match the keys in the second rdd with the values in the first. I considered making the second rdd into a dictionary, which I know how to work with, but I just think my data is too large for that.
Thank you so much, I really appreciate it.

Answer 1

join after flatMap, or cartesian makes too many shuffles.
One of the possible solutions is to use cartesian after groupBy with HashPartitioner.
(Sorry, this is scala code)
val rdd0: RDD[(String, String, Seq[Int], String)]
val rdd1: RDD[(Int, String)]
val partitioner = new HashPartitioner(rdd0.partitions.size)
// here is the point!
val grouped = rdd1.groupBy(partitioner.getPartition(_))
val result = rdd0.cartesian(grouped).map { case (left, (_, right)) =>
val map = right.toMap
(left._1, left._2, left._4) -> left._3.flatMap(v => map.get(v).map(v -> _))
}.groupByKey().map { case (key, value) =>
(key._1, key._2, value.flatten.toSeq, key._3)
}

Answer 2

I will assume that rdd1 is the input containing {a:a1, b:b1, c:[1,2,3,4], d:d1} and rdd2 has tuples [(1, x1), (2, x2), (3, x3), (4, x4), (5, x5), (6, x6), (7, x7), (8, x8)]. I will also assume that all values in the "c" field in rdd1 can be found in rdd2. If not, you need to change some of the code below.
I sometimes have to solve this type of problem. If rdd2 is small enough, I typically do a map-side join, where I first broadcast the object and then do a simple lookup.
def augment_rdd1(line, lookup):
c0 = []
for key in line['c']:
c0.append(lookup.value[key])
return c0
lookup = sc.broadcast(dict(rdd2.collect()))
output = rdd1.map(lambda line: (line, augment_rdd1(line, lookup)))
If rdd2 is too large to be broadcasted, what I normally do is use a flatMap to map every row of rdd1 to as many rows as there are elements in the "c" field, e.g. {a:a1, b:b1, c:[1,2,3,4], d:d1} would be mapped to
(1, {a:a1, b:b1, c:[1,2,3,4], d:d1})
(2, {a:a1, b:b1, c:[1,2,3,4], d:d1})
(3, {a:a1, b:b1, c:[1,2,3,4], d:d1})
(4, {a:a1, b:b1, c:[1,2,3,4], d:d1})
The flatMap is
flat_rdd1 = rdd1.flatMap(lambda line: [(key, line) for key in line['c'])])
Then, I would join with rdd2 to get an RDD which has:
({a:a1, b:b1, c:[1,2,3,4], d:d1}, x1)
({a:a1, b:b1, c:[1,2,3,4], d:d1}, x2)
({a:a1, b:b1, c:[1,2,3,4], d:d1}, x3)
({a:a1, b:b1, c:[1,2,3,4], d:d1}, x4)
The join is the following:
rdd2_tuple = rdd2.map(lambda line: line.items())
joined_rdd = flat_rdd1.join(rdd2_tuple).map(lambda x: x[1])
Finally, all you need to do is a groupByKey to get ({a:a1, b:b1, c:[1,2,3,4], d:d1}, [x1, x2, x3, x4]):
result = joined_rdd.groupByKey()