Difference between `join` and `union` followed by `groupByKey` in Spark? - join

Question

I cannot find a good reason why:
anRDD.join(anotherRDD)
should be different from:
anRDD.union(anotherRDD).groupByKey()
But, the latter gives me an error and the former doesn't. I can provide an example if absolutely needed, but I'd like to know from the perspective of functional abstraction. No one I've asked can give me a good explanation of this.

Answer 1

Here are some points that I will illustrate with some code below:
join works with two rdds each consists of pairs, and having the same key which need to be matched. The types of values of the two rdds need not be matched. The resulting rdd will alwas have entries of type (Key, (Value1, Value2))
anRDD.union(anotherRDD).groupByKey() will produce an error if anRDD and anotherRDD have different types of values; it will not produce an error if both keys and values have same type. The result will be entries of type (Key, Iterable[Value]) where Iterable need not have length 2 like in the case of join.
Example:
val rdd1 = sc.parallelize(Seq( ("a", 1) , ("b", 1)))
val rdd2 = sc.parallelize(Seq( ("a", 2) , ("b", 2)))
val rdd3 = sc.parallelize(Seq( ("a", 2.0) , ("b", 2.0))) // different Value type
val rdd4 = sc.parallelize(Seq( ("a", 1) , ("b", 1), ("a", 5) , ("b", 5)))
val rdd5 = sc.parallelize(Seq( ("a", 2) , ("b", 2), ("a", 5) , ("b", 5)))
produces the following:
scala> rdd1.join(rdd2)
res18: org.apache.spark.rdd.RDD[(String, (Int, Int))] = MapPartitionsRDD[77] at join at <console>:26
scala> rdd1.union(rdd2).groupByKey
res19: org.apache.spark.rdd.RDD[(String, Iterable[Int])] = ShuffledRDD[79] at groupByKey at <console>:26
scala> rdd1.union(rdd3).groupByKey
<console>:26: error: type mismatch;
found : org.apache.spark.rdd.RDD[(String, Double)]
required: org.apache.spark.rdd.RDD[(String, Int)]
rdd1.union(rdd3).groupByKey
Whereas notice the different result produced if you have repeated keys in your rdds:
scala> rdd4.union(rdd5).groupByKey.collect.mkString("\n")
res21: String =
(a,CompactBuffer(1, 5, 2, 5))
(b,CompactBuffer(1, 5, 2, 5))
scala> rdd4.join(rdd5).collect.mkString("\n")
res22: String =
(a,(1,2))
(a,(1,5))
(a,(5,2))
(a,(5,5))
(b,(1,2))
(b,(1,5))
(b,(5,2))
(b,(5,5))
Edit: OP is using Python, not Scala. There is a difference in type safety between Python and Scala. Scala will catch such things as type mismatch between the two RDDs as illustrated above; Python will not catch it right away, but will produce cryptic errors later on when you try to apply a method on objects of wrong type. And remember, Spark is written in Scala with a Python API.
Indeed, I tried OP code in the comment, and in pyspark, it works with simple actions like count(). However, it would produce an error if you for example try to square each value (which you can do on integers, but not strings)
Here is the data: note I ommited the list, I only have values 1 and 0.
B = [('b',1), ('c',0)]
C = [('b', 'bs'), ('c', 'cs')]
anRDD = sc.parallelize(B)
anotherRDD = sc.parallelize(C)
And here is the output:
>>> anRDD.join(anotherRDD).count()
2
>>> anRDD.union(anotherRDD).groupByKey().count()
2
>>> for y in anRDD.map(lambda (a, x): (a, x*x)).collect():
... print y
...
('b', 1)
('c', 0)
>>> for y in anRDD.union(anotherRDD).map(lambda (a, x): (a, x*x)).collect():
... print y
...
15/12/13 15:18:51 ERROR Executor: Exception in task 5.0 in stage 23.0 (TID 169)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):

Answer 2

Former and latter have different result set:
Former:
(K, V).join(K, W) = (K, (V, W))
The former result is equi-join, SQL analogy:
anRDD.K = anotherRDD.K
Latter:
Not only include equi-join results but also union non-matched part from anRDD and non-matched part anotherRDD.