Pyspark substring with values from another table - join

Question

Pyspark: I have two dataframes. First one is one column containing a long string. Second dataframe is a lookup dataframe holding some values that indicate some substring start and ends. I'd like to use the second data frame to split up the first and have a resultant dataframe with the original data and the string split values:
Dataframe A:
Data
000 456 9b
876 998 1c
Dataframe B:
Description
Start
End
Length
City
1
3
3
Country
5
7
3
IheartSpark
9
10
2
The result would be this:
Data
City
Country
IheartSpark
000 456 9b
000
456
9b
876 998 1c
876
998
1c
Dataframe b is only 30 rows or so and i was thinking of broadcasting this if possible (this will run in a cluster).
Any thoughts?

Answer 1

Try with crossJoin and pivot functions to get the desired output.
Example:
df.show()
#+----------+
#| Data|
#+----------+
#|000 456 9b|
#|876 998 1c|
#+----------+
df1.show()
#+-----------+-----+---+------+
#| Descr|start|end|length|
#+-----------+-----+---+------+
#| City| 1| 3| 3|
#| Country| 5| 7| 3|
#|IheartSpark| 9| 10| 2|
#+-----------+-----+---+------+
from pyspark.sql.functions import *
df.crossJoin(broadcast(df1)).\
withColumn("nn",expr("""substring(Data,start,length)""")).\
groupBy("Data").\
pivot("Descr").\
agg(first(col("nn"))).\
show()
#+----------+----+-------+-----------+
#| Data|City|Country|IheartSpark|
#+----------+----+-------+-----------+
#|000 456 9b| 000| 456| 9b|
#|876 998 1c| 876| 998| 1c|
#+----------+----+-------+-----------+