BeautifulSoup indexes - parsing

Question

so I am trying to parse the links for the genres and subgenres in the IMDB page http://www.imdb.com/genre/?ref_=nv_ch_gr_3
and have now been able to parse the main genre tags into something usable
with the following code
table = soup.find_all("table", {"class": "genre-table"})
for item in table:
for x in range(100):
try:
print(item.contents[x].find_all("h3"))
print(len(item.contents[x].find_all("h3")))
except:
pass
and my output is 11 sets of lists with two tags in it like this
[<h3>Action <span class="normal">»</span></h3>, <h3>Adventure <span class="normal">»</span></h3>]
2
I understand this because the containers have a class of "even" and "odd" with two h3 tags in each container, but I didnt specify it to differentiate between even and odd, actually I think I am answering my own question here, am I right in thinking that because it was in a container class odd or even, that bs4 put it in a list to just show that and its up to me to separate them?
Second more important question:
how would I get each h3 link and title into my dataframe that I have set up as
df = pd.DataFrame(columns= ['Genre', 'Sub-Genre', 'Link'])
I've tried
for y in range(2):
df.append({'Genre':'item.contents[x].find_all("h3"))[y].text)}, ignore_index = true)
This is nested in the for loop with x of course (not on its own)
but it doesnt seem to work
any thoughts? karma your way!

Answer 1

First off there's no need to find all tables since only the first one is neccessary:
table = soup.find("table", {'class': 'genre-table'})
and since every other item is redundant(starting with the first) you can iterate the table like this:
for item in list(table)[1::2]:
after this we can get the 'h3' tags in every row and loop through both of them:
row = item.find_all("h3")
for col in row:
because the text in every 'h3' element returns the genre in this format: 'Somegenre \xc2\xbb' i removed the span element before getting the the text:
col.span.extract()
link = col.a['href']
genre = col.text.strip()
after this just add the elements in to a dataframe by index:
df.loc[len(df)]=[genre, None, link]
full code:
import pandas as pd
import requests
from bs4 import BeautifulSoup
df = pd.DataFrame(columns=['Genre', 'Sub-Genre', 'Link'])
req = requests.get('http://www.imdb.com/genre/?ref_=nv_ch_gr_3')
soup = BeautifulSoup(req.content, 'html.parser')
table = soup.find("table", {'class': 'genre-table'})
for item in list(table)[1::2]:
row = item.find_all("h3")
for col in row:
col.span.extract()
link = col.a['href']
genre = col.text.strip()
df.loc[len(df)] = [genre, None, link]