how to split datasets into training and test data with sklearn - image-processing

Question

I'm using at&t faces dataset, a main directory of contains 40 sub-directories, each sub-directory contains different images of a specific person. I created a list that contains the sub-directories names. I want to use the data to train a neural network so I want to split the data into 80% training and 20% testing. Here is what I have done so far :
import os
import cv2
path = r"C:\Users\Desktop\att_faces"
directory = []
directory = [x[1] for x in os.walk(path)]
non_empty_dirs = [x for x in directory if x]
directory = [item for subitem in non_empty_dirs for item in subitem]
How should I proceed after this step?

Answer 1

You want to split your data in to train and test sets. For that, you can either
Manually or using a script separate train and test to folders and load them to train with the help of a Data Generator.
Load whole data and split them to train and test in memory.
Let's discuss the second option.
main directory contains 40 sub-directories
Let assume your main directory is Train// and there are 40 subfolders namely 1-40. Also, I assume class label is the folder name.
# imports
import cv2
import numpy as np
import os
from sklearn.model_selection import train_test_split
# seed for reproducibility
SEED = 44000
# lists to store data
data = []
label = []
# folder where data is placed
BASE_FOLDER = 'Train//'
folders = os.listdir(BASE_FOLDER)
# loading data to lists
for folder in folders:
for file in os.listdir(BASE_FOLDER + folder + '//'):
img = cv2.imread(BASE_FOLDER + folder + '//' + file)
# do any pre-processing if needed like resize, sharpen etc.
data = data.append(img)
label = label.append(folder)
# now split the data in to train and test with the help of train_test_split
train_data, test_data, train_label, test_label = train_test_split(data, label, test_size=0.2, random_state=SEED)