How to print comma seperated output? - machine-learning

from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators = 200)
model.fit(x_train, y_train)
y_pred = model.predict(x_test)
preds=le.inverse_transform(y_pred)
datatocsv=pd.DataFrame({'id':range(1,len(preds)+1),'taste':preds})
datatocsv.to_csv('prediction.csv',index=False)
Suppose I have saved the prediction output in a csv file named prediction.csv
and this csv file has two columns 'id' and 'taste' column
But I want to print the prediction output in the format specified in the image.
Please guide

X = pd.DataFrame(['taste', 'good', 'normal'])
X
Out[12]:
0
0 taste
1 good
2 normal
X.to_csv('predicition.csv', sep=',')
datatocsv.to_csv('prediction.csv')
This leads to:

Related

ValueError: X has 5 features, but RandomForestClassifier is expecting 2607 features as input

This is how i am converting text to count vector.
cv1 = CountVectorizer()
x_traincv=cv1.fit_transform(x_train)
a = x_traincv.toarray()
a
this the model using for predict.
from sklearn.ensemble import RandomForestClassifier as RFC
rfc_b = RFC()
rfc_b.fit(a,y_train)
y_pred = rfc_b.predict(a)
this is how i am using the live details to predict
from sklearn.feature_extraction.text import CountVectorizer
document = ["Single Hargrave France Female Graduation",]
# Create a Vectorizer Object
vectorizer = CountVectorizer()
vectorizer.fit(document)
print("Vocabulary: ", vectorizer.vocabulary_)
vector = vectorizer.transform(document)
print("Encoded Document is:")
print(vector.toarray())
I AM NOW USING THE MODEL TO PREDICT.
rfc_b.predict(vector)
THE ERROR I AM GETTING
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-62-7cc301d916e6> in <module>()
----> 1 rfc_b.predict(vector)
4 frames
/usr/local/lib/python3.7/dist-packages/sklearn/base.py in _check_n_features(self, X, reset)
399 if n_features != self.n_features_in_:
400 raise ValueError(
--> 401 f"X has {n_features} features, but {self.__class__.__name__} "
402 f"is expecting {self.n_features_in_} features as input."
403 )
ValueError: X has 5 features, but RandomForestClassifier is expecting 2607 features as input.
IT IS WORKING FINE WHEN WORKING WITH TEST SET, DID GET THE OUTPUT.
from sklearn.metrics import accuracy_score
print('Train accuracy score:',accuracy_score(y_train,y_pred))
print('Test accuracy score:', accuracy_score(y_test,rfc_b.predict(b)))
Train accuracy score: 0.987375
Test accuracy score: 0.773
BUT NOT WHEN I USE THE ABOUVE TO INPUT A SINGLE INPUT TO CHECK THE OUTPUT
You have to store your vectorizer used during training, and just call .transform on it, if you create a new one you lose meaning of dimensions used during training, and in particular - you lack many of them, but your vectorizer has no idea about this (as it only has access to your one sample).
cv1 = CountVectorizer()
x_traincv=cv1.fit_transform(x_train)
a = x_traincv.toarray()
from sklearn.ensemble import RandomForestClassifier as RFC
rfc_b = RFC()
rfc_b.fit(a,y_train)
y_pred = rfc_b.predict(a)
document = ["Single Hargrave France Female Graduation",]
vector = cv1.transform(document)
print("Encoded Document is:")
print(vector.toarray())
rfc_b.predict(vector)

How to do a single value prediction in NLP

My dataset was restaurants review with two columns review and liked.
Based on the review it shows if they liked the restaurant or not
I cleaned up the data in NLP as the first step.Then as second step used bag of words model as below.
from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer(max_features = 1500)
X = cv.fit_transform(corpus).toarray()
y = dataset.iloc[:, 1].values
This above gave X as 1500 columns with 0 and 1 with 1000 rows according to my dataset.
I predicted as below
y_pred = classifier.predict(X_test)
So now I have review as "Food was good",how do I predict if they like it or not.A single value to predict.
Please can you help me out.Please let me know if additional information is required.
Thanks
All you need is to apply cv.transform first just like so:
>>> test = ['Food was good']
>>> test_vec = cv.transform(test)
>>> classifier.predict(test_vec)
# returns predicted class
For training and testing here is simple example:
Training:
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
text = ["This is good place","Hyatt is awesome hotel"]
count_vect = CountVectorizer()
tfidf_transformer = TfidfTransformer()
X_train_counts = count_vect.fit_transform(text)
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
pd.DataFrame(X_train_tfidf.todense(), columns = count_vect.get_feature_names())
# Now apply any classification u want to on top of this data-set
Now Testing:
Note: use the same transformation as done in training:
new = ["I like the ambiance of this hotel "]
pd.DataFrame(tfidf_transformer.transform(count_vect.transform(new)).todense(),
columns = count_vect.get_feature_names())
Apply model.predict on top of this now.
you can also use sklearn pipeline.
from sklearn.pipeline import Pipeline
model_pipeline = Pipeline([('vect', CountVectorizer()),('tfidf', TfidfTransformer()), ('model', classifier())]) #call the Model which you want to use
model_pipeline.fit_transform(x,y) # here x is your text data, and y is going to be your target
model_pipeline.predict(['Food was good"']) # predict your new sentence

Found input variables with inconsistent numbers of samples: [2, 144]

I am having a training data set consisting of 144 feedback with 72 positive and 72 negative respectively. there are two target labels positive and negative respectively. Consider the following code segment :
import pandas as pd
feedback_data = pd.read_csv('output.csv')
print(feedback_data)
data target
0 facilitates good student teacher communication. positive
1 lectures are very lengthy. negative
2 the teacher is very good at interaction. positive
3 good at clearing the concepts. positive
4 good at clearing the concepts. positive
5 good at teaching. positive
6 does not shows test copies. negative
7 good subjective knowledge. positive
from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer(binary = True)
cv.fit(feedback_data)
X = cv.transform(feedback_data)
X_test = cv.transform(feedback_data_test)
from sklearn import svm
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
target = [1 if i<72 else 0 for i in range(144)]
# the below line gives error
X_train, X_val, y_train, y_val = train_test_split(X, target, train_size = 0.50)
I do not understand what the problem is. Please help.
You are not using the count vectorizer right. This what you have now:
from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer(binary = True)
cv.fit(df)
X = cv.transform(df)
X
<2x2 sparse matrix of type '<class 'numpy.int64'>'
with 2 stored elements in Compressed Sparse Row format>
So you see that you don't achieve what you want. you do not transform each line correctly. You don't even train the count vectorizer right because you use the entire DataFrame and not just the corpus of comments.
To solve the issue we need to make sure that the Count is well done:
if you do this (Use the right corpus):
cv = CountVectorizer(binary = True)
cv.fit(df['data'].values)
X = cv.transform(df)
X
<2x23 sparse matrix of type '<class 'numpy.int64'>'
with 0 stored elements in Compressed Sparse Row format>
you see that we are coming close to what we want. We just have to transform it right (transform each line):
cv = CountVectorizer(binary = True)
cv.fit(df['data'].values)
X = df['data'].apply(lambda x: cv.transform([x])).values
X
array([<1x23 sparse matrix of type '<class 'numpy.int64'>'
with 5 stored elements in Compressed Sparse Row format>,
...
<1x23 sparse matrix of type '<class 'numpy.int64'>'
with 3 stored elements in Compressed Sparse Row format>], dtype=object)
We have a more suitable X! Now we just need to check if we can split:
target = [1 if i<72 else 0 for i in range(8)] # The dataset is here of size 8
# the below line gives error
X_train, X_val, y_train, y_val = train_test_split(X, target, train_size = 0.50)
And it works!
You need to be sure you understand what CountVectorizer do to use it the right way

Random Forest Regression not give 0 or 1

I'm currently using RandomForestRegression for Titanic(Kaggle).
%%timeit
model = RandomForestRegressor(n_estimators=200, oob_score=False,n_jobs=1,random_state=42)
model.fit(X,y)
#y_oob = model.oob_prediction_
#print("c-stat:", roc_auc_score(y,model.oob_prediction_))
prediction_regression = model.predict(X_test)
# dataframe with predictions
kaggle = pd.DataFrame({'PassengerId': passengerId, 'Survived': prediction_regression})
# save to csv
kaggle.to_csv('./csvToday/prediction_regression.csv', index=False)
but it returns not 0 or 1 . it gives decimal points
892: 0.3163
893: 0.07 such and such
How to make RandomForestRegression return as 0 or 1
Regression is a machine learning problem of predicting quantity/amount/price (such as market stock prediction, home price prediction, e.t.c). As far, as I remember, the goal of titanic competition is to predict whether a passenger survive. It's sounds like a binary classification problem. If it's a classification problem you should use RandomForestClassifier (docs).
So your code would look like:
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(
#some parameters
)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
submit_df = pd.DataFrame({'PassengerId': passengerId, 'Survived': y_pred})
submit_df.to_csv('./csvToday/submission.csv', index=False)
This kernel can provide you with some more insights.

Keras LSTM input features and incorrect dimensional data input

So I'm trying to practice how to use LSTMs in Keras and all parameter (samples, timesteps, features). 3D list is confusing me.
So I have some stock data and if the next item in the list is above the threshold of 5 which is +-2.50 it buys OR sells, if it is in the middle of that threshold it holds, these are my labels: my Y.
For my features my X I have a dataframe of [500, 1, 3] for my 500 samples and each timestep is 1 since each data is 1 hour increment and 3 for 3 features. But I get this error:
ValueError: Error when checking model input: expected lstm_1_input to have 3 dimensions, but got array with shape (500, 3)
How can I fix this code and what am I doing wrong?
import json
import pandas as pd
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
"""
Sample of JSON file
{"time":"2017-01-02T01:56:14.000Z","usd":8.14},
{"time":"2017-01-02T02:56:14.000Z","usd":8.16},
{"time":"2017-01-02T03:56:15.000Z","usd":8.14},
{"time":"2017-01-02T04:56:16.000Z","usd":8.15}
"""
file = open("E.json", "r", encoding="utf8")
file = json.load(file)
"""
If the price jump of the next item is > or < +-2.50 the append 'Buy or 'Sell'
If its in the range of +- 2.50 then append 'Hold'
This si my classifier labels
"""
data = []
for row in range(len(file['data'])):
row2 = row + 1
if row2 == len(file['data']):
break
else:
difference = file['data'][row]['usd'] - file['data'][row2]['usd']
if difference > 2.50:
data.append((file['data'][row]['usd'], 'SELL'))
elif difference < -2.50:
data.append((file['data'][row]['usd'], 'BUY'))
else:
data.append((file['data'][row]['usd'], 'HOLD'))
"""
add the price the time step which si 1 and the features which is 3
"""
frame = pd.DataFrame(data)
features = pd.DataFrame()
# train LSTM
for x in range(500):
series = pd.Series(data=[500, 1, frame.iloc[x][0]])
features = features.append(series, ignore_index=True)
labels = frame.iloc[16000:16500][1]
# test
#yt = frame.iloc[16500:16512][0]
#xt = pd.get_dummies(frame.iloc[16500:16512][1])
# create LSTM
model = Sequential()
model.add(LSTM(3, input_shape=features.shape, activation='relu', return_sequences=False))
model.add(Dense(2, activation='relu'))
model.add(Dense(1, activation='relu'))
model.compile(loss='mse', optimizer='adam', metrics=['accuracy'])
model.fit(x=features.as_matrix(), y=labels.as_matrix())
"""
ERROR
Anaconda3\envs\Final\python.exe C:/Users/Def/PycharmProjects/Ether/Main.py
Using Theano backend.
Traceback (most recent call last):
File "C:/Users/Def/PycharmProjects/Ether/Main.py", line 62, in <module>
model.fit(x=features.as_matrix(), y=labels.as_matrix())
File "\Anaconda3\envs\Final\lib\site-packages\keras\models.py", line 845, in fit
initial_epoch=initial_epoch)
File "\Anaconda3\envs\Final\lib\site-packages\keras\engine\training.py", line 1405, in fit
batch_size=batch_size)
File "\Anaconda3\envs\Final\lib\site-packages\keras\engine\training.py", line 1295, in _standardize_user_data
exception_prefix='model input')
File "\Anaconda3\envs\Final\lib\site-packages\keras\engine\training.py", line 121, in _standardize_input_data
str(array.shape))
ValueError: Error when checking model input: expected lstm_1_input to have 3 dimensions, but got array with shape (500, 3)
"""
Thanks.
This is my first post here I wish that could be useful I will try to do my best
First you need to create 3 dimension array to work with input_shape in keras you can watch this in keras documentation or in a better way:
from keras.models import Sequential
Sequential?
Linear stack of layers.
Arguments
layers: list of layers to add to the model.
# Note
The first layer passed to a Sequential model
should have a defined input shape. What that
means is that it should have received an input_shape
or batch_input_shape argument,
or for some type of layers (recurrent, Dense...)
an input_dim argument.
Example
```python
model = Sequential()
# first layer must have a defined input shape
model.add(Dense(32, input_dim=500))
# afterwards, Keras does automatic shape inference
model.add(Dense(32))
# also possible (equivalent to the above):
model = Sequential()
model.add(Dense(32, input_shape=(500,)))
model.add(Dense(32))
# also possible (equivalent to the above):
model = Sequential()
# here the batch dimension is None,
# which means any batch size will be accepted by the model.
model.add(Dense(32, batch_input_shape=(None, 500)))
model.add(Dense(32))
After that how to transform arrays 2 dimensions in 3 dimmension
check np.newaxis
Useful commands that help you more than you expect:
Sequential?,
-Sequential??,
-print(list(dir(Sequential)))
Best

Resources