this is my second post. I am really sorry If I sound awkward. I am new to Machine Learning. Reporting the question or giving a negative point will help me. Again I am sorry for unable to clear my question.
Now coming to my question, I am working on an assignment on the next word Prediction. I am trying to create a model that can be used to generate the next words based on the input like the swift keyboard. I have created a model that is able to generate the next word. But I want to generate more than one word for one input word. For example, I ate hot ___. For the blank space, I want Predictions like dog, pizza, chocolate. However, my current model is able to generate the next word which is having the maximum probability. But I want 3 outputs. I am using LSTM and Keras as a framework. I am using Keras's sequential model. There are three main parts of the code: dataset preparation, model training, and generating the prediction.
def dataset_preparation():
# basic cleanup
corpus = text.split("\n\n")
# tokenization
tokenizer.fit_on_texts(str(corpus).split("##"))
total_words = len(tokenizer.word_index) + 1
# create input sequences using list of tokens
input_sequences = []
for line in str(corpus).split("\n\n"):
#print(line)
token_list = tokenizer.texts_to_sequences([line])[0]
#print("printing token_list",token_list)
for i in range(1, len(token_list)):
n_gram_sequence = token_list[:i+1]
#print("printing n_gram_sequence",n_gram_sequence)
#print("printing n_gram_sequence length",len(n_gram_sequence))
input_sequences.append(n_gram_sequence)
#print("Printing Input Sequence:",input_sequences)
# pad sequences
max_sequence_len = 378
input_sequences = np.array(pad_sequences(input_sequences, maxlen=max_sequence_len, padding='pre'))
# create predictors and label
predictors, label = input_sequences[:,:-1],input_sequences[:,-1]
label = ku.to_categorical(label, num_classes=total_words)
return predictors, label, max_sequence_len, total_words
def create_model(predictors, label, max_sequence_len, total_words):
model = Sequential()
model.add(Embedding(total_words, 10, input_length=max_sequence_len-1))
model.add(LSTM(150, return_sequences = True))
# model.add(Dropout(0.2))
model.add(LSTM(100))
model.add(Dense(total_words, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
earlystop = EarlyStopping(monitor='loss', min_delta=0, patience=5, verbose=0, mode='auto')
model.fit(predictors, label, epochs=1, verbose=1, callbacks=[earlystop])
print (model.summary())
return model
def generate_text(seed_text, next_words, max_sequence_len):
for _ in range(next_words):
token_list = tokenizer.texts_to_sequences([seed_text])[0]
print("Printing token list:",token_list)
token_list = pad_sequences([token_list], maxlen=max_sequence_len-1, padding='pre')
predicted = model.predict_classes(token_list, verbose=0)
output_word = ""
for word, index in tokenizer.word_index.items():
if index == predicted:
output_word = word
break
seed_text += " " + output_word
return seed_text
I am following the mentioned article from medium Thanking you in advance.