1

this is my second post. I am really sorry If I sound awkward. I am new to Machine Learning. Reporting the question or giving a negative point will help me. Again I am sorry for unable to clear my question.

Now coming to my question, I am working on an assignment on the next word Prediction. I am trying to create a model that can be used to generate the next words based on the input like the swift keyboard. I have created a model that is able to generate the next word. But I want to generate more than one word for one input word. For example, I ate hot ___. For the blank space, I want Predictions like dog, pizza, chocolate. However, my current model is able to generate the next word which is having the maximum probability. But I want 3 outputs. I am using LSTM and Keras as a framework. I am using Keras's sequential model. There are three main parts of the code: dataset preparation, model training, and generating the prediction.

def dataset_preparation():

    # basic cleanup
    corpus = text.split("\n\n")

    # tokenization  
    tokenizer.fit_on_texts(str(corpus).split("##"))
    total_words = len(tokenizer.word_index) + 1

    # create input sequences using list of tokens
    input_sequences = []
    for line in str(corpus).split("\n\n"):
        #print(line)
        token_list = tokenizer.texts_to_sequences([line])[0]
        #print("printing token_list",token_list)
        for i in range(1, len(token_list)):
            n_gram_sequence = token_list[:i+1]
            #print("printing n_gram_sequence",n_gram_sequence)
            #print("printing n_gram_sequence length",len(n_gram_sequence))
            input_sequences.append(n_gram_sequence)
    #print("Printing Input Sequence:",input_sequences)
    # pad sequences 
    max_sequence_len = 378
    input_sequences = np.array(pad_sequences(input_sequences, maxlen=max_sequence_len, padding='pre'))

    # create predictors and label
    predictors, label = input_sequences[:,:-1],input_sequences[:,-1]
    label = ku.to_categorical(label, num_classes=total_words)

    return predictors, label, max_sequence_len, total_words

def create_model(predictors, label, max_sequence_len, total_words):

    model = Sequential()
    model.add(Embedding(total_words, 10, input_length=max_sequence_len-1))
    model.add(LSTM(150, return_sequences = True))
    # model.add(Dropout(0.2))
    model.add(LSTM(100))
    model.add(Dense(total_words, activation='softmax'))

    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    earlystop = EarlyStopping(monitor='loss', min_delta=0, patience=5, verbose=0, mode='auto')
    model.fit(predictors, label, epochs=1, verbose=1, callbacks=[earlystop])
    print (model.summary())
    return model

def generate_text(seed_text, next_words, max_sequence_len):
    for _ in range(next_words):
        token_list = tokenizer.texts_to_sequences([seed_text])[0]
        print("Printing token list:",token_list)
        token_list = pad_sequences([token_list], maxlen=max_sequence_len-1, padding='pre')
        predicted = model.predict_classes(token_list, verbose=0)
        output_word = ""
        for word, index in tokenizer.word_index.items():
            if index == predicted:
                output_word = word
                break
        seed_text += " " + output_word
    return seed_text

I am following the mentioned article from medium Thanking you in advance.

  • What specifically are you doing to get the maximum-probability word? If you can show the code you currently have for getting that one word, perhaps people can identify what needs to change to obtain multiple words. – NicholasM Nov 16 '19 at 14:54
  • https://medium.com/coinmonks/word-level-lstm-text-generator-creating-automatic-song-lyrics-with-neural-networks-b8a1617104fb – DKM Nov 16 '19 at 14:58
  • This might help – DKM Nov 16 '19 at 14:58
  • @NicholasM I am following https://medium.com/@shivambansal36/language-modelling-text-generation-using-lstms-deep-learning-for-nlp-ed36b224b275 – srikant kumar Nov 16 '19 at 15:21
  • @srikantkumar, when I clicked that Medium link, a dialog box popped up with a title "Pardon the Interruption," asking me to sign in, after which I closed the page. You should include the important code in this question itself, so it will always be here for future readers. – NicholasM Nov 16 '19 at 15:26
  • @NicholasM sorry, I have updated my question. – srikant kumar Nov 16 '19 at 15:27
  • you should check https://stackoverflow.com/questions/43034960/many-to-one-and-many-to-many-lstm-examples-in-keras – Jainil Patel Nov 16 '19 at 15:59

0 Answers0