I am using Tensorflow 1.14 along with Python 3.7. I have text files that contain conversations and a pickle file that contains words for the conversations.
The training program uses the text files and the pickle file to create a conversational chatbot. This is what I do to open and then build a vocabulary...
unique_words = pickle.load(open('aug_unique_words.p', 'rb')) # The method I use for opening the file
word2idx, idx2word, vocab_size = build_vocabs(unique_words)
print(vocab_size, word2idx["<START>"], word2idx["<PAD>"], word2idx["<EOS>"], word2idx["<unk>"])
Once the program begins to run, I encounter this in my run terminal...
Traceback (most recent call last):
File "C:/Users/ex/PycharmProjects/ConversationML/Conversational-Chatbot/conversationaltraining.py", line 50, in <module>
unique_words = pickle.load(open('aug_unique_words.p', 'rb'))
_pickle.UnpicklingError: invalid load key, '\xef'.
150946
['how', 'do', 'birds', 'fly', '?', '<EOS>']
['<START>', 'i', 'do', "n't", 'know', 'the', 'details', 'as', 'i', 'have', 'not', 'learned', 'aerodynamics', '.', 'but', 'i', 'think', 'it', "'s", 'the', 'air', 'that', 'is', 'pushing', 'the', 'bird', 'up', 'and', 'fly', '<EOS>']
['<START>', 'i', 'do', "n't", 'know', 'the', 'details', 'as', 'i', 'have', 'not', 'learned', 'aerodynamics', '.', 'but', 'i', 'think', 'it', "'s", 'the', 'air', 'that', 'is', 'pushing', 'the', 'bird', 'up', 'and', 'fly', '<EOS>']
Process finished with exit code 1
The results of the words vary but it is irrelevant here as the error _pickle.UnpicklingError: invalid load key, '\xef' stays as a constant. I have been looking into different ways of opening the file such as Pandas package:
unique_words = pd.read_pickle(r'myfilepath')
I have still encountered this error and continued to look into it. Any advice would be greatly appreciated.
This is different from the suggested question because the error is not caused by how the pickle file is opened. I have tried numerous methods.
