How do I create an array from a txt file converted by a HTML file (Ruby)?

Question

I'm trying to complete the first task to our assignment:

Get 5 regular emails and 5 advance-­‐fee fraud emails (aka spam). Convert them all into text files and then turn each into an array of words (split may help here). Then use a bunch of regular expressions to search the array of words looking for keywords to classify which files are spam or not. If you want to get fancy you could give each array a spam-­‐score out of 10.

1. Open HTML page and read file.
2. Strip script, links etc from file.
3. Have body/para on its own.
4. Open text file (file2) & write to it (UTF-8).
5. Pass content from HTML document (file 1).
6. Now put the words from text file (file2) into an array and later split.
7. Go through array finding any words that are considered spam and print message to screen stating if the email is a spam or not.

Here is my code:

require 'nokogiri'
file = File.open("EMAILS/REG/Membership.htm", "r")
doc = Nokogiri::HTML(file)
#What ever is passed from elements to the newFile is being put into the new array however the euro sign doesn't appear correctly
elements = doc.xpath("/html/body//p").text
#puts elements

newFile = File.open("test1.txt", "w")
newFile.write(elements)
newFile.close()


#I want to open the file again and print the lines to the screen
#
array_of_words = {}
puts "\n\tRetrieving test1.txt...\n\n"
File.open("test1.txt", "r:UTF-8").each_line do |line|
    words = line.split(' ')
    words.each do |word|
        puts "#{word}"
        #array_of_words[word] = gets.chomp.split(' ')
    end
end

EDITED: Here I've edited the file, however, I'm unable to retrieve the UTF-8 encoding of the euro sign in the array (see the image).

require 'nokogiri'

doc = Nokogiri::HTML(File.open("EMAILS/REG/Membership.htm", "r:UTF-8"))

#What ever is passed from elements to the newFile is being put into the new 
#array however the euro sign doesn't appear correctly
elements = doc.xpath("//p").text
#puts elements

File.write("test1.txt", elements)

puts "\n\tRetrieving test1.txt...\n\n"

#I want to open the file again and print the lines to the screen
#
word_array = Array.new
File.read("test1.txt").each_line do |line|
    line.split(' ').each do |word|
        puts "#{word}"
        word_array << word
    end
end

Answer 1

You're making things harder for yourself. You already have the paragraph text in elements so there's no need to read test1.txt after writing to it. Then use String#split without arguments to split on all whitespace.

Answer 2

Because this is an assignment, I'm not going to try to answer how you're supposed to do this; You're supposed to figure it out on your own.

What I will do is show you how you should have written what you've already done, and point you in a direction:

require 'nokogiri'

doc = Nokogiri::HTML(File.read("EMAILS/REG/Membership.htm"))

# What ever is passed from elements to the newFile is being put into the new
# array however the euro sign doesn't appear correctly
elements = doc.xpath("//p").text

File.write("test1.txt", elements)

print "\n\tRetrieving test1.txt...\n\n"

# I want to open the file again and print the lines to the screen
word_hash = {}
File.open("test1.txt", "r:UTF-8").each_line do |line|
  line.split(' ').each do |word|
    puts "#{word}"
    #word_hash[word] = gets.chomp.split(' ')
  end
end

Many of Ruby's IO methods, and File's by inheritance, can take advantage of blocks, which automatically close the stream when the block exits. Use that capability as leaving files open throughout the run-time of an app is not good.

array_of_words = {} doesn't define an array, it's a hash.

#array_of_words[word] = gets.chomp.split(' ') wouldn't work because of where gets wants to read from. By default it's STDIN, which would be the console, meaning the keyboard. You've already got word at that point so do something with it.

But think, you're basically creating the basis for a Bayesian Filter. You need to be counting the number of occurrences of words, so merely assigning the word to the hash won't get you what you want to know, you need to know how many times a particular word was seen. Stack Overflow has a lot of questions answered about how to count the number of words found in a string, so search for those.