2

I've got a 100,000+ long list of English words in plain text. I want to use split() to convert the list into an array, which I can then convert to an associative array, giving each list item a key equal to its own name, so I can very efficiently check whether or not a string is an English word.

Here's the problem:

The list is new-line separated.

aa
aah
aahed
aahing
aahs
aal
aalii
aaliis
aals

This means that var list = ' <copy/paste list> ' isn't going to work, because JavaScript quotes don't work multi-line.

Is there any way for me to work with this 100,000 item new-line separated string?

  • 2
    Are you working in the browser, in nodejs, or some other environment? My initial thought is just to read it in from a file rather than as a string literal, but the details of how to do that vary depending on the environment you're in. – Chris Tavares Jul 22 '14 at 22:58
  • Generate the correct array literal from the start. You might find this useful: http://stackoverflow.com/q/4833480/139010 – Matt Ball Jul 22 '14 at 22:58
  • @ChrisTavares I can use either. –  Jul 22 '14 at 22:59
  • Why do you think splitting into an array, then converting to an object is more efficient than say [*String.prototype.indexOf*](http://ecma-international.org/ecma-262/5.1/#sec-15.5.4.7) on the original string? – RobG Jul 22 '14 at 23:03
  • Look at what other people have done with scrabble dictionaries. http://ejohn.org/blog/dictionary-lookups-in-javascript/ – epascarello Jul 22 '14 at 23:10

7 Answers7

5

replace the newlines with commas in any texteditor before copying to your js file

momo
  • 3,885
  • 4
  • 33
  • 54
3

One workaround would be to use paste the list into notepad++. Then select all and Edit>Line Operations>Join lines.

This removes new lines and replaces them with spaces.

Kade Keith
  • 208
  • 1
  • 11
3

If you're doing this client side, you can use jQuery's get function to get the words from a text file and do the processing there:

jQuery.get('wordlist.txt', function(results){
    //Do your processing on results here
});

If you're doing this in Node.js, follow the guide here to see how to read a file into memory.

ahjohnston25
  • 1,915
  • 15
  • 36
3

You can use notepad++ or any semi-advanced text editor.

  1. Go to notepad++ and push Ctrl+H to bring up the Replace dialog.

  2. Towards the bottom, select the "Extended" Search Mode

  3. You want to find "\r\n" and replace it with ", "

This will remove the newlines and replace it with commas

Bijan
  • 7,737
  • 18
  • 89
  • 149
3

jsfiddle Demo

Addressing this purely from having a string and trying to work with it in JavaScript through copy paste. Specifically the issues regarding, "This means that var list = ' ' isn't going to work, because JavaScript quotes don't work multi-line.", and "Is there any way for me to work with this 100,000 item new-line separated string?".

You can treat the string like a string in a comment in JavaScript . Although counter-intuitive, this is an interesting approach. Here is the main function

function convertComment(c) {
 return c.toString().
  replace(/^[^\/]+\/\*!?/, '').
  replace(/\*\/[^\/]+$/, '');
}

It can be used in your situation as follows:

var s = convertComment(function() {
 /*

 aa
 aah
 aahed
 aahing
 aahs
 aal
 aalii
 aaliis
 aals

 */
});

At which point you may do whatever you like with s. The demo simply places it into a div for displaying.

jsFiddle Demo

Further, here is an example of taking the list of words, getting them into an array, and then referencing a single word in the array.

//previously shown code
var all = s.match(/[^\r\n]+/g);
var rand = parseInt(Math.random() * all.length);

document.getElementById("random").innerHTML = "Random index #"+rand+": "+all[rand];
Travis J
  • 81,153
  • 41
  • 202
  • 273
2

For variety, another solution is to put the unaltered content into

  1. A data attribute - HTML attributes can contain newlines
  2. or a "non-script" script - eg. <SCRIPT TYPE="text/x-wordlist">
  3. or an HTML comment node
  4. or another hidden element that allows content

Then the content could be read and split/parsed. Since this would be done outside of JavaScript's string literal parsing it doesn't have the issue regarding embedded newlines.

user2864740
  • 60,010
  • 15
  • 145
  • 220
2

If the words are in a separate file, you can load them directly into the page and go from there. I've used a script element with a MIME type that should mean browsers ignore the content (provided it's in the head):

<script type="text/plain" id="wordlist">
aa
aah
aahed
aahing
aahs
aal
aalii
aaliis
aals
</script>

<script>

var words = (function() {
  var words = '\n' + document.getElementById('wordlist').textContent + '\n';
  return {
    checkWord: function (word) {
      return words.indexOf('\n' + word + '\n') != -1;
    }
  }
}());

console.log(words.checkWord('aaliis')); // true
console.log(words.checkWord('ahh'));    // false

</script>

The result is an object with one method, checkWord, that has access to the word list in a closure. You could add more methods like addWord or addVariant, whatever.

Note that textContent may not be supported in all browsers, you may need to feature detect and use innerText or an alternative for some.

RobG
  • 142,382
  • 31
  • 172
  • 209