How to populate a large datalist (~2000 items) from a dictionary

Question

Right now I'm using the following code but it takes ~10 seconds on Chrome and ~2 minutes on IE11, which is where its mostly going to end up being used.

for (var key in dict) {
    if (dict.hasOwnProperty(key)) {
        combo.innerHTML += "<option value=\"" + dict[key] + "\">" + key + "</option>";
    }
}

I was reading this tutorial: http://blog.teamtreehouse.com/creating-autocomplete-dropdowns-datalist-element which suggested using ajax like so when dealing with larger quantities, though I'm not sure if large refers to 100 items or 100,000 items.

var request = new XMLHttpRequest();

request.onreadystatechange = function(response) {
  if (request.readyState === 4) {
    if (request.status === 200) {

      var jsonOptions = JSON.parse(request.responseText);

      jsonOptions.forEach(function(item) {

        var option = document.createElement('option');
        option.value = item;
        dataList.appendChild(option);

      });

    } else {
      console.log("Failed to load datalist options");
    }
  }
};

request.open('GET', 'html-elements.json', true);
request.send();

I've been trying to get this to work for a dictionary by replacing request.responseText with JSON.parse(JSON.stringify(dict)); but I'm running into problems getting it to make the request to begin with because it's not in a file.

How should I do this? And if I shouldn't be using a DataList for this, what alternative do you recommend?

Thanks in advance.

Maybe http://stackoverflow.com/questions/12603567/pass-array-to-typeahead-bootstrap or http://tatiyants.com/how-to-use-json-objects-with-twitter-bootstrap-typeahead/ helps. It's another approach, they use a typeahead instead of a select box. — Reeno, May 19 '15 at 21:33
One major issue with your code is that you're touching the DOM ~2000 times. See http://stackoverflow.com/a/1683041/1682509 for another solution which touches the DOM only once (with jQuery, but can be easily rewritten to vanilla JS) — Reeno, May 19 '15 at 21:36
If you have a large datalist, another option is dynamically inserting datalist items as you're typing. You can see this in the emoji selector on [Chit Chat](https://chitchat.tohodo.com/lobby) -- enter a colon (:), then search for an emoji; you can see the `` being updated dynamically. — thdoan, May 25 '23 at 15:25

score 7 · Accepted Answer · answered May 19 '15 at 21:39

7

One area in which you could speed up performance is with a document fragment as writing to the DOM is slow.

var frag = document.createDocumentFragment();

for (var key in dict) {
    if (dict.hasOwnProperty(key)) {
        var option = document.createElement("OPTION");
        option.textContent = key;
        option.value = dict[key];
        frag.appendChild(option);
    }
}

combo.appendChild(frag);

answered May 19 '15 at 21:39

znap026

429
1
5
15

1

Thanks, that's quite a bit faster. I tried appending to see if that improved the performance before, but I was appending an option to the combo each iteration, so it was equally bad. I've never used `documentFragment` before. Good to know. – Charles Clayton May 19 '15 at 21:52

score 2 · Answer 2 · answered May 19 '15 at 21:49

On instant way to get better performance is to build the HTML string up first, then assign it to the innerHTML.

var htmlStr = '';
for (var key in dict) {
    if (dict.hasOwnProperty(key)) {
        htmlStr += "<option value=\"" + dict[key] + "\">" + key + "</option>";
    }
}
combo.innerHTML = htmlStr;

The difference is huge: http://jsperf.com/string-append-vs-dom

score 1 · Answer 3 · edited Dec 07 '21 at 18:39

1

The DOM is notoriously slow. You could try filtering manually and only showing the first X elements. As znap026 pointed out, using document fragments will also help speed things up.

"use strict";

var data = Object.getOwnPropertyNames(window).sort(),
  datalist = document.getElementById("datalist"),
  input = document.getElementById("input");
  
const processedData = Object.fromEntries(data.map(d => [d.toLowerCase(), d]));

function search() {
  var term = input.value.toLowerCase();
  var found = 0;
  var frag = document.createDocumentFragment();

  for (var child of [].slice.apply(datalist.childNodes)) {
    datalist.removeChild(child);
  }

  for (var searchable in processedData) {
    if (searchable.indexOf(term) === 0) {
      let item = processedData[searchable];
      let option = document.createElement("option");
      option.value = item

      frag.appendChild(option);
      if (++found > 10) break;
    }
  }

  datalist.appendChild(frag);
}

search();
input.addEventListener("input", search);

<input id="input" list="datalist" placeholder="window properties"/>
<datalist id="datalist"></datalist>

edited Dec 07 '21 at 18:39

trinalbadger587

1,905
1
18
36

answered May 19 '15 at 22:27

redbmk

4,687
3
25
49

Very clever use of large datasets! – Edyn Sep 17 '16 at 15:10
To best understand, if this is truly faster then it is also saying that JavaScript array filtering is faster then native C code (Chrome) doing the same filtering. I would have bet the farm that plopping the entire set into the datalist would allow Chrome to perform the same filtering but via optimized machine code instead of JIT JavaScript runtime code. As is above there are now two filter loops: one by the JS on the input change to mutate the DOM and the second by the browser also filtering the now filtered set. – Sukima Feb 21 '20 at 18:40
1

@Sukima - I believe it's *slightly* different than you describe. While the native C code may be faster than JavaScript at array filtering, it may not get to even *start* filtering until after a few thousand DOM nodes are created. Also, modern JavaScript engines are able to optimize certain "hot" functions... so even the difference in filtering performance may be negligible. I'd definitely recommend running some performance tests before betting that farm. ***All your chickens are belong to us!*** – Aaron Cicali Mar 22 '21 at 19:52

score 1 · Answer 4 · answered Dec 23 '21 at 10:56

You can also improve performance by detaching the datalist from then input and then attaching it again after populating it:

"use strict";

// init
const values = Object.getOwnPropertyNames(window);
const datalist = document.getElementById("datalist");
const arr = values.map(k => new Option(k));

let start = performance.now();
datalist.replaceChildren(...arr);
console.log('plain replace:', performance.now() - start);

// reset
datalist.replaceChildren(...[]);

start = performance.now();
datalist.setAttribute("id", undefined);
datalist.replaceChildren(...arr);
datalist.setAttribute("id", "datalist");
console.log('replace with detach and attach:', performance.now() - start);

<input list="datalist" />
<datalist id="datalist"></datalist>

How to populate a large datalist (~2000 items) from a dictionary

4 Answers4

Linked