11

I'm building a website that should collect various news feeds and would like the texts to be compared for similarity. What i need is some sort of a news text similarity algorithm. I know that php has the similar_text function and am not sure how good it is + i need it for javascript. So if anyone could point me to an example or a plugin or any instruction on how this is possible or at least where to look and start investigating.

Vadim Kotov
  • 8,084
  • 8
  • 48
  • 62
Karington
  • 253
  • 1
  • 6
  • 19
  • 4
    Why do you specifically need it in JS? You realize this will have to happen when users visit the site, it wont be something that you could necessarily run a cron job on and save on the server (well not as easily as a serverside language) – Loktar Feb 18 '11 at 15:05
  • 3
    @Loktar - There is serverside javascript too. :) And ofcourse it could be done in (clientside) Javascript as well by just retreiving the feeds and merge them on the client. It will save you a server that supports server side scripting. – GolezTrol Feb 18 '11 at 15:09
  • Yeah but something like this is traditionally done on the server, and would be much faster.. plus you could do it once, cache the results and serve to new users. I doubt the OP was referring to something like NodeJS :P – Loktar Feb 18 '11 at 15:14
  • The point is that there is an admin panel where admins group similar news together. I have quite a lot of options to make it easier for them but i need the use of when they select a title it compares its text to all other news texts and stick's out the most likely to be similar. – Karington Feb 18 '11 at 15:20

2 Answers2

12

There's a javascript implementation of the Levenshtein distance metric, which is often used for text comparisons. If you want to compare whole articles or headlines though you might be better off looking at intersections between the sets of words that make up the text (and frequencies of those words) rather than just string similarity measures.

Flexo
  • 87,323
  • 22
  • 191
  • 272
11

The question whether two texts are similar is a philosophical one as long as you don't specify exactly what it should mean. Consider the Strings "house" and "mouse". Seen from a semantic level they are not very similar, but they are very similar regarding their "physical appearance", because only one letter is different (and in this case you could go by Levenshtein distance).

To decide about similarity you need an appropriate text representation. You could – for instance – extract and count all n-grams and compare the two resulting frequency-vectors using a similarity measure as e.g. cosine similarity. Or you could stem the words to their root form after having removed all stopwords, sum up their occurrences and use this as input for a similarity measure.

There are plenty approaches and papers about that topic, e.g. this one about short texts. In any case: The higher the abstraction level where you want to decide if two texts are similar the more difficult it will get. I think your question is a non-trivial one (and hence my answer rather abstract) ... ;-)

philonous
  • 3,621
  • 3
  • 27
  • 24