I am trying to calculate edit distances of a string against a collection to find the closest match. My current problem is that the collection is very large (about 25000 items), so I had to narrow down the set to just strings of similar lengths but that still would only narrow it down to a few thousand strings and this still is very slow. Is there a datastructure that allows for a quick lookup of similar strings or is there another way I could address this problem?
Asked
Active
Viewed 727 times
5
-
How are you doing it right now? Can you show some code? – juergen d Feb 04 '12 at 08:12
-
By similar I mean comparing words that are common spelling mistakes such as "exanple" and "example" or "weird" and "wierd". – Lezan Feb 04 '12 at 09:01
-
1Looks like you want an implementation of the levenstein distance: http://stackoverflow.com/questions/6087281/similarity-score-levenshtein. – Kurt Du Bois Feb 04 '12 at 09:05
-
I am currently doing it the following way: String currentString; List distanceList; (for word: wordList){ int distance = calculateDistance(currentString,word) distanceList.add(distance) } – Lezan Feb 04 '12 at 09:07
3 Answers
8
Sounds like a BK-tree might be what you want. Here's an article discussing them: http://blog.notdot.net/2007/4/Damn-Cool-Algorithms-Part-1-BK-Trees. A quick Google yields some Java implementations.
SimonC
- 6,590
- 1
- 23
- 40
-
-
Yup that did it, needed a different implementation of the search but it was perfect! Thank you!! – Lezan Feb 05 '12 at 11:16
6
Levenshtein Automata allow for fast selection of a set of words from a large dictionary such that they are within the given Levenshtein distance from a given word.
See: Schulz K, Mihov S. (2002) Fast String Correction with Levenshtein-Automata.
kkm inactive - support strike
- 5,190
- 2
- 32
- 59
2
If your criteria for 'similar' define a total ordering, you should be able to define a Comparator and use a TreeSet to find the closest matches (eg using the ceiling and floor methods).
Mark Rotteveel
- 100,966
- 191
- 140
- 197