6

I encountered a bug where I couldn't match two seemingly 'identical' strings together. For example, the following two strings fail to match: "sample" and "​sample".

To replicate the issue, one can run the following in Clojure.

(= "sample" "​sample") ; returns false

After an hour of frustrated debugging, I discovered that there was a zero-width space at the front of the second string! Removing it from this particular example via a backspace is trivial. However I have a database of strings that I'm matching, and it seems like there are multiple strings facing this issue. My question is: is there a general method to trim zero-width spaces in Clojure?

Some method's I've tried:

(count (clojure.string/trim "​abc")) ; returns 4

(count (clojure.string/replace "​abc" #"\s" "")) ; returns 4

This thread Remove zero-width space characters from a JavaScript string does provide a solution with regular expressions that works in this example, i.e.

(count (clojure.string/replace "​abc" #"[\u200B-\u200D\uFEFF]" "")) ; returns 3

However, as stated in the post itself, there are many other potential ascii characters that may be invisible. So I'm still interested if there's a more general method that doesn't rely on listing all possible invisible unicode symbols.

Alan Thompson
  • 29,276
  • 6
  • 41
  • 48
Desmond Cheong
  • 798
  • 6
  • 19
  • 1
    I'm not aware of a built-in function in Clojure that does what you want. I just read https://en.wikipedia.org/wiki/Zero-width_space to be more familiar with 0 width characters. To solve your issue, unfortunately, I think you should maintain manually a list of characters to trim. – Rozar Fabien Jul 15 '20 at 12:03

2 Answers2

4

I believe, what you are referring to are so-called non-printable characters. Based on this answer in Java, you could pass the #"\p{C}" regular expression as pattern to replace:

(defn remove-non-printable-characters [x]
  (clojure.string/replace x #"\p{C}" ""))

However, this will remove line breaks, e.g. \n. So in order to keep those characters, we need a more complex regular expression:

(defn remove-non-printable-characters [x]
  (clojure.string/replace x #"[\p{C}&&^(\S)]" ""))

This function will remove non-printable characters. Let's test it:

(= "sample" "​sample")
;; => false

(= (remove-non-printable-characters "sample")
   (remove-non-printable-characters "​sample"))
;; => true

(remove-non-printable-characters "sam\nple")
;; => "sam\nple"

The \p{C} pattern is discussed here.

Rulle
  • 4,496
  • 1
  • 15
  • 21
  • 1
    Very nice, thank you :) Also useful to note that this method removes control characters including '\n', but leaves normal spaces ` ` unaffected. – Desmond Cheong Jul 15 '20 at 13:18
  • Ah, thanks for pointing that out! Didn't notice. I will see if I can improve the answer a bit regarding that. – Rulle Jul 15 '20 at 13:20
1

The regex solution from @Rulle is very nice. The tupelo.chars namespace also has a collection of character classes and predicate functions that could be useful. They work in Clojure and ClojureScript, and also include the ^nbsp; for browsers. In particular, check out the visible? predicate.

The tupelo.string namespace also has a number of helper & convenience functions for string processing.

(ns tst.demo.core
  (:use tupelo.core tupelo.test)
  (:require
    [tupelo.chars :as chars]
    [tupelo.string :as str] ))

(def sss
"Some multi-line
string." )

(dotest
  (println "result:")
  (println
    (str/join
      (filterv
        #(or (chars/visible? %) 
             (chars/whitespace? %))
        sss))))

with result

result:
Some multi-line
string.

To use, make your project.clj look like:

  :dependencies [
                 [org.clojure/clojure "1.10.2-alpha1"]
                 [prismatic/schema "1.1.12"]
                 [tupelo "20.07.01"]
                 ]
Alan Thompson
  • 29,276
  • 6
  • 41
  • 48