How to generative test function-under-test with must-to-test distinct edge cases for each execution?

Question

Hello Clojure experts..!,

Update: 08/03/2018 1:25PM CDT - Rephrased the question to make it more descriptive.

Update: 08/03/2018 4:10PM CDT - Added negative scenario test to make more sense.

I have a function/spec-under-test (jvm-languages), i want to test positive scenario(for now) using Clojure generative test API.

Positive scenario test: Given the jvm-language as one of the five {"clojure" "haskell" "erlang" "scala" "python"}, result should be true.

My Question/requirement: I should want to test my function-under-test with each and every possible valid scenario (which i listed in a set below) for each execution (each time i run, lein test) so that i can get rid of regular unit tests.

As per my little testing knowledge, i think there is no value(for me) to test my function multiple times with the same test-scenario in the same execution. Please educate me, if you think my thought process is wrong.

Is there a way to fulfill my use-case using Clojure generative testing?

If not, is it not a great feature to have?

   (ns spec.gen-test.explore
(:require [clojure.test                    :refer :all]
      [clojure.spec                    :as spec]
      [clojure.test.check.generators   :as gen]
      [clojure.test.check.properties   :as prop :include-macros true]
      [clojure.test.check.clojure-test :refer [defspec]]))
(def languages-set #{"clojure" "haskell" "erlang" "scala" "python"})

(spec/def ::jvm-languages 
(spec/and string?
 languages-set))

;;Generative test case for jvm-languages spec:
(defspec jvm-languages-positive-generative-test
 5
 (prop/for-all [language (gen/elements ["clojure" "haskell" "erlang" 
 "scala" "python"])]
            (println "Testing for language:" language)
            (spec/valid? ::jvm-languages language)))

(defspec jvm-languages-negative-generative-test
  100
  (prop/for-all [language (gen/such-that
                           #(not (contains? languages-set %))
                           (gen/one-of
                             [ gen/string
                               (gen/elements ["" nil])
                               gen/int
                               gen/keyword
                               gen/char-alphanumeric
                               gen/double ]))]
            (println "Testing for language:" language)
            (not (spec/valid? ::jvm-languages language))))
(run-tests)

Output: 
#'spec.gen-test.explore/jvm-languages-positive-generative-test
Testing for language: scala
Testing for language: haskell
Testing for language: erlang
Testing for language: erlang
Testing for language: scala
{:result true, :num-tests 5, :seed 1533232724897, :test-var "jvm-languages-generative-test"}

I'd use [`clojure.test/are`](https://clojuredocs.org/clojure.test/are) for this instead of a generative test. — Taylor Wood, Aug 02 '18 at 13:59
@Olim Saidov Yes, it is one of the option, but is there a guarantee that generator picks all my expected edge cases that i listed has been tested? Haven't found in the documentation. — Dhanapathi Javvadi, Aug 03 '18 at 16:31
Generative testing is all about writing invariants that should hold for any possible input that fits a spec - in particular, covering many inputs you wouldn't have thought to write a test for. If you have a small, specific set of tests you want to run exactly once each, generative testing is the wrong tool. — amalloy, Aug 03 '18 at 20:26
Hello @amalloy, i just made an edit and added both positive scenario and negative scenario. I agree with your point "Generative testing is all about writing invariants that should hold for any possible input that fits a spec - in particular, covering many inputs you wouldn't have thought to write a test for" for my negative scenario. If generative test could guarantee that it can run tests for my given scenarios(similar to parameterized testing in JUnit), i could avoid writing cumbersome unit test cases. — Dhanapathi Javvadi, Aug 03 '18 at 21:40
What you've written is exactly a cumbersome unit test. It is more verbose than just using clojure.test to test your positive cases. Your negative test is fine, if it's important for you to test that random garbage inputs don't qualify. — amalloy, Aug 03 '18 at 21:55

Carcigenicate · Accepted Answer · 2018-08-02T14:52:05.467

0

I'd just iterate the collection since you want to test each case anyway.

You could get some reasonably nice-reading code using is and every?:

(ns your-ns
  (:require [clojure.test :refer [is testing]]))

(let [test-cases ["clojure" "haskell" "erlang" "scala" "python"]]
  (testing "if my things are working."
    ; Just iterating the test cases using every?
    (is (every? #(your-test-predicate-here %) test-cases))))

edited Aug 02 '18 at 14:52

answered Aug 02 '18 at 14:37

Carcigenicate

43,494
9
68
117

Thanks buddy. One of my peer suggested the similar approach. trying to find out of the box solution from clojure generative API, if it has one. – Dhanapathi Javvadi Aug 02 '18 at 17:37
@DhanapathiJavvadi Why are you trying to generate a fixed set of cases? It sounds like you're forcibly trying to use the wrong tool for the job here. – Carcigenicate Aug 02 '18 at 19:45
Hello @Carcigenicate, does it make sense now, why i was trying to generate a fixed samples of data? – Dhanapathi Javvadi Aug 03 '18 at 21:45
@DhanapathiJavvadi TBH, I've never used spec before, so I can't comment on that. Maybe I'll just delete my answer if you're sure that you do need generative testing. – Carcigenicate Aug 03 '18 at 21:47

How to generative test function-under-test with must-to-test distinct edge cases for each execution?

1 Answers1