Possible bug in kaocha's output if test fails

[mikeananev]

Kaocha Version
1.71.1119

Platform
Operating System: macOS m1 12.5
Clojure version: 1.11.1
JDK vendor and version: OpenJDK Runtime Environment Homebrew (build 19.0.1) with --enable-preview parameter for Virtual Threads

Symptom
If test fails then I get awful kaocha's Exception output

Reproduction

1. Clone my repo SWIM
   git clone https://github.com/redstarssystems/swim/
cd swim
2. Select particular commit
   git checkout f9ef925837ff41afdb75f3cc10ba1e4fd4887841
3. Run tests to make sure that all of them are green
   clj -M:test
4. Then change value 3 to 4 on line 1993 to make test fail.
5. Run tests again and see result
   clj -M:test

[alysbrooks]

@mikeananev Thanks for the bug report (and reproduction instructions)! Do you happen to know if you see a similar problem when other tests in this project fail? I'll probably play around with your repository myself, too.

[mikeananev]

Hi @alysbrooks !
Yes, when other tests fail I often see this error. But it happens only with tests with meta ^:logic or ^:event-processing.

[plexus]

The fail-fast exception should normally never bubble to the top, so I'm labeling this as a bug. Might be related to specific user setup or config, but we should evaluate if it's something we can and should fix on our end.

@mikeananev that's a huge repro repo you shared, any chance you could make a more minimal example that still reproduces the issue?

[alysbrooks]

Hi @mikeananev, we're reviewing old issues. In the case of bug reports, we're trying to make sure we have all the information we'd need to reproduce and fix it. Would you be able to minimize the reproduction example?

[onetom]

Kaocha 1.87.1366 still has this issue.
I have a very short repro example.
Given this test:

(deftest thrown-match?-test
  (is (thrown-match? 1 1)))

without the fail fast feature:

(kaocha.repl/run #'thrown-match?-test {:kaocha/fail-fast? false})

results in a concise test summary:

EXPECTED:
(thrown-match? 1 1)

ACTUAL:
the expected exception wasn't thrown
1 tests, 1 assertions, 1 failures.
=>
{:kaocha.result/count 1,
 :kaocha.result/pass 0,
 :kaocha.result/error 0,
 :kaocha.result/fail 1,
 :kaocha.result/pending 0}

with with fail fast active

(kaocha.repl/run #'thrown-match?-test {:kaocha/fail-fast? true})

we would get the whole kaocha test plan printed, unless we restrict printing depth with (set! *print-level* 6):

EXPECTED:
(thrown-match? 1 1)

ACTUAL:
(mismatch
 (expected 1)
 (actual
  {:kaocha/fail-fast true,
        :caused-by
        {:file destination_test.clj,
         :line 469,
         :kaocha/testable #,
         :kaocha/test-plan #,
         :type :fail,
         :expected #,
         :actual the expected exception wasn't thrown,
         :message nil}}))

1 tests, 2 assertions, 2 failures.
=>
{:kaocha.result/count 1,
 :kaocha.result/pass 0,
 :kaocha.result/error 0,
 :kaocha.result/fail 2,
 :kaocha.result/pending 0}

I was wondering whether this is a kaocha or matcher-combinators issue, but I'm not sure.
Maybe it's an issue between the two?
Maybe fail-fast in kaocha shouldn't be implemented via throwing an exception?
Borkdude mentioned here that implementing fail-fast behaviour with clojure.test is pretty easy

[onetom]

actually, i have a custom version of kaocha.report/testing-vars-str and kaocha.report/fail-summary for :kaocha/fail-type & :error dispatch values, but even without that customization the problem is the same, just the output is slightly different

FAIL in ..../thrown-match?-test (xxx_test.clj:468)
expected: (thrown-match? 1 1)
  actual: the expected exception wasn't thrown

FAIL in ..../thrown-match?-test (xxx_test.clj:468)
expected: (thrown-match? 1 1)
  actual: (mismatch
 (expected 1)
 (actual
  {:kaocha/fail-fast true,
        :caused-by
        {:file "destination_test.clj",
         :line 468,
         :kaocha/testable #,
         :kaocha/test-plan #,
         :type :fail,
         :expected #,
         :actual the expected exception wasn't thrown,
         :message nil}}))

[plexus]

I see now, fail-fast as you mention is implemented by throwing an exception. It has to be, because it's the only way we can break out of a test block early. We normally catch it higher up so it's invisible to the user, but thrown-match? catches it instead.

Note that borkdude's "fail-fast" is not actually fail fast, it short circuits after the first failing test var, not after the first failing assertion.

I'd have to go through the code path and implementations to see what the right way is to solve this. We could redefine (monkey patch) thrown-match? but maybe there's a more elegant solution.

[onetom]

We hit this issue again last Friday, but luckily we were pair-programming and I remembered this issue.

[alysbrooks]

This isn't a very elegant solution either, but I wonder if the reporter could detect this situation and display "the expected exception wasn't thrown" instead of the fail-fast exception.