zio and scalatest

If you use scalatest with zio, you need to recognize that there are a few ways to hook up scalatest. In the end, the problem is that scalatest uses exceptions to signal failure and that does not really work well with a library that models errors explicitly (or fp in general which treats errors as values).

Some notes:

• scalatest has two async-test error channels–throwing an exception in open code and putting an exception into a Future . You can throw an exception or return an Assertion. The Assertion can be true or not true. However, a failed Assertion really just throws a TestFailedException immediately there is no “failed” Assertion value. You can get about as much debugging information from throwing an exception as you do an Assertion if you convert the zio Cause to a string.
• You can run async tests but you need to return a Future[Assertion].
• I wanted to keep everything async so I could run this in JS when that’s available for zio.
• scalatest’s Assertion is really only Succeeded. If the assertion fails, it throws. So if you call assert, you really need to wrap it immediately and convert the throw to a value. The nice thing about assert is that it is easier to attach a note to it without having to use withClue. But if you know its a failure, just call fail(message).
• If you use an implicit class to make the conversion easier, you lose source location information as all failures are reported from the source position where fail is called.

Here’s what I used for my async test spec class. I do not use the zio-interop-future module although I could have:

 package ttg

abstract class ZIOTestSpec extends AsyncFlatSpec with RTS {
  // used to allow pattern matching
  private[ttg] case class ZIOTestException[E](c: ExitResult.Cause[E])
      extends RuntimeException(c.toString())
  implicit class ToFuture[E](io: IO[E, Assertion]) {
    private[ttg] def toTestFuture2: Future[Assertion] = {
      // if we were using sync vs async, we could just use the following line
      //unsafeRun(io.toFutureE(e => new RuntimeException(e.toString)))
      val p = scala.concurrent.Promise[Assertion]()
      unsafeRunAsync(io){ // or _.fold(...)
        case ExitResult.Failed(c) => p.failure(ZIOTestException(c))
        case ExitResult.Succeeded(a) => p.complete(Try(a))
      }
      p.future
    }
    // push E into a TestFailedException
    def toTestFuture: Future[Assertion] = {
      val p = scala.concurrent.Promise[Assertion]()
      unsafeRunAsync(io){
        // fail() throws a TestFailedException (private to scalatest)
        case ExitResult.Failed(e) => p.complete(Try(fail(e.toString)))
        case ExitResult.Succeeded(a) => p.complete(Try(a))
      }
      p.future
    }
  // other extensions
  // ...
}

You could use the future-interop module to produce a IO[Nothing, Future[A]] and keep the exception in the Future but if you want a TestFailedException, you have to manually create it yourself vs using fail().

We use a ZIOTestException in toTestFuture2 because with declaring the exception class, we are unable to pick out the TestFailedException from scalatest because TestFailedException is private to scalatest. When we know our zio is failed, we need to flip it like:

ioa.toTestFuture2.recover { case ZIOTestException(c) => Succeeded }

which we can do:

  private[ttg] implicit class RichFuture(f: Future[Assertion]) {
    // flip a ZIOTestException to Succeeded.
    def expectFailure: Future[Assertion] =
      f.recover {
        case ZIOTestException(c) => Succeeded
      }
    def expectFailure[E](e: ExitResult.Cause[E]): Future[Assertion] =
      f.recover {
        case ZIOTestException(c) if e == c => Succeeded
      }    
  }

  implicit class RichIO[E](io: IO[E, Assertion]) {
    def expectFailure: Future[Assertion] =
      io.toTestFuture2.expectFailure
    def expectFailure(c: ExitResult.Cause[E]): Future[Assertion] =
      io.toTestFuture2.expectFailure(c)
  }

This is the FP version of intercept or assertThrows in scalatest. Of course, you could have just handled the E in the IO directly perhaps using IO.redeemPure or IO.attempt and then fiddling with Either[E,A].

Now, we can return the scalatest required Future[Assertion] using:

myio.toTestFuture // puts the E into TestFailedException via fail() if E is present
// or
myio.expectFailure // if we want to ensure myio is failed

It’s interesting to note that just putting this together highlights the total lack of discipline I always had around exceptions as I was usually focused on larger machine learning/ai batch or information management programs (training or scoring) so the precision was never needed. In UIs, a bit more precision can be helpful to tailor the UI based on the error details.

It looks like specs2 returns a Result value that can be success or failure. For this type of testing, specs2 seems less confusing. Note that async testing in specs2 for scala.js is new and uses Future. Most importantly though, it does not use exceptions by default to signal failure. So as long as you push your E to a Throwable, you can do async exceptions in specs2.

That’s it!