Paying homage to the Compall-Michael pattern, and anticipating Scala 3 Opaque Types

14 May 2021

Neil J. L. Benn

by Neil J. L. Benn

Many Scala developers are excited by the arrival of Scala 3 in 2021. Despite the risks that any language evolution brings, in the case of the evolution from Scala 2 to Scala 3, these are more than compensated for by the impressive range of new language features.

As for me, I already have my favourite new language feature — Opaque Types. Scala for me has always been about the ability to express rich models of some domain in a type-safe way. Opaque types are another great addition to the toolbox.

But the introduction of this new feature also means that it is time for me to say goodbye to one of my favourite Scala 2 techniques, something I call the "Compall-Michael" pattern — and I should point out that I am the only person on the planet that uses this term so there is no use googling it!

"Compall" refers to Stephen Compall who outlined this technique in his talk at LamdaConf 2018 entitled "Opaque Types to Infinity" — see the talk here and download the slides here. I am a big fan of Stephen Compall’s writings so this is my own way of paying homage. "Michael" refers to Julien Michael who is cited in the talk as the original inspiration.

What is the problem that we are trying to solve?

In a nutshell, we want to have "compile-time wrapper types" that provide type-safety at compile-time but without runtime overhead (e.g. no extra memory allocations). This is more easily illustrated with an example (even a toy one), which I will borrow from the Opaque Types section of the Moving Forward from Scala 2 to Scala 3 course that a team of us at Lunatech, led by Eric Loots, prepared and released in 2020.

I will start by illustrating the example using well-known features of Scala 2 - Plain type aliases, Case classes, and Value classes - before moving on to show the "Compall-Michael" representation, and then finally finishing with a solution using Scala 3 Opaque Types.

(Note that I do not cover all the ways that we can solve this in Scala — e.g. using tagged types or even other approaches that make use of excellent libraries such as scala-newtype. Such approaches using third-party libraries are well worth considering in your own real-world use-cases.)

The example talks about Rockets, Boosters, Kilometers, and Miles and was inspired by the famous Mars Probe story of faulty conversion between metric and imperial units. (I feel the need to stress that if you are building an application and you need to work with dimensions and units of measure in a type-safe way then you would almost certainly want to use a proven library like Squants. The snippets below simply constitute toy examples to illustrate and compare different approaches.)

Plain (Transparent) Type Aliases

A typical Scala 2 approach is to use plain type aliases for readability. In the following code snippet, instead of using Double, we introduce two aliases Kilometers and Miles which help us to document when we are working in kilometres and when we are working in miles — e.g. we can document that the Booster class works with miles as a unit of measure.

object Units {
  type Kilometers = Double
  type Miles = Double
}

import Units._

class Booster() {
  def provideLaunchBoost(): Miles = 100
}

However, this does not provide any type-safety because the aliased type is transparent. Consider the following code snippet where we use the other defined alias Kilometers to help us to document that the Rocket class works with kilometres as a unit. The fact that Miles is transparently a Double means that we can inadvertently add kilometres and miles together and the code will compile just fine - meaning that a bug has just slipped through.

class Rocket(booster: Booster) {
  private var distance: Kilometers = 0

  def launch(): Unit = {
    // Kilometers and Miles are transparent. They are both Double so this bug slips through
    distance += booster.provideLaunchBoost()
  }

  def distanceTravelled: Kilometers = distance
}

val rocket: Rocket = new Rocket(new Booster())
rocket.launch();

// Will think it has travelled 100km rather than 160km
rocket.distanceTravelled

Similarly, as these are just values of type Double, with just these plain type aliases we are unable to enforce certain constraints such as "distance cannot be negative". Ideally we want to make this kind of illegal state unrepresentable.

Case class wrappers

The simplest approach to achieve type-safety and catch the bug of mixing up miles and kilometres at compile-time would be to create distinct new "wrapper" types. In the following code snippet, we introduce two case classes Kilometers and Miles that each "wrap" their real underlying Double value. And because these are distinct types we are not able to just simply operate on miles when kilometres is the expected unit of measure. The case classes provide a way to create values of Kilometers and Miles and we provide two helper functions Units.add and Units.toKilometers for adding two values of Kilometers and for converting from Miles to Kilometers.

object Units {
  final case class Kilometers(value: Double)
  final case class Miles(value: Double)

  def add(km1: Kilometers, km2: Kilometers): Kilometers = Kilometers(km1.value + km2.value)
  def toKilometers(miles: Miles): Kilometers = Kilometers(miles.value * 1.6)
}

import Units._

class Booster() {
  def provideLaunchBoost(): Miles = Miles(100)
}

class Rocket(booster: Booster) {
  private var distance: Kilometers = Kilometers(0)

  def launch(): Unit = {
    // Kilometers and Miles are different types. So compiler prevents the previous bug
    val launchBoost: Kilometers = toKilometers(booster.provideLaunchBoost())
    distance = add(distance, launchBoost)
  }

  def distanceTravelled: Kilometers = distance
}

val rocket: Rocket = new Rocket(new Booster())
rocket.launch();

// Will represent the correct distance travelled
rocket.distanceTravelled

Furthermore, if we wanted to enforce the constraints about distance only being positive, we could take the "smart constructor" approach to restrict how we obtain values of type Kilometers and Miles. There are quite a number of ways to achieve this in Scala. For a good explanation of the various ways to implement smart constructors I invite you to check out this recent blog post. As I am using Scala 2.13.5 to compile these examples, I will take advantage of the newer cleaner approach of using a private[foo] access modifier that is possible with the -Xsource:3 compiler option. (If you are on a version of Scala pre-2.13.2, then I would personally recommend the approach of using sealed abstract case class, which is also explained in the linked blog post).

In the following code snippet, the private[Units] modifier means we can only access the Kilometers() and Miles() constructors from within the scope of object Units — externally the only way to obtain Kilometers and Miles values is through the kilometers and miles "smart constructors" that perform some validation.

object Units {
  final case class Kilometers private[Units] (value: Double)
  final case class Miles private[Units] (value: Double)

  val ZeroKm: Kilometers = Kilometers(0)
  val ZeroMi: Miles = Miles(0)

  def kilometers(value: Double): Option[Kilometers] = if (value < 0) None else Some(Kilometers(value))
  def miles(value: Double): Option[Miles] = if (value < 0) None else Some(Miles(value))

  def add(km1: Kilometers, km2: Kilometers): Kilometers = Kilometers(km1.value + km2.value)
  def toKilometers(miles: Miles): Kilometers = Kilometers(miles.value * 1.6)
}

import Units._

class Booster() {
  def provideLaunchBoost(): Miles = miles(100).getOrElse(ZeroMi)
}

class Rocket(booster: Booster) {
  private var distance: Kilometers = ZeroKm

  def launch(): Unit = {
    // Kilometers and Miles are different types. So compiler prevents the previous bug
    val launchBoost: Kilometers = toKilometers(booster.provideLaunchBoost())
    distance = add(distance, launchBoost)
  }

  def distanceTravelled: Kilometers = distance
}

val rocket: Rocket = new Rocket(new Booster())
rocket.launch();

// Will represent the correct distance travelled
rocket.distanceTravelled

So we have achieved what we wanted — which is that we prevent the bug at compile-time — but at the cost of some runtime overhead because we now have to allocate the wrapper Kilometers and Miles objects in memory. It is safe to bet that in practice for most applications this extra overhead is not critical and personally I reach for this simple approach most of the times. However, it is not unheard of to have some performance-critical use-cases where you want to avoid the overhead of unnecessarily allocating and garbage-collecting objects.

Value-class wrappers

Extending the wrappers with AnyVal promises to eliminate the overhead of using a the case-class wrapper presented in the previous section. Classes that extend AnyVal, in addition to meeting a number of other criteria as explained here on this Scala Lang page, are known as value classes. The compiler can still prevent the bug of incorrectly mixing up values of Miles and Kilometers, but without the need to allocate wrapper objects.

object Units {
  final case class Kilometers private[Units] (value: Double) extends AnyVal
  final case class Miles private[Units] (value: Double) extends AnyVal
  ...
}

// Same as before
...

However, in practice allocations can still happen in a number of situations. For example, imagine that we decide to allow the toKilometers method to accept values of a super-type Distance (of which Kilometers and Miles would be two sub-types), we could have code like the following.

object Units {
  sealed trait Distance extends Any
  final case class Kilometers private[Units] (value: Double) extends AnyVal with Distance
  final case class Miles private[Units] (value: Double) extends AnyVal with Distance

  ...

  def toKilometers(distance: Distance): Kilometers = distance match {
    case miles: Miles => Kilometers(miles.value * 1.6)
    case kilometers: Kilometers => kilometers
  }
}

This code is perfectly fine, but if you had chosen the AnyVal route to avoid allocations, then you would be disappointed to discover that now with the supertype-subtype relationship you will once again have allocations when you pass a value of Kilometers or Miles to the function toKilometers.

import Units._

...

class Rocket(booster: Booster) {
  private var distance: Kilometers = ZeroKm

  def launch(): Unit = {
    val launchBoost: Kilometers = toKilometers(booster.provideLaunchBoost()) // Allocation of Miles object
    distance = add(distance, launchBoost)
  }

  def distanceTravelled: Kilometers = distance
}

The various limitations of Value-classes are extensively discussed on the Scala Lang page on Value Classes and on the SIP-15 page there is recognition that improvements can be made for certain use-cases (e.g. numerical computing).

As a quick aside, one interesting limitation is that value classes cannot be nested. Although not strictly necessary for our particular use-case, imagine that we wanted to add some convenient syntax for invoking our helper functions add and toKilometers — e.g. using + infix operator instead of add and being able to invoke .toKm on a Miles value instead of toKilometers. One typical usage of Value-classes is in combination with implicit classes to provide allocation-free extension methods. However, if we attempt to define extension methods in this way, as in the following code snippet, then the code does not compile:

object Units {
  final case class Kilometers private[Units] (value: Double) extends AnyVal
  final case class Miles private[Units] (value: Double) extends AnyVal
  ...

  implicit class KmOps(val km: Kilometers) extends AnyVal {
    def +(km2: Kilometers): Kilometers = add(km, km2)
  }

  implicit class MiOps(val miles: Miles) extends AnyVal {
    def toKm: Kilometers = toKilometers(miles)
  }
}

This gives the following compilation error:

implicit class KmOps(val km: Kilometers) extends AnyVal {
                         ^
error: value class may not wrap another user-defined value class

So in this case, to get around the limitation of nested value-classes, we have to define the extension methods using just implicit class without the extends AnyVal (accepting that this will allocate new instances of these implicit classes when using the extension methods):

object Units {
  final case class Kilometers private[Units] (value: Double) extends AnyVal
  final case class Miles private[Units] (value: Double) extends AnyVal
  ...

  implicit class KmOps(val km: Kilometers) {
    def +(km2: Kilometers): Kilometers = add(km, km2)
  }

  implicit class MiOps(val miles: Miles) {
    def toKm: Kilometers = toKilometers(miles)
  }
}

import Units._

...

class Rocket(booster: Booster) {
  ...

  def launch(): Unit = {
    distance += booster.provideLaunchBoost().toKm
  }

  ...
}

What it looks like with the "Compall-Michael" pattern

In contrast to the preceding well-known and widely-used techniques seen thus far in the post, the approach that I am calling here the "Compall-Michael" pattern is almost certainly less well known (even though in some respects it already has a very long heritage in Scala). This technique makes use of Scala’s ability to emulate the ML module system, which I first heard Martin Odersky discuss in his flatMap Oslo talk in 2014 (click here for the slides) and which is very well explained in this blog post from that same year entitled "Scala’s Modular Roots" by Dan James.

For an in-depth explanation of how the emulation of ML modules in Scala can be achieved, I invite you to read the blog post linked above (and if you want a crash course on ML modules then this blog post is a good quick read). Here I will just say that the main aspects that interest us are emulating ML signatures using sealed trait s and emulating ML structures by implementing the signature with new { …​ } and assigning it to a val. (In our use case we will not need to emulate ML functors which is the third important aspect of the ML module system).

The signature can be thought of as the public API of our module without any implementation. This is an API that not only consists of abstract functions (def declarations) but also unassigned values (val declarations) and abstract types (type declarations). In the code snippet below we have our signature UnitsModule that we define as a sealed trait (it could equally by a sealed abstract class). Inside the signature, we reproduce the same public API that we introduced already in the "Case class wrappers" section. Specifically, we have: - abstract type members Kilometers and Miles - abstract val declarations ZeroKm and ZeroMi that will represent "zero" values of our two types of unit - abstract def smart constructor declarations kilometers and miles that will allow us to create instances of our two types of unit - abstract def declarations that allow us to do useful things with our two types of unit, in this case a method add for adding two Kilometers values and a method toKilometers for converting a Miles value to a Kilometers value - and finally, if we want to, we can even define our extension methods from before, defined in terms of the add and toKilometers methods. (Note, however, that once again we cannot make use of "allocation-free extension methods" because of the limitation of only being able to define them at the top-level or enclosed in a statically accessible object.)

sealed trait UnitsModule {
  type Kilometers
  type Miles

  val ZeroKm: Kilometers
  val ZeroMi: Miles

  def kilometers(value: Double): Option[Kilometers]
  def miles(value: Double): Option[Miles]

  def add(km1: Kilometers, km2: Kilometers): Kilometers
  def toKilometers(miles: Miles): Kilometers

  implicit class KmOps(val km: Kilometers) {
    def +(km1: Kilometers): Kilometers = add(km, km1)
  }

  implicit class MiOps(val miles: Miles) {
    def toKm: Kilometers = toKilometers(miles)
  }
}

The structure can be thought of as the implementation of the signature. In the following code snippet, we instantiate an anonymous class that implements the UnitsModule trait and assign the instance to val Units. In the body of the anonymous class we provide concrete definitions of the two types Kilometers and Miles, defining them both as Double. Similarly, we provide implementations of all the val and def declarations, and in these implementations we can treat values of type Kilometers and Miles as Double values (meaning e.g. that we can directly perform arithmetic operations such as + and * on these values.

val Units: UnitsModule = new UnitsModule {
  type Kilometers = Double
  type Miles = Double

  val ZeroKm: Kilometers = 0
  val ZeroMi: Miles = 0

  def kilometers(value: Double): Option[Kilometers] = if (value < 0) None else Some(value)
  def miles(value: Double): Option[Miles] = if (value < 0) None else Some(value)

  def add(km1: Kilometers, km2: Kilometers): Kilometers = km1 + km2
  def toKilometers(miles: Miles): Kilometers = miles * 1.6
}

However, even though on the inside of the body of new UnitsModule { …​ } we know that both Kilometers and Miles are represented as Double values, from the outside the types Units.Kilometers and Units.Miles are completely different and cannot be used interchangeably. So the compiler can again prevent the bug of adding a value of type Miles to a value of type Kilometers. But in this case, there is absolutely no wrapping involved and the runtime representation of Miles and Kilometers values is indeed Double.

import Units._

class Booster() {
  def provideLaunchBoost(): Miles = miles(100).getOrElse(ZeroMi)
}

class Rocket(booster: Booster) {
  private var distance: Kilometers = ZeroKm

  def launch(): Unit = {
    // Kilometers and Miles are different types. So compiler prevents the previous bug
    distance += booster.provideLaunchBoost().toKm
  }

  def distanceTravelled: Kilometers = distance
}

val rocket: Rocket = new Rocket(new Booster())
rocket.launch();

// Will represent the correct distance travelled
rocket.distanceTravelled

The one big gotcha of this approach is that the ascription val Units: UnitModule is crucial. If you omit that then you break the abstraction and allow the outside world to see that Kilometers and Miles are just aliases for Double and you are back to square one. Having the type annotation :UnitModule means that the outside world can only treat Kilometers and Miles as existential types (meaning, in a nutshell, that the outside world knows only that types of these two names exist, but nothing else). It is this subtle move that gives us the needed opacity around the real representation of Kilometers and Miles.

So consider the following code snippet where we have forgotten the annotation and we have simply val Units = new UnitsModule (or we could have the same effect with object Units extends UnitsModule), now were are back to the situation where Kilometers and Miles are transparently Double and we are back to being able to make the original mistake of adding miles to kilometres.

val Units = new UnitsModule {
  type Kilometers = Double
  type Miles = Double

  val ZeroKm: Kilometers = 0
  val ZeroMi: Miles = 0

  def kilometers(value: Double): Option[Kilometers] = if (value < 0) None else Some(value)
  def miles(value: Double): Option[Miles] = if (value < 0) None else Some(value)

  def add(km1: Kilometers, km2: Kilometers): Kilometers = km1 + km2
  def toKilometers(miles: Miles): Kilometers = miles * 1.6
}

import Units._


class Booster() {
  def provideLaunchBoost(): Miles = miles(100).getOrElse(ZeroMi)
}

class Rocket(booster: Booster) {
  private var distance: Kilometers = ZeroKm

  def launch(): Unit = {
    // Kilometers and Miles are once again transparent so back to initial bug
    distance += booster.provideLaunchBoost()
  }

  def distanceTravelled: Kilometers = distance
}

val rocket: Rocket = new Rocket(new Booster())
rocket.launch();

// Will think it has travelled 100km rather than 160km
rocket.distanceTravelled

I highly recommend that you watch the talk and/or read the slides as it goes much further than the basic usage that I have described here. Particularly fascinating is the example on infinitely recursive types around minute 40 of the talk. I should also point out that the main motivation put forward in Stephen Compall’s talk is "improving abstraction" rather than any concerns about memory allocations or performance-critical use-cases.

What it looks like with Scala 3 Opaque Type Aliases

Opaque types were originally proposed in SIP-35 (cf. the Motivation section). According to the Dotty docs, they aim to "provide type abstraction without any overhead”. Scala 3 introduces the opaque keyword that can be added in front of a plain type alias.

object Units {
  opaque type Kilometers = Double
  opaque type Miles = Double
}

However, these type aliases by themselves are not very useful. That is because, outside of the scope of Units we only know the type names Kilometers and Miles but we cannot do anything useful. At a minimum we need to provide a way to introduce values of our opaque types and a public API for working with values of our opaque types. So here again we reproduce the public API that we first introduced in the "Case class wrappers" section, as well as the extension methods that provide convenient syntax for our Units.add and Units.toKilometers helper functions. However, for this we make use of the new Extension methods feature of Scala 3. The combination of Opaque Types and Extension methods go well together in Scala 3 and make for a cleaner final solution.

object Units {
  opaque type Kilometers = Double
  opaque type Miles = Double

  val ZeroKm: Kilometers = 0
  val ZeroMi: Miles = 0

  def kilometers(value: Double): Option[Kilometers] = if (value < 0) None else Some(value)
  def miles(value: Double): Option[Miles] = if (value < 0) None else Some(value)

  def add(km1: Kilometers, km2: Kilometers): Kilometers = km1 + km2
  def toKilometers(miles: Miles): Kilometers = miles * 1.6

  extension (km: Kilometers) {
    def + (km2: Kilometers): Kilometers = add(km, km2)
  }

  extension (miles: Miles) {
    def toKm: Kilometers = toKilometers(miles)
  }
}

And now, even though inside the body of Units we can treat both Kilometers and Miles as Double values, on the outside, we cannot use these types interchangeably and we have to do the necessary conversion. Note that the extension methods make it a cleaner to add two Kilometers values and to convert from Miles to Kilometers.

import Units._

class Booster() {
  def provideLaunchBoost(): Miles = miles(100).getOrElse(ZeroMi)
}

class Rocket(booster: Booster) {
  private var distance: Kilometers = ZeroKm

  def launch(): Unit = {
    // Kilometers and Miles are different types. So compiler prevents the previous bug
    distance += booster.provideLaunchBoost().toKm
  }

  def distanceTravelled: Kilometers = distance
}

// For fun, let's make use of Scala 3's Universal Apply Methods to omit the 'new'
val rocket: Rocket = Rocket(Booster())
rocket.launch();

// Will represent the correct distance travelled
rocket.distanceTravelled

Wrap up

Hopefully this blog post has made you as keen as I am to really get going with Scala 3 to make use of the new expressive capabilities, Opaque Types being just one of many. If you want to start exploring this very promising new evolution of Scala then the best place to start is the official Scala Lang Scala 3 page. If you want self-paced, hands-on practical exercises to get acquainted with some of these new expressive capabilities, I invite you to run through Lunatech’s Moving Forward from Scala 2 to Scala 3 course. However, if you are still likely to be using Scala 2 for the near to medium term, I hope this post was a gentle introduction to the interesting "Compall-Michael" technique for declaring "compile-time wrapper types" that makes use of existing language features.