FULL STEAK DEVby VICTOR NAKORIAKOV
🌿 Nov 2021

ReScript JSON Typed Strongly

As a follow-up to the “Full stack ReScript” post, I’d like to zoom in on an architectural layer that was intentionally omitted in that article to avoid focus blurring. Namely, the layer responsible for making strongly-typed and business-valid domain objects out of bits and bytes coming over the wire.

More generally, how can we deserialize raw data provided as JSONs (or XML, or Protobuf, or whatever) into domain objects that make sense for the business logic and serialize them back?

Go cooking! Imagine we’re making a cookbook application and have domain objects describing recipe ingredients. The app should be smart, allowing users to adjust the number of portions they want to cook.

// === RecipeDomain.resi ===

module Quantity: {
  type units = [
    | #pc // pieces
    | #tsp // tea spoon
    | #tbsp // table spoon
    | #cup // half a pint
    | #ml
    | #l
    | #g
    | #kg
    | #oz
    | #lb
    | #clove
  ]

  type t

  let make: (float, units) => t

  /* Multiplies value by the given factor */
  let scale: (t, float) => t

  let rawValue: t => (float, units)

  /* Rounds value to nearest 1, ½, ⅓, or ¼ and converts
   units to more appropriate if possible */
  let humanValue: (t, [#us | #metric]) => (float, units)
}

module Ingredient: {
  type t
  let make: (string, Quantity.t) => t
  let product: t => string
  let quantity: t => Quantity.t
}

module IngredientList: {
  type t

  let make: (array<Ingredient.t>, ~numberOfPortions: int) => t
  let all: (t, ~numberOfPortions: int) => array<Ingredient.t>
  let originalNumberOfPortions: t => int
}

While building our front-end app prototype, we work in this cozy land of business domain objects. At some moment we need to save the user’s recipe somewhere and be able to load it later. It might be our back-end or browser local storage. Can we pass IngredientList there as is? The general answer is “no” because wire, file, and browser storage require a continuous sequence of bytes. In contrast, our domain object runtime presentation is unknown and is unlikely to be sequential in RAM.

Well, technically you can use IngredientList almost as is, as long as you’re developing in ReScript. It compiles to JavaScript, and JS objects ≈ JSON objects that trivially serialize into strings. But such way would be too fragile and unsafe 🤞

  • If the internal presentation of any domain object model changes, you break backward compatibility
  • If data comes from an untrusted source (a user), you pass invalid and inconsistent data to the house
  • The JSON schema can be already defined by another party: external service API or another team member
  • A requirement to support other formats besides JSON can appear

OK, domain objects and their JSON representation are different things, and we should learn how to transform one to another and back. For the example, we’re going to work with the following JSON shape:

{
  "number-of-portions": 1,
  [
    {
      "product": "Jumbo Egg",
      "quantity": "2 pc"
    },
    {
      "product": "Salt",
      "quantity": "1 tspn"
    },
    {
      "product": "Milk",
      "quantity": "2 tbspn"
    }
  ]
}

How do we convert between this and our IngredientList? There’re obviously several ways to do it. I’ll describe one that might seem complicated at first sight. But it has proven to scale well if a project has many types of objects and messages.

Three layers

Yep, layers again.

layers

The first one we had described already: it’s the domain layer defining our IngredientList, Ingredient, and Quantity objects. To keep architecture well-shaped, this layer must not care about serialization/deserialization, know something about app message format, or even know anything about such thing as JSON. It should just do the business; constructing and destructing domain objects is others’ responsibility.

DTO

Next comes DTO layer. DTO stands for “Data Transfer Object.” This layer contains type definitions that more or less resemble the shape of JSON data. DTO layer is also responsible for converting DTOs to domain objects and back. In our case, it might be:

// === RecipeDto.res ===

// This module works only with DTO types defined in it and domain
// object types. This domain `open` makes code less cluttered.
open RecipeDomain

//======================================================================
// DTOs
//======================================================================

type ingredient = {
  product: string,
  quantity: string,
}

type ingredientList = {
  numberOfPortions: int,
  items: array<ingredient>,
}

//======================================================================
// Converters
//======================================================================

let unitsFromDomain = (u: Quantity.units) =>
  switch u {
  | #pc => "pc"
  | #tsp => "tsp"
  | #tbsp => "tsp"
  | #cup => "cup"
  | #ml => "ml"
  | #l => "ltr" // there might be difference
  | #g => "gr" // there might be difference
  | #kg => "kg"
  | #oz => "oz"
  | #lb => "lb"
  | #clove => "clove"
  }

let unitsToDomain = s =>
  switch s {
  | "pc" => Ok(#pc)
  | "tsp" => Ok(#tsp)
  | "tbsp" => Ok(#tbsp)
  | "cup" => Ok(#cup)
  | "ml" => Ok(#ml)
  | "ltr" => Ok(#l)
  | "gr" => Ok(#g)
  | "kg" => Ok(#kg)
  | "oz" => Ok(#oz)
  | "lb" => Ok(#lb)
  | "clove" => Ok(#clove)
  // it can fail
  | _ => Error(#UnkonwnUnit(s))
  }

let ingredientFromDomain = (ingredient: Ingredient.t): ingredient => {
  let (amount, units) = ingredient->Ingredient.quantity->Quantity.rawValue

  {
    product: ingredient->Ingredient.product,
    quantity: amount->Float.toString ++ " " ++ units->unitsFromDomain,
  }
}

let ingredientToDomain = (dto: ingredient): result<Ingredient.t, 'err> => {
  // Handle that non-machine-friendly "2 tspn" format
  let quantityR = switch dto.quantity->Js.String2.split(" ") {
  | [amountStr, unitsStr] =>
    switch (amountStr->Float.fromString, unitsStr->unitsToDomain) {
    | (Some(amount), Ok(units)) => Quantity.make(amount, units)->Ok
    | (_, Error(_) as err) => err
    | _ => Error(#BadQuantityFormat(dto.quantity))
    }
  | _ => Error(#BadQuantityFormat(dto.quantity))
  }

  quantityR->Result.map(quantity => Ingredient.make(dto.product, quantity))
}

let ingredientListFromDomain = (ingredients: IngredientList.t): ingredientList => {
  let n = ingredients->IngredientList.originalNumberOfPortions
  {
    numberOfPortions: n,
    items: ingredients->IngredientList.all(~numberOfPortions=n)->Array.map(ingredientFromDomain),
  }
}

let ingredientListToDomain = (dto: ingredientList): result<IngredientList.t, 'err> => {
  dto.items
  ->Array.map(ingredientToDomain)
  ->ResultX.sequence
  ->Result.map(items => IngredientList.make(items, ~numberOfPortions=dto.numberOfPortions))
}

You might ask, what’s the point of creating another set of types that is similar to the set already defined in the domain layer?

  • Domain objects are opaque. We’re free to tweak their internal presentation without fear of breaking communication; DTOs are transparent and much closer resemble the data shape rather than domain objects. And the difference between a DTO and a domain object is sometimes significant.
  • Domain objects and DTOs can evolve independently. For example, DTOs may change in response to an API update, and business logic will not be affected.
  • Domain objects include strict runtime checks to verify business rules and often make it impossible to have an invalid state; DTOs, on the other side, hold data as is providing only minimal validation like field presence or types.

In summary, the separation gives you more flexibility in the long run.

Whew, we are almost done supporting our JSON data schema. But… where is JSON itself here?

Codecs

Here comes codec layer. Its responsibility is converting between a basic primitive type such as string or Js.Json.t and data transfer objects. Why another layer in addition to DTO? It’s all about flexibility again. First, JSON field names might differ (e.g., use kebab-case-names) or imply some defaults; second, you are not limited to JSON only or a particular JSON parser. Having a dedicated layer allows creating different variations for different cases: full-fledged JSON parser/validator for user-facing JSONs, XML parser/validator for interaction with an external system, blind JSON “any-cast” for internal performance-critical message exchange.

To build a solid JSON codec layer I use Jzon library (disclosure: I’m the author of Jzon). Here are codecs which do the job for our ingredient list:

// === RecipeJson.res ===

// This `open` lets refer to DTO fields without prefixes
open RecipeDto

let ingredient = Jzon.object2(
  // A function to convert DTO to a tuple that is friendly to Jzon
  dto => (dto.product, dto.quantity),
  // A function to convert Jzon-friendly tuple back to DTO
  ((product, quantity)) => Ok({product: product, quantity: quantity}),
  // Enumeration of fields comprising the tuple
  Jzon.field("product", Jzon.string),
  Jzon.field("quantity", Jzon.string),
)

// The similar codec, but for another DTO. See Jzon docs for more examples
let ingredientList = Jzon.object2(
  dto => (dto.numberOfPortions, dto.items),
  ((numberOfPortions, items)) => Ok({numberOfPortions: numberOfPortions, items: items}),
  // here we handle a kebab-case-field-name used in JSON
  Jzon.field("number-of-portions", Jzon.int),
  Jzon.field("items", Jzon.array(ingredient)),
)

Aren’t codecs just boilerplate? Indeed, they closely follow the shapes of data types defined in the DTO layer. But some code is anyway required to lift an arbitrary-shaped Js.Json.t to well-defined DTOs. Someone has to write it down. I’m thinking about a tool to auto-generate Jzon codecs out of types defined in a DTO module. Perhaps it’s possible, but I’d leave it at the level of an idea until some actual demand.

As I told already, you might use other tools to create the codecs layer. Take bs-json if it feels better. Or even force Js.Json.t to DTO conversion with Obj.magic if you know what you do.

Putting all together

Finally, all ropes are stretched across the layers. What’s next? We can use it!

// === Serialize ===
let myJson =
  ingredients->RecipeDto.ingredientListFromDomain->Jzon.encodeWith(RecipeJson.ingredientList)

// === Deserialize ===
let ingredientsR =
  myJson
  ->Jzon.decodeWith(RecipeJson.ingredientList)
  ->ResultX.mapError(err => #JsonDecodingError(err))
  ->Result.flatMap(dto => dto->RecipeDto.ingredientListToDomain)

switch ingredientsR {
| Ok(ingredients) =>
  /* show recipe to user */
| Error(#JsonDecodingError(err)) =>
  /* handle low-level error somehow */
  Js.log(err->Jzon.DecodingError.toString)
| Error(#BadQuantityFormat(str)) =>
  /* handle other errors, e.g. show an error in UI */
| Error(#UnkonwnUnit(str)) =>
  /* handle other errors, e.g. show an error in UI */
| Error(_) =>
  /* and so on */
}

Alternatives (or not)

Isn’t this three-layer system too complicated to get a JSON object and just extract some field values? Maybe. However, in a system with many data types required to be serialized/deserialized, such layer separation adds its maintainability points, despite seemingly much boilerplate.

Anyway, you always have alternatives to consider:

  • ReScript Apollo Client generates type-safe bindings to GraphQL schemas. Nice option if you only develop the front-end part in ReScript and the server provides GraphQL API.
  • JSON Schema might be used to describe data shape along with a validator such as Ajv to verify the validity of a JSON. This can save you from creating codec layers.

And in some quick’n’dirty or straightforward cases, you are free to just use Object.magic to any-cast between Js.Json.t and {..}, then using this object fields directly hoping/knowing the fields are indeed there and have valid types.

So, it all depends on your project. Pick a strategy and go ahead. You can always “upgrade” if it does not fit your needs later.