Verifying upgrade compatibility

Overview

When upgrading a canister, it is important to verify that the upgrade can proceed without:

Introducing an incompatible change in stable declarations.
Breaking clients due to a Candid interface change.

dfx checks these properties statically before attempting the upgrade. Moreover, with enhanced orthogonal persistence, Motoko rejects incompatible changes of stable declarations.

Upgrade example

The following is a simple example of how to declare a stateful counter:

import Debug "mo:base/Debug";

actor Counter_v0 {
  var state : Nat = 0;

  public func increment() : async () {
    state += 1;
    Debug.print(debug_show (state));
  };
};

Importantly, in this example, when the counter is upgraded, its state is lost. This is because actor variables are by default flexible, meaning they get reinitialized on an upgrade.

To fix this, you can declare a stable variable that is retained across upgrades:

import Debug "mo:base/Debug";

actor Counter_v1 {
  stable var state : Nat = 0;

  public func increment() : async () {
    state += 1;
    Debug.print(debug_show (state));
  };
};

If the variable state were not declared stable, state would restart from 0 on upgrade.

Evolving the stable declarations

Changing counter from Nat to Int is a compatible change in stable declarations. The counter value is retained during the upgrade.

import Debug "mo:base/Debug";

actor Counter_v2 {
  stable var state : Int = 0;

  public func increment() : async () {
    state += 1;
    Debug.print(debug_show (state));
  };
};

Stable type signatures

A stable type signature describes the stable content of a Motoko actor. You can think of this as the interior interface of the actor, that it presents to its future upgrades.

For example, v1's stable types:

actor {
  stable var state : Nat
};

An upgrade from v1 to v2's stable types consumes a Nat as an Int, which is valid because Int <: Nat.

actor {
  stable var state : Int
};

Evolving the Candid interface

In this extension of the interface, old clients remain satisfied, while new ones get extra features such as the decrement function and the read query in this example.

actor Counter_v3 {
  stable var state : Int = 0;

  public func increment() : async () {
    state += 1;
  };

  public func decrement() : async () {
    state -= 1;
  };

  public query func read() : async Int {
    return state;
  };
};

Dual interface evolution

An upgrade is safe provided that both the Candid interface and stable type signatures remain compatible:

Each stable variable must either be newly declared, dropped, or re-declared at a supertype of its old type.
The Candid interface evolves to a subtype.

Consider the following four versions of the counter example:

Version v0 with Candid interface v0.did and stable type interface v0.most:

service : {
  increment : () -> ();
};

actor {  
};

Version v1 with Candid interface v1.did and stable type interface v1.most,

service : {
  increment : () -> ();
};

actor {
  stable var state : Nat
};

Version v2 with Candid interface v2.did and stable type interface v2.most,

service : {
  increment : () -> ();
};

actor {
  stable var state : Int
};

Version v3 with Candid interface v3.did and stable type interface v3.most:

service : {
  increment : () -> ();
  decrement : () -> ();
  read : () -> (int) query;
};

actor {
  stable var state : Int
};

Incompatible upgrade

Let's take a look at another example where the counter's type is again changed, this time from Int to Nat:

import Float "mo:base/Float";

actor Counter_v4 {
  stable var state : Float = 0.0;

  public func increment() : async () {
    state += 0.5;
  };

  public func decrement() : async () {
    state -= 0.5;
  };

  public query func read() : async Float {
    return state;
  };
};

This version is neither compatible to the Candid interface nor to the stable type declarations.

Since Float </: Int, the new type of state is not compatible to the old type.
The return type change of read is also not valid.

Motoko rejects upgrades with incompatible state changes with enhanced orthogonal persistence. This is to guarantee that the stable state is always kept safe.

Error from Canister ...: Canister called `ic0.trap` with message: RTS error: Memory-incompatible program upgrade.

In addition to Motoko's check, dfx raises a warning message for these incompatible changes, including the breaking Candid change.

Versions of Motoko using classical orthogonal persistence will drop the state and reinitialize the counter with 0.0, if the dfx warning is ignored.

Explicit migration

There is always a migration path to change structure of stable state, even if a direct type change is not compatible.

For this purpose, a user-instructed migration can be done in three steps:

Introduce new variables of the desired types, while keeping the old declarations.

Write logic to copy the state from the old variables to the new variables on upgrade.

While the previous attempt of changing state from Int to Nat was invalid, you now can realize the desired change as follows:

import Debug "mo:base/Debug";
import Float "mo:base/Float";

actor Counter_v5 {
  stable var state : Int = 0;
  stable var newState : Float = Float.fromInt(state);

  public func increment() : async () {
    newState += 0.5;
  };

  public func decrement() : async () {
    newState -= 0.5;
  };

  public query func read() : async Int {
    Debug.trap("No longer supported: Use `readFloat`");
  };

  public query func readFloat() : async Float {
    return newState;
  };
};

To also keep the Candid interface, the readFloat has been added, while the old read is retired by keeping its declaration and raising a trap internally.

Drop the old declarations once all data has been migrated:

import Debug "mo:base/Debug";
import Float "mo:base/Float";

actor Counter_v6 {
  stable var newState : Float = 0.0;

  public func increment() : async () {
    newState += 0.5;
  };

  public func decrement() : async () {
    newState -= 0.5;
  };

  public query func read() : async Int {
    Debug.trap("No longer supported: Use `readFloat`");
  };

  public query func readFloat() : async Float {
    return newState;
  };
};

Alternatively, the type of state can be changed to Any, also implying that this variable is no longer used.

Upgrade tooling

dfx incorporates an upgrade check. For this purpose, it uses the Motoko compiler (moc) that supports:

moc --stable-types …: Emits stable types to a .most file.
moc --stable-compatible <pre> <post>: Checks two .most files for upgrade compatibility.

Motoko embeds .did and .most files as Wasm custom sections for use by dfx or other tools.

To upgrade e.g. from cur.wasm to nxt.wasm, dfx checks that both the Candid interface and stable variables are compatible:

didc check nxt.did cur.did  // nxt <: cur
moc --stable-compatible cur.most nxt.most  // cur <<: nxt

Using the versions above, the upgrade from v3 to v4 fails this check:

> moc --stable-compatible v3.most v4.most
(unknown location): Compatibility error [M0170], stable variable state of previous type
  var Int
cannot be consumed at new type
  var Float

With enhanced orthogonal persistence, compatibility errors of stable variables are always detected in the runtime system and if failing, the upgrade is safely rolled back.

With classical orthogonal persistence, however, an upgrade attempt from v2.wasm to v3.wasm is unpredictable and may lead to partial or complete data loss if the dfx warning is ignored.

Adding record fields

A common, real-world example of an incompatible upgrade can be found on the forum.

In that example, a user was attempting to add a field to the record payload of an array, by upgrading from stable type interface:

actor {
  type Card = {
    title : Text;
  };
  stable var map : [(Nat32, Card)] = [(0, { title = "TEST"})];
};

to incompatible stable type interface:

actor {
  type Card = {
    title : Text;
    description : Text;
  };
  stable var map : [(Nat32, Card)] = [];
};

Problem

When trying this upgrade, dfx issues the following warning:

Stable interface compatibility check issued an ERROR for canister ...
Upgrade will either FAIL or LOSE some stable variable data.

(unknown location): Compatibility error [M0170], stable variable map of previous type
  var [(Nat32, Card)]
cannot be consumed at new type
  var [(Nat32, Card__1)]

Do you want to proceed? yes/No

It is recommended not to continue, as you will lose the state in older versions of Motoko that use classical orthogonal persistence. Upgrading with enhanced orthogonal persistence will trap and roll back, keeping the old state.

Adding a new record field to the type of existing stable variable is not supported. The reason is simple: The upgrade would need to supply values for the new field out of thin air. In this example, the upgrade would need to conjure up some value for the description field of every existing card in map. Moreover, allowing adding optional fields is also a problem, as a record can be shared from various variables with different static types, some of them already declaring the added field or adding a same-named optional field with a potentially different type (and/or different semantics).

Solution

To resolve this issue, an explicit is needed:

You must keep the old variable map with the same structural type. However, you are allowed to change type alias name (Card to OldCard).
You can introduce a new variable newMap and copy the old state to the new one, initializing the new field as needed.
Then, upgrade to this new version.

import Array "mo:base/Array";

actor {
  type OldCard = {
    title : Text;
  };
  type NewCard = {
    title : Text;
    description : Text;
  };
  
  stable var map : [(Nat32, OldCard)] = [];
  stable var newMap : [(Nat32, NewCard)] = Array.map<(Nat32, OldCard), (Nat32, NewCard)>(
    map,
    func(key, { title }) { (key, { title; description = "<empty>" }) },
  );
};

After we have successfully upgraded to this new version, we can upgrade once more to a version, that drops the old map.

actor {
  type Card = {
    title : Text;
    description : Text;
  };
  stable var newMap : [(Nat32, Card)] = [];
};

dfx will issue a warning that map will be dropped.

Make sure, you have previously migrated the old state to newMap before applying this final reduced version.

Stable interface compatibility check issued a WARNING for canister ...
(unknown location): warning [M0169], stable variable map of previous type
  var [(Nat32, OldCard)]
 will be discarded. This may cause data loss. Are you sure?

Verifying upgrade compatibility

Overview​

Upgrade example​

Evolving the stable declarations​

Stable type signatures​

Evolving the Candid interface​

Dual interface evolution​

Incompatible upgrade​

Explicit migration​

Upgrade tooling​

Adding record fields​

Problem​

Solution​