**CC with 1-step withdrawal only**: this milestone allows you to support deposits and withdrawals of CC. It includes earning app rewards for all CC deposits. The workflows build on the Canton Network Token [Standard]() which is the foundation for supporting all CN tokens in the next milestone. We consider it an intermediate milestone, as it does not support:
* all CN tokens
* CC users that prefer to control the receipt of transfers, and thus do not want to setup preapprovals
* earning app rewards for all deposits and withdrawals
See the following sections for details on the work items it depends on.
* Setup DevNet node and/or use LocalNet
* `exchange-parties-setup`
* `one-step-deposit-workflow`
* `one-step-withdrawal-workflow`
* Support restore from validator node backup
* Support hard synchronizer migration
**MVP for all CN Tokens**: this milestone allows you to support deposits and withdrawals of all CN tokens. It comes with the limitation that application rewards are only earned on deposits of CC, but not on deposits of other CN tokens. It depends on the MVP for CC and the following additional work items:
* `multi-step-deposit-workflow`
* `multi-step-withdrawal-workflow`, which resolves the limitation that users must setup a CC transfer preapproval to receive withdrawals.
* `token-onboarding`
**Earn app rewards for all CN tokens**: is a milestone that improves the profitability of the integration by implementing changes so the exchange earns application rewards on both withdrawals and deposits of all CN tokens. Sharing application rewards is an optional steps.
* `withdrawal-app-rewards`
* `deposit-app-rewards`
* `share-rewards-with-customers`
## Integration support code
Use the following support code to simplify your integration development for:
> * **JavaScript/TypeScript**: use the functions from the Wallet SDK to simplify building your integration.
> * **Java/JVM**: use the sample code from the [https://github.com/digital-asset/ex-java-json-api-bindings](https://github.com/digital-asset/ex-java-json-api-bindings) repository as a starting point.
> * **Other languages**: use the code from the Wallet SDK or the Java sample code as a blueprint.
{/* COPIED_START source="splice-wallet-kernel:docs/wallet-integration-guide/src/exchange-integration/architecture.rst" hash="56b6899e" */}
# Integration Architecture
## High-Level Overview
There are three main information flows:
1. **Tx History Ingestion**: ingests the transactions (tx) affecting the `treasuryParty` from the Exchange Validator Node into the Canton Integration DB. Arrow 1.a represents the transaction data being read using the `/v2/updates/flats` Ledger API endpoint using either plain [HTTP]() or [websockets](). It is parsed by the Tx History Ingestion service to update the status of funds, deposits, and withdrawals in the Canton Integration DB (Arrow 1.b).
This data is queried by Exchange Internal Systems (Arrow 1.c), for example to serve the Exchange UI. For brevity, the diagram shows direct access to the Canton Integration DB by the Exchange Internal Systems. However using a micro-services architecture, the Exchange Internal Systems would typically access the Canton Integration DB through a dedicated API layer. Choose whatever architecture best fits your exchange's needs.
2. **Withdrawal Automation**: starts with the Exchange Internal Systems writing a withdrawal request to the Canton Integration DB (Arrow 2.a). The Withdrawal Automation service reads the request from the DB (Arrow 2.b), and prepares, signs, and executes a Canton Network Token standard transfer corresponding to the withdrawal request using the Ledger API (Arrow 2.c).
Note that the status of transfers becomes visible in the transaction history ingested by the Tx History Ingestion service; and is communicated to both the Exchange Internal Systems and the Withdrawal Automation service via the Canton Integration DB. This routing of information through the Canton Integration DB is intentional to simplify disaster recovery.
Note also that the Withdrawal Automation may write back to the Canton Integration DB to mark a withdrawal as failed.
3. **Multi-Step Deposit Automation**: is required to support offer-and-accept style transfers for tokens that do not support direct transfers. It relies on the Tx Ingestion Service to ingest transfer offers as part of Arrow 1.c.
The workflow starts with the Multi-Step Deposit Automation service querying the Canton Integration DB to see whether there are pending transfers for deposits from customers (Arrow 3.a). The service then checks whether the deposit address specified in the transfer is known. If yes, it prepares, signs, and executes an accept transaction using the Ledger API (Arrow 3.b). If no, then it takes no action, and lets the transfer offer expire or be withdrawn by the sender.
Note that there is an arrow from Multi-Step Deposit Automation back to the Canton Integration DB, as the Multi-Step Deposit Automation may write back to the Canton Integration DB to store that the transaction to accept the deposit could not be committed even after retrying multiple times.
The other information flows interact with the main flows as part of a deposit or withdrawal. We explain them in the `integration-workflows` section.
{/* COPIED_START source="splice-wallet-kernel:docs/wallet-integration-guide/src/exchange-integration/workflows.rst" hash="60badcf7" */}
# Integration Workflows
## Overview
The workflows below are grouped into two milestones.
* `mvp-for-cc` contains the minimum viable product (MVP) workflows for integrating Canton Coin (CC) into the exchange. It comes with the limitation that both the exchange and the customers need to set up a `TransferPreapproval` contract to enable 1-step transfers of CC.
* `mvp-for-cn-tokens` contains the additional workflows required to support all CN tokens. They are the workflows to onboard a new token and to support multi-step transfers for both deposits and withdrawals. Multi-step transfers gives the receiver a choice to: reject an incoming transfer as well as enable additional asynchronous checks on transfers by the token admin (e.g. KYC/AML checks).
Further extensions of these two MVPs to address Day-2 requirements are explored in `integration-extensions`.
> 
Assumptions:
* The Exchange has set up a CC `TransferPreapproval` for their `treasuryParty` as explained in `treasury-party-setup`.
* The Exchange has associated deposit account “abc123” with the Customer in the Canton Integration DB.
Example flow:
1. Customer uses Exchange UI to retrieve `treasuryParty` and the deposit account-id (“abc123”) to use for the deposit
2. Customer uses Customer Wallet to initiate a token standard transfer of 100 CC to `treasuryParty` with metadata key `splice.lfdecentralizedtrust.org/reason` set to “abc123”.
1. Customer Wallet selects CC `Holding` UTXOs to fund the transfer and queries Canton Coin Scan to retrieve registry-specific `TransferFactory` and extra transfer context. The returned data includes the `TransferPreapproval` for the `treasuryParty`.
2. Customer wallet submits the command to exercise the `TransferFactory_Transfer` choice together with the extra transfer context. The resulting transaction:
> * archives the funding CC `Holding` UTXOs
> * creates a CC `Holding` UTXO with contract-id `coid234` owned by the `treasuryParty`
> * creates another CC `Holding` UTXO for the change owned by the Customer.
3. The resulting transaction gets committed across the Customer, Exchange, and SV validator nodes. It is assigned an update-id `upd567` and a record time `t1` by the Global Synchronizer. It is assigned offset `off1` by the Exchange Validator Node. (The other validator nodes will have a different `offset` value.)
3. Tx History Ingestion observes `upd567` at record time `t1` with offset `off1` and updates the Canton Integration DB as follows.
1. Tx History Ingestion parses `upd567` using the token standard tx history parser from the Wallet SDK to determine:
* The deposit amount of 100 CC.
* The deposit account “abc123” from the `splice.lfdecentralizedtrust.org/reason` metadata value.
* The new `Holding` UTXO `coid234` owned by the `treasuryParty`
2. Tx History ingestion writes the following in a single, atomic transaction to the Canton Integration DB
* The latest ingested update-id `upd567`, its record time `t1`, its offset `off1`, and the `synchronizerId` of the Global Synchronizer.
* The new CC `Holding` UTXO `coid234` for the 100 CC that was received.
* The credit of 100 CC on the Customer’s account at the exchange.
4. Customer observes the successful deposit in their Exchange UI, whose data is retrieved from the Canton Integration DB via the Exchange Internal Systems.
### 1-Step Withdrawal Workflow
Assumptions:
1. Customer set up a CC `TransferPreapproval` for their `customerParty`.
Example flow:
1. Customer requests withdrawal of 100 CC to `customerParty` using the Exchange UI.
2. Exchange Internal Systems process that request and update the Canton Integration DB to store:
* The deduction of 100 CC from the Customer's trading account.
* The pending withdrawal with id `wid123` of 100 CC to `customerParty`.
* The CC `Holding` UTXOs `coids` to use to fund the transfer to `customerParty` for `wid123`. See `utxo-management` for more information.
* The target record time `trecTgt` on the Global Synchronizer until which the transaction for the CC transfer must be committed. The `coids` are considered to be reserved for funding the transfer for withdrawal `wid123` until `trecTgt` has passed.
3. Withdrawal Automation queries the Canton Integration DB in a polling fashion, observes the pending withdrawal `wid123`, and commits the corresponding CC transfer as follows.
1. Withdrawal Automation queries Canton Coin Scan to retrieve the `TransferFactory` for CC and extra transfer context.
2. Withdrawal automation checks that transfer is indeed a 1-step transfer by checking that `transfer_kind` = `"direct"` in the response from Canton Coin Scan. If that is not the case, then it marks the withdrawal as failed in the Canton Integration DB with reason "lack of CC transfer-preapproval for `customerParty`" and stops processing.
3. Withdrawal Automation prepares, signs, and submits the command to exercise the `TransferFactory_Transfer` choice with the exclusive upper-bound for the record time of the commit set to `trecTgt`. It also sets the value for key `splice.lfdecentralizedtrust.org/reason` in the `Transfer` metadata to `wid123`.
4. The resulting transaction:
> * archives the CC `Holding` UTXOs `coids` used to fund the transfer
> * creates a CC `Holding` UTXO with contract-id `coid345` owned by the `customerParty`
> * creates a CC `Holding` UTXO with contract-id `coid789` owned by `treasuryParty` representing the change returned to the Exchange.
The transaction is committed across the Customer, Exchange, and SV validator nodes. It is assigned an update-id `upd567` and a record time `t1` \< `trecTgt` by the Global Synchronizer. It is assigned `off1` by the Exchange Validator Node. It is assigned `off2` by the Customer Validator Node.
4. Tx History Ingestion observes `upd567` at `t1` with offset `off1` and updates the Canton Integration DB as follows.
1. Tx History Ingestion parses `upd567` using the token standard tx history parser from the Wallet SDK to determine:
* The withdrawal-id `wid123` from the `splice.lfdecentralizedtrust.org/reason` metadata value.
* The new `Holding` UTXO `coid789` owned by the `treasuryParty`
2. Tx History ingestion writes the following in a single, atomic transaction to the Canton Integration DB
* The latest ingested update-id `upd567`, its record time `t1` and offset `off1`.
* The successful completion of withdrawal `wid123` by the transaction with update-id `upd567` at record time `t1`.
* The deduction of 100 CC from the Customer's trading account.
* The archival of the CC `Holding` UTXOs `coids`.
* The new CC `Holding` UTXO `coid789` for the change returned after funding the CC transfer.
5. Customer Wallet observes `upd567` at `t1` with offset `off2` on the Customer Validator Node, parses it using the token standard tx history parser and updates its UI as follows:
* Its tx history shows the receipt of 100 CC from `exchangeParty` with “Reason” `wid123` that was committed as update `upd567` at `t1`.
* Its holding listing shows the new CC `Holding` with contract id `coid345`.
6. Customer observes the completion of the withdrawal at `t1` in the Exchange UI and the receipt of the expected funds in their Customer Wallet.
### UTXO Selection and Management
Executing a withdrawal requires selecting `Holding` UTXOs to fund the withdrawal, as described for example in `one-step-withdrawal-workflow`. You likely already have a UTXO management strategy in place for your existing UTXO-chain integrations. Here some considerations to take into account when adapting your strategy to work with Canton:
* Canton Coin charges a small holding fee of about \$1 per year for each `Holding` UTXO to allow archiving dust [coins]() once their holding fee surpasses their value.
* Canton Coin limits the number of UTXOs for a single transfer to 100 `Holding` UTXOs to avoid large transactions that are expensive to process.
* Canton Coin transactions also merge all input `Holding` UTXOs and return the change to the sender as a single `Holding` UTXO to allow batching the merging of `Holding` UTXOs with transfers.
* Other tokens are likely to follow similar strategies for the same rationale.
* At the time of writing (2025-08-29), the Canton Network Token Standard recommends to use self-transfers (i.e., `sender` = `receiver`) to be used to merge `Holding` UTXOs into two `Holding` UTXOs: one for the transferred `amount` and another one for the change. It does not (yet) support requesting multiple `Holding` UTXOs to be created for the change.
We therefore recommend the following approach:
* Limit the number of input UTXOs to less than 100 UTXOs per transfer. Thus staying with the Canton Coin limits and keeping transaction size small, which also helps you to reduce your traffic spend when having to retry transaction execution.
* Consider using a UTXO selection strategy for withdrawals that favors smaller UTXOs so that they get merged automatically as part of executing transfers.
* Consider keeping a pool of `k` large amount UTXOs to be able to execute up to `k` withdrawals at the same time. Run a periodic background job to manage this pool using self-transfers.
* From an implementation perspective, these self-transfers are a special kind of withdrawal. We thus recommend to implement them using the same code path as withdrawals: start with writing the self-transfer request into the Canton Integration DB and have the Withdrawal Automation execute it.
## MVP for all Canton Network Tokens
The MVP for supporting all Canton Network tokens builds on the MVP for Canton Coin. The key changes required are:
* Change Tx History Ingestion to also ingest the `TransferInstruction` UTXOs, which are used by the Canton Network Token Standard to represent in-progress transfers (see [docs](), [code]()).
* Adjust the Exchange UI to show the status of in-progress transfers.
* Adjust the user funds tracking done as part of Tx History Ingestion to credit funds back to the user if they reject a withdrawal transfer. Consider deducting a fee for the failed withdrawal.
* Implement the Multi-Step Deposit Automation service to auto-accept incoming transfers that are pending receiver acceptance. Ensure that the deposit address is known before accepting the transfer.
* Add support for configuring the URL of a token admin's Registry API Server and to deploy their .dar files as described in `token-onboarding`.
The sections below provide worked examples for the resulting multi-step deposit and withdrawal workflows. All examples assume that:
1. There is a token admin called **Acme** who issues a token called **AcmeToken** on the Canton Network and operates their own Admin Validator Node and their own Registry API Server.
2. The Exchange and Customer have onboarded AcmeToken as per `token-onboarding`.
### Multi-Step Deposit Workflow
#### Example flow: deposit offer and acceptance
The flow uses essentially the same initial four steps as the `one-step-deposit-workflow` above. We list them in full for completeness.
1. Customer uses Exchange UI to retrieve `treasuryParty` and deposit account-id “abc123” to use for the deposit.
2. Customer uses Customer Wallet to initiate a token standard transfer of 100 AcmeToken to `treasuryParty` with metadata key `splice.lfdecentralizedtrust.org/reason` set to “abc123”.
1. Customer Wallet selects AcmeToken `Holding` UTXOs to fund the transfer and queries Acme's Registry API Server to retrieve registry-specific `TransferFactory` and extra transfer context. The URL for this server was configured in the Customer Wallet as part of `token-onboarding`.
2. Customer wallet submits the command to exercise the `TransferFactory_Transfer` choice together with the extra transfer context. The resulting transaction:
> * archives the funding AcmeToken `Holding` UTXOs
> * creates a locked 100 AcmeToken `Holding` UTXO with contract-id `coid234` owned by the `customerParty`
> * creates another AcmeToken `Holding` UTXO for the change owned by the Customer.
The transaction also creates a `TransferInstruction` UTXO with contract-id `coid567`, which represents the transfer offer to the Exchange.
3. The resulting transaction gets committed across the Customer, Exchange, and Acme validator nodes. It is assigned an update-id `upd567` and a record time `t1` by the Global Synchronizer. It is assigned offset `off1` by the Exchange Validator Node.
3. Tx History Ingestion observes `upd567` at `t1` with offset `off1` and updates the Canton Integration DB as follows.
1. Tx History Ingestion parses `upd567` using the token standard tx history parser from the Wallet SDK to determine:
* The deposit amount of 100 AcmeToken.
* The deposit account “abc123” from the `splice.lfdecentralizedtrust.org/reason` metadata value.
* The `TransferInstruction` UTXO `coid567` representing the transfer offer for the deposit.
2. Tx History ingestion writes the following in a single, atomic transaction to the Canton Integration DB
* The latest ingested update-id `upd567` its record time `t1` and offset `off1`.
* The `TransferInstruction` UTXO `coid567` representing the transfer offer from `customerParty` for a deposit of 100 AcmeToken in account "abc123".
4. Customer Wallet ingests update `upd567` and Customer observes the pending transfer offer for the deposit in the Customer Wallet. Customer also sees the 100 AcmeToken `Holding` UTXO `coid234` locked to the deposit.
This is where the main difference to the `one-step-deposit-workflow` starts. The Multi-Step Deposit Automation service will now auto-accept the transfer offer.
5. The Multi-Step Deposit Automation regularly queries the Canton Integration DB for pending transfer offers for known deposit accounts. It thus observes the pending transfer offer `coid567` and accepts it as follows.
> 1. Multi-Step Deposit Automation retrieves the URL for Acme's Registry API Server from the Canton Integration DB.
> 2. Multi-Step Deposit Automation queries Acme's Registry API Server to retrieve the extra context to exercise the `TransferInstruction_Accept` choice on `coid567`.
> 3. Multi-Step Deposit Automation prepares, signs, and submits the command to exercise the `TransferInstruction_Accept` choice on `coid567`.
> 4. The resulting transaction gets committed across the Customer, Exchange, and Acme validator nodes. It is assigned an update-id `upd789` and a record time `t2` the Global Synchronizer. It is assigned `off3` by the Exchange Validator Node. The resulting transaction has the following effects:
> * It archives the `TransferInstruction` UTXO `coid567`.
> * It archives the locked 100 AcmeToken `Holding` UTXO `coid234` owned by the `customerParty`.
> * It creates a 100 AcmeToken `Holding` UTXO `coid999` owned by the `treasuryParty`.
At this point the workflow again proceeds the same way as the `one-step-deposit-workflow`.
6. Tx History Ingestion observes `upd789` at `t2` with offset `off3` and updates the Canton Integration DB as follows.
1. Tx History Ingestion parses `upd789` using the token standard tx history parser from the Wallet SDK to determine:
* The deposit amount of 100 AcmeToken.
* The deposit account “abc123” from the `splice.lfdecentralizedtrust.org/reason` metadata value.
2. Tx History ingestion writes the following in a single, atomic transaction to the Canton Integration DB
* The latest ingested update-id `upd789`, its record time `t2` and offset `off3`.
* The new AcmeToken `Holding` UTXO `coid999` for the 100 AcmeToken that was received.
* The credit of 100 AcmeToken on the Customer's account at the exchange.
7. Customer Wallet observes `upd789` at `t2` on the Customer Validator Node, parses it using the token standard tx history parser and updates its UI as follows:
* Its tx history shows the successful transfer of 100 AcmeToken to `exchangeParty` with “Reason” `wid123` that was committed as update `upd789` at `t2`.
8. Customer observes the successful deposit in their Exchange UI, whose data is retrieved from the Canton Integration DB via the Exchange Internal Systems.
#### Example: handling deposits with unknown deposit accounts
To minimize traffic cost, we recommend not acting on deposits with unknown deposit accounts. The sender can use their wallet to withdraw the offer.
Ingesting deposit offers with unknown deposit accounts is still valuable to allow the exchange's support team to handle customer inquiries about these transfers.
### Multi-Step Withdrawal Workflow
#### Example flow: withdrawal offer and acceptance
The flow uses essentially the same initial six steps as the `one-step-withdrawal-workflow` above. We list them in full for completeness.
1. Customer requests withdrawal of 100 AcmeToken to `customerParty` using the Exchange UI.
2. Exchange Internal Systems process that request and update the Canton Integration DB to store:
* The deduction of 100 AcmeToken from the Customer's trading account.
* The pending withdrawal with id `wid123` of 100 AcmeToken to `customerParty`.
* The AcmeToken `Holding` UTXOs `coids` to use to fund the transfer to `customerParty` for `wid123`. See `utxo-management` for more information.
* The target record time `trecTgt` on the Global Synchronizer until which the transaction for the AcmeToken transfer must be committed using the `coids` UTXOs for funding `wid123`. The `coids` are considered to be reserved to funding this transfer until `trecTgt` has passed.
3. Withdrawal Automation queries the Canton Integration DB in a polling fashion, observes the pending withdrawal `wid123`, and commits the corresponding AcmeToken transfer as follows.
1. Withdrawal Automation retrieves the URL for Acme's Registry API Server from the Canton Integration DB.
2. Withdrawal Automation queries Acme's Registry API Server to retrieve the `TransferFactory` for AcmeToken and extra transfer context.
3. Withdrawal Automation prepares, signs, and submits the command to exercise the `TransferFactory_Transfer` choice with the exclusive upper-bound for the record time of the commit set to `trecTgt`. It also sets the value for key `splice.lfdecentralizedtrust.org/reason` in the `Transfer` metadata to `wid123`; and it sets the upper bound for the customer to accept the transfer far enough in the future, so that the customer has sufficient time to act (e.g. 1 year).
4. The resulting transaction gets committed across the Customer, Exchange, and Acme validator nodes. It is assigned an update-id `upd567` and a record time `t1` \< `trecTgt` by the Global Synchronizer. It is assigned `off1` by the Exchange Validator Node. It is assigned `off2` by the Customer Validator Node. The resulting transaction has the following effects:
* It archives the AcmeToken `Holding` UTXOs `coids` used to fund the transfer.
* It creates an AcmeToken `Holding` UTXO with contract-id `coid789` owned by `treasuryParty` representing the change returned to the Exchange.
* It creates one locked AcmeToken `Holding` UTXO with amount 100 and contract-id `coid345` owned by the `treasuryParty`.
* It creates a `TransferInstruction` UTXO with contract-id `coid567` representing the transfer offer. This `TransferInstruction` includes a copy of the `Transfer` specification and its metadata.
4. Tx History Ingestion observes `upd567` at `t1` with offset `off1` and updates the Canton Integration DB as follows.
1. Tx History Ingestion parses `upd567` using the token standard tx history parser from the Wallet SDK to determine:
* The withdrawal-id `wid123` from the `splice.lfdecentralizedtrust.org/reason` metadata value.
* The new locked AcmeToken `Holding` UTXO `coid345` owned by the `treasuryParty` and locked to the withdrawal `wid123` of 100 AcmeToken to `customerParty`.
* The new AcmeToken `Holding` UTXO `coid789` owned by the `treasuryParty`
* The `TransferInstruction` UTXO `coid567` representing the transfer offer for the withdrawal.
2. Tx History ingestion writes the following in a single, atomic transaction to the Canton Integration DB:
* The latest ingested update-id `upd567`, its record time `t1` and offset `off1`.
* The successful transfer offer for withdrawal `wid123` by the transaction with update-id `upd567` at record time `t1`.
* The `Holding` UTXO `coid345` locked to the withdrawal.
* The `TransferInstruction` UTXO `coid567` representing the transfer offer.
* The archival of the AcmeToken `Holding` UTXOs `coids`.
* The new AcmeToken `Holding` UTXO `coid789` for the change returned after funding the AcmeToken transfer.
5. Exchange UI displays that withdrawal `wid123` is pending transfer offer acceptance by the Customer.
6. Customer Wallet observes update with update-id `upd567` at `t1` with offset `off2` on the Customer Validator Node.
1. It parses the transaction using the token standard transaction history parser and updates its UI so that its transaction history shows the offer for a transfer of 100 AcmeToken from `exchangeParty` with “Reason” `wid123` that was committed as update `upd567` at `t1`.
This is where the main difference to the `one-step-withdrawal-workflow` starts. The customer has a choice whether to accept or reject the transfer offer. Here they choose to accept it.
7. Customer uses their Customer Wallet to accept the offer using the `TransferInstruction_Accept` choice.
1. The resulting transaction is committed across Exchange, Acme, and Customer validator nodes and assigned update-id `upd789` and record time `t2`. The transaction has the following effects:
* It archives the locked `Holding` UTXO `coid345`.
* It archives the `TransferInstruction` UTXO `coid567`.
* It creates a 100 AcmeToken `Holding` UTXO `coid999` owned by the `customerParty`.
8. Tx History Ingestion observes update `upd789` at `t2` and offset `off3` assigned by the Exchange Validator Node.
1. It parses the update using the token standard parser to extract the withdrawal-id `wid123` from the `splice.lfdecentralizedtrust.org/reason` metadata value.
2. Tx History Ingestion writes the following in a single, atomic transaction to the Canton Integration DB
* The latest ingested update-id `upd789`, its record time `t2` and offset `off3`.
* The successful completion of the withdrawal `wid123` by the transaction with update-id `upd789` at record time `t2`.
* The archival of the locked AcmeToken `Holding` UTXO `coid345`.
9. Customer Wallet observes `upd789` at `t2` and updates its display to reflect its effects.
10. Customer observes the completion of the withdrawal at `t2` in Exchange UI and confirms the receipt of funds in their Customer Wallet.
#### Example flow: customer rejects transfer offer
The Customer might decide to reject the offer in Step 7 in the example above. The corresponding transaction will
> * archive the locked `Holding` UTXO `coid345`,
> * archive the `TransferInstruction` UTXO `coid567`, and
> * create a new 100 AcmeToken `Holding` UTXO `coid999` owned by the `treasuryParty`.
Steps 8 - 10 are largely the same as for the successful acceptance with the difference that Tx History Ingestion will see this transaction and update the Canton Integration DB to such that
> * withdrawal `wid123` is marked as failed because the customer rejected the offer, and
> * the customer account is credited back the 100 AcmeToken, potentially minus a fee for the failed withdrawal.
And the user will ultimately see in both the Exchange UI and the Customer Wallet that the transfer was offered, but rejected by them.
Below you learn how to handle crashes of the integration components, how to handle RPC errors, and how to perform disaster recovery for the Exchange Validator Node.
## Handling Crashes
Validator nodes are crash-fault tolerant and do not lose data shared on the Ledger API in case of a crash. Thus a restart is sufficient to recover from crashes. Likewise, we assume that the Canton Integration DB is backed by a crash-fault tolerant database (e.g., PostgreSQL or MySQL).
For the integration components that you build, we recommend the following strategy to handle crashes and restarts:
* **Tx History Ingestion**: keep track of the last ingested offset in the Canton Integration DB. On restart, continue from that offset. If none is set, then that means it never ingested any transaction. In that case, start from the beginning of the transaction history, i.e., start from offset `0`.
For this to work, it is important that you store the ingested offset in the same transaction as you store the ingested data. See the individual `integration-workflows` descriptions for details.
* **Withdrawal Automation**: make it stateless, so that it can just restart. This is in line with how we recommend to implement both the `one-step-withdrawal-workflow` and the `multi-step-withdrawal-workflow`.
* **Multi-Step Deposit Automation**: make it stateless, so that it can just restart. This is in line with how we recommend to implement the `multi-step-deposit-workflow`.
## Handling RPC Errors
Below we explain our recommendation for handling RPC errors in the integration components you are building. We focus on handling errors from interacting with the Ledger API and the Registry API Servers of the token admins. We do not cover handling errors from accessing DBs or other internal systems, as we assume you have strategies in place for those.
* **Tx History Ingestion**: only reads from the Ledger API. We recommend to retry these reads a bounded number of times on `retryable-errors`. Wait at least a few seconds between retries and consider using exponential [backoff]() to avoid overloading the Validator Node. Consider crashing the ingestion component if the bounded number of retries is exceeded to recover from bugs in the in-memory state of the ingestion component.
- **Withdrawal Automation**: recall from `one-step-withdrawal-workflow` that the Withdrawal Automation first retrieves extra context from the Registry API Server of the token admin and then prepares, signs, and executes the transaction to submit the transfer for the withdrawal using the `/v2/interactive-submission/execute` endpoint of the Ledger API.
The endpoint is asynchronous and observing its response only means that the transaction has been accepted for processing. You can retrieve the status of the execution via the `/v2/commands/completions` endpoint of the Ledger API; or alternatively, by observing the effect of the execution via the Tx History Ingestion component. The latter option is more robust, as it ensures that you observe the effect of the execution in a persistent manner.
We recommend that you retry the steps from the start when not observing the successful completion of the withdrawal within the expected time or when encountering a retryable error on the execution itself. You thereby ensure that you prepare the withdrawal transaction using the latest state of the Validator Node and the latest extra context from the Registry API Server. Use a bounded number of retries with at least a few seconds between retries and consider using exponential backoff.
Retrying all steps is safe from a consistency perspective because the withdrawal transaction is idempotent, as it archives the UTXOs used to fund the transfer once the transfer is submitted. You nevertheless want to avoid retrying too often, as executing a transaction costs traffic.
Stop retrying once the withdrawal has been marked as definitely failed in the Canton Integration DB by the Tx History Ingestion component. A withdrawal is considered definitely failed once its target record time `trecTgt` is below the last ingested record time.
- **Multi-Step Deposit Automation**: the approach is analogous to the one for Withdrawal Automation.
Recall from `multi-step-deposit-workflow` that the Multi-Step Deposit Automation discovers a pending deposit by reading from the Canton Integration DB, then retrieves extra context from the Registry API Server of the token admin and finally prepares, signs, and executes the transaction to accept the transfer offer using the `/v2/interactive-submission/execute` endpoint of the Ledger API.
The endpoint is asynchronous and observing its response only means that the transaction has been accepted for processing. You can retrieve the status of the execution via the `/v2/commands/completions` endpoint of the Ledger API; or alternatively, by observing the effect of the execution via the Tx History Ingestion component. The latter option is more robust, as it ensures that you observe the effect of the execution in a persistent manner.
We recommend that you retry the steps from the start when not observing the successful completion of the transfer offer acceptance within the expected time or when encountering a retryable error on the execution itself. You thereby ensure that you prepare the transaction to accept the transfer offer using the latest state of the Validator Node and the latest extra context from the Registry API Server. Use a bounded number of retries with at least a few seconds between retries and consider using exponential backoff.
Retrying all steps is safe from a consistency perspective because the accept transaction is idempotent, as it archives the transfer offer once it is accepted. You nevertheless want to avoid retrying too often, as executing a transaction costs traffic.
You can stop retrying after a bounded number of retries. The sender can reclaim their funds at any point by withdrawing the offer. The Multi-Step Deposit Automation will learn about the withdrawal of the offer via the Tx History Ingestion component, which will mark the transfer offer as withdrawn in the Canton Integration DB.
### Retryable errors
For increased robustness and fault tolerance, we recommend to retry by default on all errors and manage an exclude list of non-retryable errors. As a starting opint, we suggest to exclude the following HTTP error codes from retries:
* 401 Unauthorized
* 403 Forbidden
* 500 Internal Server Error
* 501 Not Implemented
### Reading from Canton Coin Scan
As explained in `mvp-for-cc`, the Registry API Server of the token admin for Canton Coin is provided by the Canton Coin Scan [services](). They are run as part of every SV node.
For convenience, every Validator Node provides a Scan proxy [service]() to read from the Scan instances run by SVs with Byzantine fault tolerance. The Scan proxy service also implements the Token Standard Registry API for Canton Coin.
We recommend to use Scan proxy service of the Exchange Validator Node to retrieve the extra context for Canton Coin transfers.
If that is not possible, then you can read from a random Canton Coin Scan instance for the purpose of retrieving extra context for Canton Coin transfers. The on-ledger validation of the transfers ensures that you do not need to trust the Scan instance for correctness. Ensure that you read from a different Scan instance on every retry to avoid being affected by a faulty Scan instance for too long.
{/* COPIED_START source="splice-wallet-kernel:docs/wallet-integration-guide/src/exchange-integration/node-operations.rst" hash="7a5fbddf" */}
# Validator Node Operations
## Reward Minting and Traffic Funding
As explained in `tokenomics-and-rewards`, your validator node will need traffic to submit the transactions to execute withdrawals or accept multi-step deposits. As also explained in that section, the network provides rewards that can be used to fund traffic.
Note also that every validator node has an associated **validator operator party** that represents that validator node's administrator ([docs]()). The validator node automatically mints rewards for that party. It can further be configured to automatically purchase [traffic]() using that party's CC balance, which includes the minted rewards.
We thus recommend the following setup as a starting point to mint rewards and automatically fund traffic:
1. Use the validator operator party as your featured `exchangeParty`. Follow `exchange-party-setup` to get it featured.
2. `treasury-party-setup` to create a `treasuryParty` with a transfer preapproval managed by your `exchangeParty`.
3. Setup automatic traffic purchases in the validator [app]().
4. Optional: setup [auto-sweep]() from the `exchangParty` to your `treasuryParty` to limit the funds managed directly by the validator node.
As a starting point for the automatic traffic purchase configuration, set `targetThroughput` to 2kB/s and `minTopupInterval` to 1 minute, which should be sufficient to execute about one withdrawal or deposit acceptance every 10 seconds. Please test this with your expected traffic pattern and adjust as needed. See this FAQ to measure the traffic spent on an individual [transaction]().
## Setup Exchange Parties
### Setup the featured exchange party
As explained above in `reward-minting-and-traffic-funding`, we recommend to use the validator operator party as your featured `exchangeParty`. This party is automatically created when you deploy your validator [node](). Thus the only setup step is to get it featured by the SVs:
**On DevNet**, you can self-feature your validator operator party as follows:
1. Log into the wallet UI for the validator [user](), which presents itself as in this screenshot:
2. Tap 20 \$ of CC to ensure that your validator operator party has enough funds to purchase traffic.
3. Click on the "Self-grant featured app rights" button.
4. The button is replaced with a star ⭐ icon once the `FeaturedAppRight` contract has been created for your validator operator party. This may take about 10 seconds.
That's all. Continue with setting up your treasury party.
**On MainNet**, apply for featured status for your validator operator party as follows:
1. Log into the wallet UI for the validator [user]() on your MainNet validator node.
2. Copy the party-id of your validator operator party using the copy button right of the abbreviated `"google-oaut.."` party name in the screenshot above.
3. Apply for featured application status using this link: [https://sync.global/featured-app-request/](https://sync.global/featured-app-request/)
Wait until your application is approved. The validator node will automatically pick up the featured status via the corresponding `FeaturedAppRight` contract issued by the DSO party for its validator operator party.
**On TestNet** there is currently no official process, but you should be able to use the same procedure as the one for MainNet.
### Setup the treasury party
Setup the `treasuryParty` as follows with a transfer preapproval managed by your `exchangeParty`:
1. Create the `treasuryParty` using the wallet SDK to `create-an-external-party` with a key managed in a system of your choice
2. Copy the party id of your `exchangeParty` from the Splice Wallet UI as explained above, or retrieve it by calling `/v0/validator-user` on the Validator [API]().
3. Call `/v2/commands/submit-and-wait` on the Ledger [API]() to create a `#splice-wallet:Splice.Wallet.TransferPreapproval:TransferPreapprovalProposal` ([code]()) directly with the `provider` set to your `exchangeParty`.
Note that setting up this transfer preapproval requires the `exchangeParty` to pay a small fee of about 0.25 \$ worth of CC. The funds for this fee usually come from the validator liveness rewards that a validator node starts minting about 30 minutes after it is created. On DevNet or LocalNet, you don't have to wait that long: just "Tap" the required funds from the built-in faucet.
### Testing the party setup
You can test the party setup on LocalNet or DevNet as follows:
1. Setup your `exchangeParty` and `treasuryParty` as explained above.
2. Setup an additional `testParty` representing a customer.
3. Transfer some CC from the `testParty` to the `treasuryParty` to simulate a deposit.
4. Observe the successful deposit by listing holdings of the `treasuryParty`.
5. Observe about 30' later in the Splice Wallet UI of your validator operator user that the `exchangeParty` minted app rewards for this deposit. It takes 30', as activity recording and rewards minting happen in different phases of a minting round.
## Setup Ledger API Users
Clients need to authenticate as a Ledger API [user]() to access the Ledger API of your Exchange Validator Node. You can manage Ledger API users and their rights using the `/v2/users/...` endpoints of the Ledger [API]().
You will need to authenticate as an existing user that has `participant_admin` rights to create additional users and grant rights. One option is to authenticate as the `ledger-api-user` that you configured when setting up authentication for your validator [node](). Another option is to log-in to your Splice Wallet UI for the validator operatory [party]() and use the JWT token used by the UI.
We recommend that you setup one user per service that needs to access the Ledger API. This way you can easily manage permissions and access rights for each service independently. The [rights]() required by the integration components are as follows:
| Component | Required Rights | Purpose |
| ------------------------------------------------------------------- | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Tx History Ingestion | `canReadAs(treasuryParty)` | Read transactions and contracts for the `treasuryParty`. |
| Withdrawal Automation | `canActAs(treasuryParty)` | Prepare and execute transactions on behalf of the `treasuryParty`. |
| Multi-Step Deposit Automation | `canActAs(treasuryParty)` | Prepare and execute transactions on behalf of the `treasuryParty`. |
| Automated exchange parties setup for `exchange-integration-testing` | `participant_admin` and `canActAs(treasuryParty)` | Create parties and use the `treasuryParty` to create its `TransferPreapprovalProposal`. Hint: grant `canActAs(treasuryParty)` to the user doing the setup after allocating the `treasuryParty`. |
Required Ledger API User Rights
## .dar File Management
`.dar` files define the Daml workflows used by the token admins for their tokens. They must be uploaded to your Exchange Validator Node to be able to process withdrawals and deposits for those tokens.
The `.dar` files for Canton Coin are managed by the Validator Node itself. The `.dar` files for other tokens need to be uploaded by you using the `/v2/packages` endpoint of the Ledger [API](). See this how-to [guide]() for more information.
## Backing up the Exchange Validator Node
Follow the Splice documentation on how to backup a validator [node]().
## Backing up the Canton Integration DB
As explained in the `canton-integration-components` section of the `integration-architecture`, the Canton Integration DB is more of a logical component. Whether you implement it as a separate DB or as part of the DBs backing your Exchange Internal Systems is up to you.
Follow your internal guidance and best practices on what DB system to use and how to back it up.
## Restoring the Exchange Validator Node from a Backup
Follow the Splice documentation on how to restore a validator node from a [backup]() to restore the Exchange Validator Node from a backup that is less than 30 days old.
The node will resubscribe to transaction data from the synchronizer and recover all committed transactions and the corresponding changes to the set of active contracts (i.e. UTXOs). However validator-node local data written after the backup will be lost, as described on the Canton documentation [page]().
In the context of the recommended `integration-workflows`, this data loss affects:
* **.dar file uploads**: handle this by repeating the upload of all `.dar` files that were uploaded after the backup. This should be a rare event, as token onboarding is infrequent.
* **Ledger API offsets**: offsets assigned to transactions received from the Ledger API may change. This only affects the Tx History Ingestion component of the integration.
### Runbook
Follow these steps to restore the Exchange Validator Node from a backup:
1. Stop Tx History Ingestion before restoring the Exchange Validator Node from a backup.
2. Retrieve the record time `tRecovery` and `synchronizerId` of the last ingested transaction from the Canton Integration DB.
3. Restore the Exchange Validator Node from the backup.
4. Reupload all `.dar` files that were uploaded after the backup.
5. Log into the Canton Console of your validator [node]() and query the offset `offRecovery` assigned to `tRecovery` using
```scala theme={"theme":{"light":"github-light","dark":"github-dark"}}
def parseTimestamp(t: String) = {
val isoFormat = java.time.format.DateTimeFormatter.ISO_INSTANT.withZone(java.time.ZoneId.of("Z"))
isoFormat.parse(t, java.time.Instant.from(_))
}
val synchronizerId = SynchronizerId.tryFromString("example::1220b1431ef217342db44d516bb9befde802be7d8899637d290895fa58880f19accc") // example
val tRecovery = parseTimestamp("2024-05-01T12:34:56.789Z") // example
val offRecovery = participant.parties.find_highest_offset_by_timestamp(synchronizerId, tRecovery)
```
Alternatively, you can use `grpcurl` to query the offset `offRecovery` from the command line as shown in the example below:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
grpcurl -plaintext -d \
'{"synchronizerId" : "example::1220be58c29e65de40bf273be1dc2b266d43a9a002ea5b18955aeef7aac881bb471a",
"timestamp": "2025-11-27T06:50:00.000Z"}' \
localhost:5002 \
com.digitalasset.canton.admin.participant.v30.PartyManagementService.GetHighestOffsetByTimestamp
```
If you use authentication for the Canton Admin gRPC API, then you need to add the appropriate authentication flags to the `grpcurl` command above.
6. Configure the Tx History Ingestion component to start ingesting from offset `offRecovery`.
7. Restart the Tx History Ingestion component.
Once Tx History Ingestion has caught up, the integration workflows will continue as before the disaster.
[https://github.com/canton-network/wallet-gateway/issues/272](https://github.com/canton-network/wallet-gateway/issues/272) Update this when wallet SDK support is available
As part of the initial treasury party setup, you generate the `PartyToParticipant` topology transaction which lists both the confirming nodes and the confirmation threshold. To host a party on multiple nodes, you need to include all confirming nodes in the `PartyToParticipant` mapping when you setup the party initially. Note that at this point, the wallet SDK library does not yet support this so you must go directly through the Canton APIs. This is expected to change soon.
Until then, the easiest way to do so at the moment is through the Canton console. You can find a full reference for all required steps in the integration test. Note in particular that you must sign the `PartyToParticipant` mapping not just by your party's key but also by all confirming participants. This is accomplished through the `participant2.topology.transactions.authorize` step in the test.
### .dar File Management
Any .dar file that you upload, both as part of the initial setup but also whenever you upload newer versions to upgrade an existing package, must be uploaded to all validator nodes hosting your party.
### Reading Data and Submitting Transactions
Both nodes serve all transactions for the `treasuryParty` and can thus be used in principle to read them. However, offsets are not comparable across nodes so it is recommended that to run Tx History Ingestion against the same node under normal operations. If you do need to switch nodes, you can do so following the same procedure used for restoring a validator from a backup to resynchronize Tx History Ingestion against the offsets of the new node.
Preparation and execution of transactions can also be done against any of the confirming nodes of the party. However, Command Deduplication is only performed by the executing node so if you submit across nodes you cannot rely on it. It is therefore recommend \_[not]() to rely on command deduplication at all in favor of UTXO and max record time based deuplication.
Link to recommended deduplication strategy [https://github.com/canton-network/wallet-gateway/issues/423](https://github.com/canton-network/wallet-gateway/issues/423)
### Changing the set of Confirming Nodes
There are some limitations on changing the set of confirming nodes:
Removing confirming nodes is possible by submitting a new `PartyToParticipant` topology transaction. However, this can leave the nodes that you remove in a broken state so this should be limited to cases where that node got compromised or is no longer needed for other purposes.
Adding new confirming nodes is not currently possible. If this is required, you need to instead:
1. Setup a new treasury party with the desired set of confirming nodes.
2. Either transfer all funds from the existing treasury party to the new one and switch only to the new treasury party or rely on `treasury-sharding` to use both treasury parties until you are ready to phase out the old party.
Changing the confirmation threshold is possible at any point by submitting a new `PartyToParticipant` topology transaction with the updated threshold.
Future versions of Canton will allow changing the confirming nodes without the need for setting up a new party.
## Using a KMS for Validator Node Keys
See the Splice docs for how to setup you validator node with keys stored in a [KMS](). Consider doing so as an additional security hardening measure to protect the keys of the confirming node(s)\_ of your `treasuryParty`.
## Using the gRPC Ledger API
Feel free to do so if you prefer using gRPC. It is functionally equivalent to the JSON Ledger API. See this Ledger API [overview]() for more information.