Documentation Index

Fetch the complete documentation index at: https://docs.canton.network/llms.txt

Use this file to discover all available pages before exploring further.

​

Scaling and Performance

​

Network Scaling

​

Node Scaling

• The database-backed drivers (Postgres and Oracle) can run in an active-active setup with parallel processing, supporting multiple writer and reader processes. Thus, such nodes can scale horizontally.
• The enterprise participant node processes transactions in parallel (except the process of conflict detection which by definition must be sequential), allowing much higher throughput than the community version. The community version processes each transaction sequentially. Canton processes make use of multiple CPUs and will detect the number of available CPUs automatically. The number of parallel threads can be controlled by setting the JVM properties scala.concurrent.context.numThreads to the desired value.

​

Performance and Sizing

​

Batching

​

Asynchronous Submissions

  canton.participants.participant1.ledger-api.command-service.max-commands-in-flight = 256 // default value

​

Storage Estimation

• Two encrypted envelopes with payload Y each, one view seed per view, one symmetric key per informee group, two root hashes for each participant and the participant IDs as recipients at the sequencer store, and the informee tree for the mediator (informees and transaction metadata, but no payload), together with the sequencer database indexes.
• Two encrypted envelopes with payload Y each and the symmetric keys for the views, in the participant events table of each participant (as both receive the data)
• Decrypted new resulting contract of size Y in the private contract store and some status information of that contract on the active contract journal of the sync service.
• The full decrypted transaction with a payload of size Y for the created contract, in the sync service linear event log. This transaction does not contain the input contract arguments.
• The full decrypted transaction with Y in the indexer events table, excluding input contracts, but including newly divulged input contracts.

​

Set Up Canton to Get the Best Performance

​

System Design / Architecture

​

Hardware and Database

​

Configuration

participant1.resources.set_resource_limits(
  ResourceLimits(
    // Allow for submitting at most 200 commands per second
    maxSubmissionRate = Some(200),

    // Limit the number of in-flight requests to 500.
    // A "request" includes every transaction that needs to be validated by participant1:
    // - transactions originating from commands submitted to participant1
    // - transaction originating from commands submitted to different participants.
    // The chosen configuration allows for processing up to 100 requests per second
    // with an average latency of 5 seconds.
    maxInflightValidationRequests = Some(500),

    // Allow submission bursts of up to `factor * maxSubmissionRate`
    maxSubmissionBurstFactor = 0.5,
  )
)

• throughput is the number of requests per second Canton needs to handle and
• latency is the time taken to process a single request while Canton is receiving requests at rate throughput.

canton.participants.<participant-name>.ledger-api.rate-limit {
  max-streams = 333
  max-api-services-queue-size = 444
  max-api-services-index-db-queue-size = 555
  max-used-heap-space-percentage = 66
  min-free-heap-space-bytes = 7777777
}

max-streams = 1000
max-api-services-queue-size = 10000
max-api-services-index-db-queue-size = 1000
max-used-heap-space-percentage = 100
min-free-heap-space-bytes = 0

canton.participants.<participant-name>.ledger-api {
  rate-limit = null
}

canton.sequencers.sequencer.public-api.limits.active = {
  "com.digitalasset.canton.sequencer.api.v30.SequencerService/DownloadTopologyStateForInit" : 10,
  "com.digitalasset.canton.sequencer.api.v30.SequencerService/Subscribe" : 1000,
}

  canton.participants.participant1.sequencer-client.maximum-in-flight-event-batches = 100
  canton.mediators.mediator1.sequencer-client.maximum-in-flight-event-batches = 100
  canton.sequencers.sequencer1.sequencer-client.maximum-in-flight-event-batches = 100
  canton.monitoring.metrics.qualifiers = [debug, traffic, errors, saturation, latency]

// example setting for synchronizers nodes. database sequencer nodes have the exact same settings.
canton.sequencers.sequencer1.sequencer {
    type = BFT
    block {
        writer = {
            // choose between high-throughput or low-latency
            type = high-throughput
        }
    }
}

"bin/canton -Dscala.concurrent.context.numThreads=12 --config examples/01-simple-topology/simple-topology.conf"

"INFO  c.d.canton.environment.Environment - Deriving 12 as number of threads from '-Dscala.concurrent.context.numThreads'."

canton.participants.participant1.ledger-api.postgres-data-source.synchronous-commit = off

  canton.participants.participant1 {
      // tune caching configs of the ledger api server
      ledger-api {
          index-service {
              max-contract-state-cache-size = 1000 // default 1e4
              max-contract-key-state-cache-size = 1000 // default 1e4

              // The in-memory fan-out will serve the transaction streams from memory as they are finalized, rather than
              // using the database. Therefore, you should choose this buffer to be large enough such that the likeliness of
              // applications having to stream transactions from the database is low. Generally, having a 10s buffer is
              // sensible. Therefore, if you expect e.g. a throughput of 20 tx/s, then setting this number to 200 is sensible.
              // The default setting assumes 100 tx/s.
              max-transactions-in-memory-fan-out-buffer-size = 200 // default 1000
          }
      }
      // tune the synchronisation protocols contract store cache
      parameters.caching {
          contract-store {
              maximum-size = 1000 // default 1e6
              expire-after-access = 120s // default 10 minutes
          }
      }
  }

​

Model Tuning

​

Reduce the Number of Views

Computed transaction tree with total=27 for #root-nodes=8

doUpdate : Bool -> Party -> Party -> [ContractId Example] -> Update ()

doUpdate efficient owner obs batch = do
  if efficient then do
    batcher <- create BatchFoo with owner
    -- This only works out if obs is the same observer on all the exercised Example contracts
    batcher `exercise` Run with batch = batch; obs = obs
  else do
    -- Canton will create one view per exercise
    forA_ batch (`exercise` Foo)
  pure ()

template Example with
    owner : Party
    obs: Party
  where
    signatory owner
    observer obs

    choice Foo : ()
      controller owner
      do
        return ()

template BatchFoo with
    owner : Party
  where
    signatory owner

    choice Run : () with
        batch : [ContractId Example]
        obs : Party
      -- The observer here is a choice observer. Therefore, Canton will
      -- ship a single view with the top node of this choice if the observer of the
      -- choice aligns with the observer of the contract.
      observer obs
      controller owner
      do
        forA_ batch (`exercise` Foo)

• create: signatories, observers
• consuming exercise: signatories, observers, stakeholders of the choice (controller, choice observers)
• nonconsuming exercise: signatories, stakeholders of the choice (controller, choice observers), but not the observers of the contract
• fetch: signatories + actors of the fetch, which are all stakholders which are in the authorization context that the fetch executed in
• lookupByKey: only key maintainers

​

Reduce Ledger Events

​

Managing Latency and Throughput

​

Problem Definition

​

Possible Throughput Bottlenecks in Order of Likelihood

1. Transaction size causing high serialization/deserialization and encryption/decryption costs on participant nodes.
2. Transaction size causing sequencer backend overload, especially on blockchains.
3. High interpretation and validation cost due to calculation complexity or memory use.
4. Large number of involved nodes and associated network bandwidth on sequencer.

​

Solutions

1. Minimize transaction size.

• create
• fetch
• fetchByKey
• lookupByKey
• exercise
• exerciseByKey
• archive

template Incrementor
with
p : Party
n : Int
where
signatory p

choice Increment : ContractId Incrementor
    controller p
    do create this with n = n+1

-- This adds all m-1 intermediary versions of
-- the contract to the transaction tree
choice BadIncrementMany : ContractId Incrementor
    with m : Int
    controller p
    do foldlA (\self' _ -> exercise self' Increment) self [1..m]

-- This only adds the end result to the transaction
choice GoodIncrementMany : ContractId Incrementor
    with m : Int
    controller p
    do create this with n = n+m

template Asset
with
issuer : Party
owner : Party
quantity : Decimal
where
signatory [issuer, owner]

-- BadMerge acts on each of the otherCids three times:
-- Once for validation
-- Once to extract the quantities
-- Once to archive
choice BadMerge : ContractId Asset
 with otherCids : [ContractId Asset]
 controller owner
 do
   -- validate the cids.
   forA_ otherCids (\cid -> do
     other <- fetch cid
     assert (other.issuer == issuer && other.owner == owner))

   -- extract the quantities
   quantities <- forA otherCids (\cid -> do
     other <- fetch cid
     return other.quantity)

   -- archive the others
   forA_ otherCids archive

   create this with quantity = quantity + sum quantities

-- Allow us to do a fetch and an archive in one action
choice ConsumingFetch : Asset
 controller owner
 do return this

-- GoodMerge only acts on each of the other assets once.
choice GoodMerge : ContractId Asset
 with otherCids : [ContractId Asset]
 controller owner
 do
   -- Get and archive the others
   others <- forA otherCids (`exercise` ConsumingFetch)

   -- validate
   forA_ others (\other -> do
     assert (other.issuer == issuer && other.owner == owner))

   -- extract the quantities
   let quantities = map (.quantity) others

   create this with quantity = quantity + sum quantities

template T
with
p : Party
where
signatory p

choice Foo : ()
    controller p
    do return ()

batching : Script ()
batching = do
p <- allocateParty "p"

-- without batching we have 10 ledger
-- transactions.
cid1 <- submit p do createCmd T with ..
cid2 <- submit p do createCmd T with ..
cid3 <- submit p do createCmd T with ..
cid4 <- submit p do createCmd T with ..
cid5 <- submit p do createCmd T with ..

submit p do exerciseCmd cid1 Foo
submit p do exerciseCmd cid2 Foo
submit p do exerciseCmd cid3 Foo
submit p do exerciseCmd cid4 Foo
submit p do exerciseCmd cid5 Foo

-- With batching, there are only two ledger transactions.
cids <- submit p do
replicateA 5 $ createCmd T with ..
submit p do
forA_ cids (`exerciseCmd` Foo)

2. CPU and memory issues: Use the Daml profiler to analyze Daml code execution.
3. Once you feel interpretation is not the bottleneck, scale up your machine.

​

Managing Active Contract Set (ACS) Size

​

Problem Definition

​

Relational Databases

• Large ACS size directly affects the resource consumption and performance of a Ledger API client application dealing with a large data set that may not fit into the memory or the application database.
• ACS size directly affects the speed at which the ACS can be transmitted from the Ledger API server using the StateService. In extreme cases, it could take hours to transfer the complete set requested by the application due to the limits imposed by the gRPC channel capacity and the speed of storage queries.
• Increased latency is a less direct impact which shows up wherever a query is issued to the database index to make progress. Large ACS size means that the corresponding indices are also large, and at a certain point they will no longer fit into the shared-buffer space. It then takes increasingly longer for the database engine to produce query results. This affects activities such as contract lookups during the command submission, transaction tree streaming, or pointwise transaction lookups.
• Large ACS size may affect the speed at which the database underpinning the participant ingests new transactions. Normally, as new updates pour in the write-ahead log commits the table and index changes immediately. Those updates come in two shapes; full-page writes or differential writes. With large volumes, many are full-page writes.
• Finally, many dirty pages also translate into prolonged and expensive flushes to the disk as part of the checkpointing process.

​

Solutions

• Pay attention to the lifetime of the contracts. Make sure that the supporting and auxiliary contracts don’t clutter the ACS and archive them as soon as it is practical to do so.
• Set up a frequent pruning schedule. Be aware that pruning is only effective if there are archived contracts available for pruning. If all contracts are still active, pruning has limited success. Refer to our pruning documentation for more information.
• Implement an ODS in your ledger client application to limit reliance on read access to the ACS. Do this whenever you notice that the time to initialize the application from the ACS exceeds your pain level.
• Monitor database performance.
  • Monitor the disk read and write activity. Look for sudden changes in the operation patterns. For instance, a sudden increase in the disk’s read activity may be a sign of indices no longer fitting into the shared buffers.
  • Observe the performance of the database queries. Check our monitoring documentation for query metrics that can assist. You may also consider setting up a log_min_duration_statement parameter in the PostgreSQL configuration.
• Set up autovacuum on the PostgreSQL database. Note that, after pruning, a lot of dead tuples will need removing.

​

Avoid Contention Issues

​

Contention in Daml

1. A client submits a command based on the client’s latest view of the state of the shared ledger. The command might include references to ContractIds that the client believes are active.
2. During interpretation, ledger state is used to look up active contracts.
3. During validation, ledger state is again used to look up contracts and to validate the transaction by reinterpreting it.

• The client is constructing the command in #1 based on contracts it believes to be active. But by the time the participant performs interpretation in #2, it has processed the commit of another transaction that consumed those contracts. The participant node rejects the command due to contention.
• The participant successfully constructs a transaction in #2 based on contracts it believes to be active. But by the time validation happens in #3, another transaction that consumes the same contracts has already been sequenced. The validating participants reject the command due to contention.

​

Reduce Contention

• Add retry logic.
• Run transactions that have causality in series.
• Bundle or batch business logic to increase business transaction throughput.
• Maximize parallelism with techniques such as sharding, while ensuring no contention between shards.
• Split contracts across natural lines to reduce single, high-contention contracts.
• Avoid write-write and write-read contention on contracts. This type of contention occurs when one requester submits a transaction with a consuming exercise on a contract while another requester submits another exercise or a fetch on the same contract. This type of contention cannot be eliminated entirely, since there will always be some latency between a client submitting a command to a participant and other clients learning of the committed transaction. Here are a few scenarios and specific measures you can take to reduce this type of collision:
  • Shard data. Imagine you want to store a user directory on the ledger. At the core, this is of type [(Text, Party)], where Text is a display name and Party is the associated Party. If you store this entire list on a single contract, any two users wanting to update their display name at the same time will cause a collision. If you instead keep each (Text, Party) on a separate contract, these write operations become independent from each other. A helpful analogy when structuring your data is to envision that a template defines a table, where a contract is a row in that table. Keeping large pieces of data on a contract is like storing big blobs in a database row. If these blobs can change through different actions, you have write conflicts.
  • Use non-consuming choices, where possible, as they do not collide. Non-consuming choices can be used to model events that have occurred, so instead of creating a short-lived contract to hold some data that needs to be referenced, that data could be recorded as a ledger event using a non-consuming choice.
  • Avoid workflows that encourage multiple parties to simultaneously exercise a consuming choice on the same contract. For example, imagine an auction contract containing a field highestBid : (Party, Decimal). If Alice tries to bid $100 at the same time that Bob tries to bid $90, it does not matter that Alice’s bid is higher. The sequencer rejects the second because it has a write collision with the first transaction. It is better to record the bids in separate Bid contracts, which can be updated independently. Think about how you would structure this data in a relational database to avoid data loss due to race conditions.
  • Think carefully about storing ContractIds. Imagine that you create a sharded user directory according to the first bullet in this list. Each user has a User contract that stores their display name and party. Now assume that you write a chat application, where each Message contract refers to the sender by ContractId User. If a user changes the display name, that reference goes stale. You either have to modify all messages that the user ever sent, or you cannot use the sender contract in Daml. Contract keys can be used to make this link inside Daml. If the only place you need to link Party to User is in the user interface, it might be best to not store contract references in Daml at all.

​

Example Application with Techniques for Reducing Contention

​

The Example Minimal Settlement System

​

Basic functional requirements for the example application

1. Proposal: Alice offers Bob to swap one share for $1.
2. Acceptance: Bob agrees to the swap.
3. Settlement: The swap is settled atomically, meaning that at the same time Alice transfers $1 to Bob, Bob transfers one share to Alice.

​

Practical and security requirements for the example application

• Parties can hold asset positions of different asset types which they control.
  • An asset position consists of the type, owner, and quantity of the asset.
  • An asset type is usually the combination of an on-ledger issuer and a symbol (such as currency, CUSIP, or ISIN).
• Parties can transfer an asset position (or part of a position) to another party.
• Parties can agree on a settlement consisting of a swap of one position for another.
• Settlement happens atomically.
• There are no double spends.
• It is possible to constrain the total asset position of an owner to be non-negative. In other words, it is possible to ensure that settlements are funded. The total asset position is the sum of the quantities of all assets of a given type by that owner.

​

Performance measurement in the example application

• Settlement latency: the time it takes from one party wanting to settle (just before the proposal step) to the time that party receives final confirmation that the settlement was committed (after the settlement step). For this example, assume that the best possible path occurs and that parties take zero time to make decisions.
• Settlement throughput: the maximum number of settlements per second that the system as a whole can process over a long period.
• Transaction latency: the time it takes from when a client application submits a command or transaction to the ledger to the time it receives the commit confirmation. The length of time depends on the command. A transaction settling a batch of 100 settlements will take longer than a transaction settling a single swap. For this example, assume that transaction latency has a simple formula of a fixed cost fixed_tx and a variable processing cost of var_tx times the number of settlements, as shown here: transaction latency = fixed_tx + (var_tx * #settlements)
• Note that the example application does not assign any latency cost to settlement proposals and acceptances.
• For the example application, assume that:
  • fixed_tx = 250ms
  • var_tx = 10ms

(2 * fixed_tx) + var_tx

1000ms / (fixed_tx + var_tx) settlements per second

dlt_fixed_tx = 1 dlt_min_tx

dlt_var_tx = ratio * dlt_fixed_tx

dlt_fixed_tx = 1 dlt_min_tx

dlt_var_tx = 0.04 * dlt_min_tx

max_throughput = 10 * each CPU’s capacity

max_throughput = 10 * 100 dlt_min_tx/second

max_transaction = 3 * dlt_min_tx

(2 * dlt_min_tx)/(0.04 * dlt_min_tx)

(2 * dlt_min_tx) + dlt_var_tx

1,000/2.04 settlements per second

​

Prepare Transactions for Contention-Free Parallelism

1. Interpretation: the submitting party precalculates the transaction, which consists of input and output contracts.
2. Submission: the submitting party submits the transaction to the network.
3. Sequencing: the consensus algorithm for the network assigns the transaction a place in the total order of all transactions.
4. Validation: the transaction is validated and considered valid if none of the inputs were already spent by a previous transaction.
5. Commitment: the transaction is committed.
6. Response: the submitting party receives a response that the transaction was committed.

max_transaction > dlt_fixed_tx + 1,000 * dlt_var_tx = 41 dlt_min_tx

fixed_tx + 1,000 * var_tx = 10.25s

• 10 * $100
• 100 * $10
• 1,000 * $1

3 * fixed_tx + (fixed_tx + var_tx) = 1.01s

0.75s + (1,000 * (dlt_fixed_tx + dlt_var_tx)) / 1,000 dlt_min_tx/s = 1.79s

• There are constant incoming settlement requests, which you have limited ability to predict. Treat this as an infinite stream of random settlements from some distribution and maximize settlement throughput with reasonable latency.
• Not all settlements are successful, due to withdrawals, rejections, and business errors.

1. Proposal: proposal of the settlement
2. Acceptance: acceptance of the settlement
3. Settlement: actual settlement

​

Non-UTXO Alternative Ledger Models

1. Submission: the submitting party submits a command to the network.
2. Sequencing: the consensus algorithm of the network assigns the command a place in the total order of all commands.
3. Validation: the command is evaluated to a transaction and then validated.
4. Response: the submitting party receives a response about the effect of the command.

​

Simple Strategies for UTXO Ledger Models

​

Batch transactions sequentially

fixed_tx + N * var_tx

N / (fixed_tx + N * var_tx)

​

Use sequential processing or batching per asset type and owner

template AllAssets
  with
    -- A map from owner and type to quantity
    holdings : Map Party (Map AssetType Decimal)
  where
    signatory (keys holdings)

template
  with
    assetType : AssetType
    owner : Party
    quantity : Decimal
  where
    signatory assetType.issuer, owner

1. Buy-side allocation: the buy-side splits out an asset position from their total asset position and allocates it to the settlement.
2. Sell-side allocation: the sell-side splits out an asset position from their total asset position and allocates it to the settlement.
3. Settlement: the asset positions change ownership.
4. Buy-side merge: the buy-side merges their new position back into the total asset position.
5. Sell-side merge: the sell-side merges their new position back into the total asset position.

• Buy-side allocation is usually done as part of a settlement proposal.
• Sell-side allocation is typically handled as part of the settlement.
• Buy-side merge and sell-side merge technically do not need any action. By definition of total asset positions, merging is an optional step. It is easy to keep things organized without extra transactions. Every time a total asset position is touched as part of buy-side allocation or sell-side allocation above, you merge all positions into a single one. As long as there is a similar amount of inbound and outbound traffic on the total asset position, the number of individual positions stays low.

​

Shard Asset Positions for UTXO Ledger Models

​

Shard total asset positions without global constraints

​

Shard total asset positions with global constraints

• The automation keeps track of a synchronized pending set of asset positions, which marks asset positions that are in use.
• Every time the automation triggers (which may happen concurrently), it looks at how many asset positions there are relative to the desired 100 and how much quantity is needed to allocate the open settlements.
• It then selects an appropriate set of non-pending asset positions so that it can allocate the open settlements and return new asset positions to move the total number closer to 100.
• Before sending the transaction, it adds those positions to the pending set to make sure that another thread does not also use them.

• Keeping the total number of asset positions limited so that the number of active contracts does not impact system performance.
• Having sufficient large asset positions so that frequent small settlements can be processed in parallel.
• Having a mechanism that ensures large settlements, possibly requiring as much as 100% of the available total asset position, are not blocked.

​

Application development performance practice

​

On-Ledger vs. Off-Ledger

• State — Essential facts in the workflow. When these facts require agreement across organizations, store them on-ledger.
• Data — Non-essential facts. For efficiency, use off-ledger technologies for sharing non-critical data.
• Workflows — Only workflows that cross organizational boundaries should be on-ledger.

​

Contract Design for Performance

​

Avoid contract fan-out

-- Avoid: creates N contracts in one transaction
choice ProcessBatch : [ContractId Item]
  controller processor
  do mapA (\item -> create Item with ...) items

-- Better: one contract representing the batch
choice ProcessBatch : ContractId BatchResult
  controller processor
  do create BatchResult with items = processedItems

​

Minimize stakeholders per transaction

​

Use contract keys judiciously

​

PQS Query Optimization

​

Creating indexes

   call create_index_for_contract('token_wallet_holder_idx',
                                  'register:DA.Register:Token',
                                  '(payload->''wallet''->>''holder'')',
                                  'hash');

​

PostgreSQL tuning

• shared_buffers — Set to 25% of available RAM
• effective_cache_size — Set to 75% of available RAM
• max_connections — Match to your expected connection pool size
• autovacuum settings — Tune for your write patterns

​

Query patterns

• Use explain analyze to verify that queries use your indexes
• Prefer filtering in SQL over fetching all contracts and filtering in application code
• For time-series queries, use the event tables rather than the active contract set
• Consider materialized views for frequently-run aggregation queries

​

Parallel Command Submission

• Use async/concurrent command submission from your backend
• Assign unique command IDs to enable deduplication
• Group commands by the contracts they affect to avoid contention
• Monitor command completion times to detect bottleneck patterns

​

Traffic Management

• Reducing the number and size of on-ledger transactions
• Using batch operations where possible
• Configuring auto-top-up to avoid running out of traffic during peak usage
• Monitoring traffic consumption patterns to forecast costs