Documentation Index

Fetch the complete documentation index at: https://docs.canton.network/llms.txt

Use this file to discover all available pages before exploring further.

​

Monitoring Best Practices

​

Introduction

• Read the shortcut to learning what is important, which is described below in the section Hands-On with the Daml Enterprise - Observability Example as a starting point and inspiration when building your metric monitoring.
• For an overview of how most metrics are exposed, read the section Golden Signals and Key Metrics Quick Start below. It describes the philosophy behind metric naming and labeling.

​

Hands-On with the Daml Enterprise - Observability Example

• An example Docker compose file to create a run-time for all the components
• Some shell scripts to generate requests to the Daml Enterprise solution
• A Prometheus config file to scrape the metrics data
• A Grafana template file(s) to visualize the metrics in a meaningful way, such as shown below in the example dashboard

​

Golden Signals and Key Metrics Quick Start

• Input request rate, as a counter
• Error rate, as a counter (discussed below)
• Latency (the time to process a request), as a histogram
• Size of the payload, as a counter, following the Apache HTTP precedent

• http_verb: the HTTP verb (for example: GET, POST)
• http_status: the status code (for example: 200, 401, 403, 504)
• host: the host identifier
• daml_version: the Daml release number
• service: a string to identify what Daml service or Canton component is running in this process (for example: participant, sequencer, json_api)
• path: the request made to the endpoint (for example: /v2/commands/submit-and-wait, /v2/state/active-contracts)

• canton_version: the Canton protocol version
• grpc_code: the human-readable status code for gRPC (for example: OK, CANCELLED, DEADLINE_EXCEEDED)
• The type of the client/server gRPC request, under the labels grpc_client_type and grpc_server_type
• The protobuf package and service names, under the labels grpc_service_name and grpc_method_name

• A binary gauge indicates whether the node is healthy or not healthy. This can also be used to infer which node is passive in a highly available configuration because it will show as not being healthy, while the active node is always healthy.
• A binary gauge signals whether a node is active or passive, for identifying which node is the active node.
• A binary gauge detects when pruning is occurring.
• Each participant node measures the count of the inflight (dirty) requests so the user can see if maxDirtyRequests limit is close to being hit. The metrics are: canton_dirty_requests and canton_max_dirty_requests.
• Each participant node records the distribution of events (updates) received by the participant and allows drill-down by event type (package upload, party creation, or transaction), status (success or failure), participant ID, and application ID (if available). The counter is called daml_indexer_events_total.
• The ledger event requests are totaled in a counter called daml_indexer_metered_events_total.
• JVM garbage collection metrics are collected.

​

Set Up Metrics Scraping

​

Enable the Prometheus Reporter

canton.monitoring.metrics.reporters = [{
  type = prometheus
  address = "localhost" // default
  port = 9000 // default
}]

​

Prometheus-Only Metrics

​

Deprecated Reporters

canton.monitoring.metrics.reporters = [{ type = jmx }]

canton.monitoring.metrics.reporters = [{
  type = jmx
}, {
  type = csv
  directory = "metrics"
  interval = 5s // default
  filters = [{
    contains = "canton"
  }]
}]

canton.monitoring.metrics.reporters = [{
  type = graphite
  address = "localhost" // default
  port = 2003
  prefix.type = hostname // default
  interval = 30s // default
  filters = [{
    contains = "canton"
  }]
}]

​

Metrics

​

Logging

• -v (or --verbose) is a short option to set the Canton log level to DEBUG. This is likely the most common log option you will use.
• --debug sets all log levels except stdout to DEBUG. Stdout is set to INFO. Note that DEBUG logs of external libraries can be very noisy.
• --log-level-root=<level> configures the log-level of the root logger. This changes the log level of Canton and of external libraries, but not of stdout.
• --log-level-canton=<level> configures the log-level of only the Canton logger.
• --log-level-stdout=<level> configures the log-level of stdout. This will usually be the text displayed in the Canton console.
• --log-file-name=log/canton.log configures the location of the log file.
• --log-file-appender=flat|rolling|off configures if and how logging to a file should be done. The rolling appender will roll the files according to the defined date-time pattern.
• --log-file-rolling-history=12 configures the number of historical files to keep when using the rolling appender.
• --log-file-rolling-pattern=YYYY-mm-dd configures the rolling file suffix (and therefore the frequency) of how files should be rolled.
• --log-truncate configures whether the log file should be truncated on startup.
• --log-profile=container provides a default set of logging settings for a particular setup. Only the container profile is supported, which logs to both STDOUT and to 10-hour limited rolling log files history (to avoid storage leaks).
• --log-immediate-flush=false turns off immediate flushing of the log output to the log file.

JAVA_OPTS="-Dlogback.configurationFile=./path-to-file.xml" ./bin/canton --config ...

​

Viewing Logs

• Viewing log files of different Canton components in a single view, merged according to timestamps (lnav <log1> <log2> ...).
• Filtering specific log messages in (:filter-in <regex>) or out (:filter-out <regex>). When filtering messages (for example, with a given trace-id), a transaction can be traced across different components, especially when using the single-view-feature described earlier.
• Searching for specific log messages (/<regex>) and jumping between them (n and N).
• Automatic syntax highlighting of parts of log messages (such as timestamps) and log messages themselves (for example, WARN log messages are yellow).
• Jumping between error (e and E) and warn messages (w and W).
• Selectively activating and deactivating different filters and files (TAB and “ to activate/deactivate a filter).
• Marking lines (m) and jumping back and forth between marked lines (u and U).
• Jumping back and forth between lines that have the same trace-id (o and O).

​

Detailed Logging

canton.monitoring.logging {
    event-details = true
    api {
        message-payloads = true
        max-method-length = 1000
        max-message-lines = 10000
        max-string-length = 10000
        max-metadata-size = 10000
    }
}

​

Tracing

monitoring.tracing.tracer.exporter {
  type = jaeger
  address = ... // default: "localhost"
  port = ... // default: 14250
}

​

Sampling

monitoring.tracing.tracer.sampler = {
  type = trace-id-ratio
  ratio = 0.5
}

​

Known Limitations

​

Traces

1. The admin party for participant1 creates a Ping contract.
2. The admin party for participant2 exercises the Respond consuming choice on the contract, which results in the creation of a Pong contract.
3. The admin party for participant1 exercises the Ack consuming choice on it.

• The synchronizer routing where the participant decides which synchronizer to use for the command submission.
• The preparation of the initial set of messages to be sent.

​

Node Health Status

<node>.health.status

• Failure: if the status of the node cannot be determined, including an error message of why it failed
• NotInitialized: if the node is not yet initialized
• Success[NodeStatus]: if the status could be determined, including the detailed status

• Participant id: the participant id of the node
• Uptime: the uptime of this node
• Ports: the ports on which the participant node exposes the Ledger and the Admin API.
• Connected synchronizers: the list of synchronizers to which the participant is properly connected
• Unhealthy synchronizers: the list of synchronizers to which the participant is trying to connect, but the connection is not ready for command submission
• Active: true if this instance is the active replica (It can be false in the case of the passive instance of a high-availability deployment.)

• Synchronizer id: the unique identifier of the synchronizer
• Uptime: the uptime of this node
• Ports: the ports on which the synchronizer exposes the Public and the Admin API
• Connected Participants: the list of connected participants
• Sequencer: a boolean flag indicating whether the embedded sequencer writer is operational

• Accepts admin changes: a boolean flag indicating whether the sequencer accepts admin changes

• Node uid: the unique identifier of the node
• Uptime: the uptime of this node
• Ports: the ports on which the node hosts its APIs
• Active: true if this instance is the active replica (It can be false in the case of the passive instance of a high-availability deployment.)

​

Health Checks

​

gRPC Health Check Service

monitoring.grpc-health-server {
  address = "127.0.0.1"
  port = 5861
}

​

HTTP Health Check

canton {
  monitoring.health {
   server {
      port = 7000
   }

   check {
     type = ping
     participant = participant1
     interval = 30s
   }
}

​

Health Dumps

• Start Canton in interactive mode, with the -v option to enable debug logging: ./bin/canton -v -c <myconfig>. This provides a console prompt.
• Reproduce the error by following the steps that previously caused the error. Write down these steps so they can be provided to support staff.
• After you observe the error, type health.dump() into the Canton console to generate a ZIP file.

• The configuration you are using, with all sensitive data stripped from it (no passwords).
• An extract of the log file. Sensitive data is not logged into log files.
• A current snapshot on Canton metrics.
• A stacktrace for each running thread.

​

Remote Health Dumps

​

Configure Metrics

​

Export Metrics for Scraping

canton.monitoring.metrics {
    jvm-metrics.enabled = yes
    reporters = [{
        type = prometheus
        address = 0.0.0.0
        // This will expose the prometheus metrics on port 9000
        port = 9000
    }]
}

​

Deprecated Reporters

canton.monitoring.metrics.reporters = [{ type = jmx }]

canton.monitoring.metrics.reporters = [{
  type = jmx
}, {
  type = csv
  directory = "metrics"
  interval = 5s // default
  filters = [{
    contains = "canton"
  }]
}]

canton.monitoring.metrics.reporters = [{
  type = graphite
  port = 2003
  interval = 30s // default
  filters = [{
    contains = "canton"
  }]
}]