The Linux Kernel Logo
  • Development process
  • Submitting patches
  • Code of conduct
  • Maintainer handbook
  • All development-process docs
  • Core API
  • Driver APIs
  • Subsystems
    • Core subsystems
    • Human interfaces
    • Networking interfaces
      • Networking
        • AF_XDP
        • Bare UDP Tunnelling Module Documentation
        • batman-adv
        • SocketCAN - Controller Area Network
        • The UCAN Protocol
        • Hardware Device Drivers
        • Networking Diagnostics
        • Distributed Switch Architecture
        • Linux Devlink Documentation
          • Locking
          • Nested instances
          • Interface documentation
            • Devlink DPIPE
            • Devlink Health
              • Background
              • Overview
              • Actions
              • Devlink formatted message
              • User Interface
            • Devlink Info
            • Devlink Flash
            • Devlink Params
            • Devlink Port
            • Devlink Region
            • Devlink Resource
            • Devlink Reload
            • Devlink Selftests
            • Devlink Trap
            • Devlink Line card
            • Devlink E-Switch Attribute
          • Driver-specific documentation
        • CAIF
        • Netlink interface for ethtool
        • IEEE 802.15.4 Developer’s Guide
        • ISO 15765-2 (ISO-TP)
        • J1939 Documentation
        • Linux Networking and Network Devices APIs
        • MSG_ZEROCOPY
        • FAILOVER
        • Net DIM - Generic Network Dynamic Interrupt Moderation
        • NET_FAILOVER
        • Page Pool API
        • PHY Abstraction Layer
        • phylink
        • IP-Aliasing
        • Ethernet Bridging
        • SNMP counter
        • Checksum Offloads
        • Segmentation Offloads
        • Scaling in the Linux Networking Stack
        • Kernel TLS
        • Kernel TLS offload
        • In-Kernel TLS Handshake
        • Linux NFC subsystem
        • Netdev private dataroom for 6lowpan interfaces
        • 6pack Protocol
        • ARCnet Hardware
        • ARCnet
        • ATM
        • AX.25
        • Linux Ethernet Bonding Driver HOWTO
        • cdc_mbim - Driver for CDC MBIM Mobile Broadband modems
        • DCCP protocol
        • DCTCP (DataCenter TCP)
        • Device Memory TCP
        • DNS Resolver Module
        • Softnet Driver Issues
        • EQL Driver: Serial IP Load Balancing HOWTO
        • LC-trie implementation notes
        • Linux Socket Filtering aka Berkeley Packet Filter (BPF)
        • Generic HDLC layer
        • Generic Netlink
        • Netlink Family Specifications
        • Generic networking statistics for netlink users
        • The Linux kernel GTP tunneling module
        • Identifier Locator Addressing (ILA)
        • IOAM6 Sysfs variables
        • io_uring zero copy Rx
        • IP dynamic address hack-port v0.03
        • IPsec
        • IP Sysctl
        • IPv6
        • IPVLAN Driver HOWTO
        • IPvs-sysctl
        • Kernel Connection Multiplexor
        • L2TP
        • The Linux LAPB Module Interface
        • How to use packet injection with mac80211
        • Management Component Transport Protocol (MCTP)
        • MPLS Sysfs variables
        • Multipath TCP (MPTCP)
        • MPTCP Sysfs variables
        • HOWTO for multiqueue network device support
        • Multi-PF Netdev
        • NAPI
        • Common Networking Struct Cachelines
        • Netconsole
        • Netdev features mess and how to get out from it alive
        • Network Devices, the Kernel, and You!
        • Netfilter Sysfs variables
        • NETIF Msg Level
        • Netmem Support for Network Drivers
        • Resilient Next-hop Groups
        • Netfilter Conntrack Sysfs variables
        • Netfilter’s flowtable infrastructure
        • OPEN Alliance 10BASE-T1x MAC-PHY Serial Interface (TC6) Framework Support
        • Open vSwitch datapath developer documentation
        • Operational States
        • Packet MMAP
        • Linux Phonet protocol family
        • PHY link topology
        • HOWTO for the linux packet generator
        • PLIP: The Parallel Line Internet Protocol Device
        • PPP Generic Driver and Channel Interface
        • The proc/net/tcp and proc/net/tcp6 variables
        • Power Sourcing Equipment (PSE) Documentation
        • How to use radiotap headers
        • RDS
        • Linux wireless regulatory documentation
        • Network Function Representors
        • RxRPC Network Protocol
        • SOCKET OPTIONS
        • SECURITY
        • EXAMPLE CLIENT USAGE
        • Linux Kernel SCTP
        • LSM/SeLinux secid
        • Seg6 Sysfs variables
        • struct sk_buff
        • SMC Sysctl
        • NIC SR-IOV APIs
        • Interface statistics
        • Stream Parser (strparser)
        • Ethernet switch device driver model (switchdev)
        • Sysfs tagging
        • TC Actions - Environmental Rules
        • TC queue based filtering
        • TCP Authentication Option Linux implementation (RFC5925)
        • Thin-streams and TCP
        • Team
        • Timestamping
        • Linux Kernel TIPC
        • Transparent proxy support
        • Universal TUN/TAP device driver
        • The UDP-Lite protocol (RFC 3828)
        • Virtual Routing and Forwarding (VRF)
        • Virtual eXtensible Local Area Networking documentation
        • Linux X.25 Project
        • X.25 Device Driver Interface
        • XFRM device - offloading the IPsec computations
        • XFRM proc - /proc/net/xfrm_* files
        • XFRM
        • XFRM Syscall
        • XDP RX Metadata
        • AF_XDP TX Metadata
      • NetLabel
      • InfiniBand
      • ISDN
      • MHI
    • Storage interfaces
    • Other subsystems
  • Locking
  • Licensing rules
  • Writing documentation
  • Development tools
  • Testing guide
  • Hacking guide
  • Tracing
  • Fault injection
  • Livepatching
  • Rust
  • Administration
  • Build system
  • Reporting issues
  • Userspace tools
  • Userspace API
  • Firmware
  • Firmware and Devicetree
  • CPU architectures
  • Unsorted documentation
  • Translations
The Linux Kernel
  • Kernel subsystem documentation
  • Networking
  • Linux Devlink Documentation
  • Devlink Health
  • View page source

Devlink Health¶

Background¶

The devlink health mechanism is targeted for Real Time Alerting, in order to know when something bad happened to a PCI device.

  • Provide alert debug information.

  • Self healing.

  • If problem needs vendor support, provide a way to gather all needed debugging information.

Overview¶

The main idea is to unify and centralize driver health reports in the generic devlink instance and allow the user to set different attributes of the health reporting and recovery procedures.

The devlink health reporter: Device driver creates a “health reporter” per each error/health type. Error/Health type can be a known/generic (e.g. PCI error, fw error, rx/tx error) or unknown (driver specific). For each registered health reporter a driver can issue error/health reports asynchronously. All health reports handling is done by devlink. Device driver can provide specific callbacks for each “health reporter”, e.g.:

  • Recovery procedures

  • Diagnostics procedures

  • Object dump procedures

  • Out Of Box initial parameters

Different parts of the driver can register different types of health reporters with different handlers.

Actions¶

Once an error is reported, devlink health will perform the following actions:

  • A log is being send to the kernel trace events buffer

  • Health status and statistics are being updated for the reporter instance

  • Object dump is being taken and saved at the reporter instance (as long as auto-dump is set and there is no other dump which is already stored)

  • Auto recovery attempt is being done. Depends on:

    • Auto-recovery configuration

    • Grace period vs. time passed since last recover

Devlink formatted message¶

To handle devlink health diagnose and health dump requests, devlink creates a formatted message structure devlink_fmsg and send it to the driver’s callback to fill the data in using the devlink fmsg API.

Devlink fmsg is a mechanism to pass descriptors between drivers and devlink, in json-like format. The API allows the driver to add nested attributes such as object, object pair and value array, in addition to attributes such as name and value.

Driver should use this API to fill the fmsg context in a format which will be translated by the devlink to the netlink message later. When it needs to send the data using SKBs to the netlink layer, it fragments the data between different SKBs. In order to do this fragmentation, it uses virtual nests attributes, to avoid actual nesting use which cannot be divided between different SKBs.

User Interface¶

User can access/change each reporter’s parameters and driver specific callbacks via devlink, e.g per error type (per health reporter):

  • Configure reporter’s generic parameters (like: disable/enable auto recovery)

  • Invoke recovery procedure

  • Run diagnostics

  • Object dump

List of devlink health interfaces¶

Name

Description

DEVLINK_CMD_HEALTH_REPORTER_GET

Retrieves status and configuration info per DEV and reporter.

DEVLINK_CMD_HEALTH_REPORTER_SET

Allows reporter-related configuration setting.

DEVLINK_CMD_HEALTH_REPORTER_RECOVER

Triggers reporter’s recovery procedure.

DEVLINK_CMD_HEALTH_REPORTER_TEST

Triggers a fake health event on the reporter. The effects of the test event in terms of recovery flow should follow closely that of a real event.

DEVLINK_CMD_HEALTH_REPORTER_DIAGNOSE

Retrieves current device state related to the reporter.

DEVLINK_CMD_HEALTH_REPORTER_DUMP_GET

Retrieves the last stored dump. Devlink health saves a single dump. If an dump is not already stored by devlink for this reporter, devlink generates a new dump. Dump output is defined by the reporter.

DEVLINK_CMD_HEALTH_REPORTER_DUMP_CLEAR

Clears the last saved dump file for the specified reporter.

The following diagram provides a general overview of devlink-health:

                                               netlink
                                      +--------------------------+
                                      |                          |
                                      |            +             |
                                      |            |             |
                                      +--------------------------+
                                                   |request for ops
                                                   |(diagnose,
  driver                               devlink     |recover,
                                                   |dump)
+--------+                            +--------------------------+
|        |                            |    reporter|             |
|        |                            |  +---------v----------+  |
|        |   ops execution            |  |                    |  |
|     <----------------------------------+                    |  |
|        |                            |  |                    |  |
|        |                            |  + ^------------------+  |
|        |                            |    | request for ops     |
|        |                            |    | (recover, dump)     |
|        |                            |    |                     |
|        |                            |  +-+------------------+  |
|        |     health report          |  | health handler     |  |
|        +------------------------------->                    |  |
|        |                            |  +--------------------+  |
|        |     health reporter create |                          |
|        +---------------------------->                          |
+--------+                            +--------------------------+
Previous Next

© Copyright The kernel development community.

Built with Sphinx using a theme provided by Read the Docs.