-
Notifications
You must be signed in to change notification settings - Fork 237
Add BMCWatchdog base class to be used in SONiC BMC #702
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
chander-nexthop
wants to merge
1
commit into
sonic-net:master
Choose a base branch
from
nexthop-ai:chander.bmc-watchdog
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+321
−0
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,170 @@ | ||
| """ | ||
| bmc_watchdog.py | ||
|
|
||
| BMC-backed implementation of the SONiC platform watchdog API. | ||
|
|
||
| On BMC-based platforms the hardware watchdog device (/dev/watchdog0) can | ||
| only be opened by a single process at a time. A hw-watchdog-mgrd daemon | ||
| owns the device and pets it periodically, implementing the classic Linux | ||
| watchdog semantics. This class therefore does not access the device | ||
| directly; instead it talks to the daemon over a Unix domain socket so that | ||
| watchdogutil arm/disarm/status work while the daemon keeps the watchdog | ||
| alive. | ||
| """ | ||
|
|
||
| import json | ||
| import os | ||
| import socket | ||
| import syslog | ||
|
|
||
| from sonic_platform_base.watchdog_base import WatchdogBase | ||
|
|
||
|
|
||
| # Default IPC socket served by the hw-watchdog-mgrd daemon. This must match | ||
| # SOCKET_PATH in the platform's hw-watchdog-mgrd.py daemon. | ||
| SOCKET_PATH = "/run/hw-watchdog-mgrd.sock" | ||
| SOCKET_TIMEOUT = 5 # seconds | ||
|
|
||
| # Default read-only sysfs view of the hardware watchdog state. Used only as a | ||
| # fallback for is_armed() when the daemon is unreachable; reading sysfs does not | ||
| # open /dev/watchdog0 and so does not contend with the daemon. | ||
| WATCHDOG_SYSFS_PATH = "/sys/class/watchdog/watchdog0/" | ||
|
|
||
|
|
||
| class BMCWatchdog(WatchdogBase): | ||
| """ | ||
| BMC-backed watchdog implementation. | ||
|
|
||
| Acts as an IPC client to the hw-watchdog-mgrd daemon, which is the sole | ||
| owner of the hardware watchdog device. | ||
| """ | ||
|
|
||
| def __init__(self, socket_path=SOCKET_PATH, sysfs_path=WATCHDOG_SYSFS_PATH): | ||
| """ | ||
| Initialize the BMCWatchdog object | ||
|
|
||
| Args: | ||
| socket_path: Path to the Unix domain socket served by the | ||
| hw-watchdog-mgrd daemon. Defaults to SOCKET_PATH. | ||
| sysfs_path: Path to the read-only sysfs directory of the hardware | ||
| watchdog, used as the is_armed() fallback when the daemon is | ||
| unreachable. Defaults to WATCHDOG_SYSFS_PATH. | ||
| """ | ||
| self.socket_path = socket_path | ||
| self.sysfs_path = sysfs_path | ||
|
|
||
| def _request(self, cmd, **kwargs): | ||
| """ | ||
| Send a request to the hw-watchdog-mgrd daemon and return the response dict. | ||
|
|
||
| Returns None on any communication failure. | ||
| """ | ||
| req = {"cmd": cmd} | ||
| req.update(kwargs) | ||
| try: | ||
| sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) | ||
| sock.settimeout(SOCKET_TIMEOUT) | ||
| sock.connect(self.socket_path) | ||
| sock.sendall((json.dumps(req) + "\n").encode()) | ||
| data = sock.recv(4096) | ||
| sock.close() | ||
| except (OSError, socket.timeout) as e: | ||
| self._log_daemon_unreachable(e) | ||
| return None | ||
| if not data: | ||
| return None | ||
| try: | ||
| return json.loads(data.decode().strip()) | ||
| except ValueError: | ||
| return None | ||
|
|
||
| def _log_daemon_unreachable(self, err): | ||
| """Emit a clear diagnostic when the hw-watchdog-mgrd daemon cannot be reached.""" | ||
| syslog.syslog( | ||
| syslog.LOG_WARNING, | ||
| "watchdog: hw-watchdog-mgrd daemon not reachable at %s (%s); " | ||
| "is the hw-watchdog-mgrd service running?" | ||
| % (self.socket_path, err)) | ||
|
|
||
| def _sysfs_is_armed(self): | ||
| """ | ||
| Read the hardware watchdog state directly from sysfs. | ||
|
|
||
| Fallback for is_armed() when the daemon is unreachable so that status | ||
| still reflects the hardware (e.g. the daemon crashed while the watchdog | ||
| was armed and is now counting down to a reset). | ||
|
|
||
| This fallback is meaningful because of how a typical BMC watchdog | ||
| driver (e.g. aspeed_wdt) behaves: | ||
| * It advertises WDIOF_MAGICCLOSE and the kernel runs with nowayout=0, | ||
| so a clean stop needs the magic 'V' close (which the daemon writes | ||
| on SIGTERM) but an *unclean* close (a daemon crash) leaves the | ||
| watchdog running and counting down. | ||
| * Its max_hw_heartbeat_ms is far larger than the timeouts we use, so | ||
| the watchdog core does not start its own keepalive worker; after a | ||
| crash nothing re-pets the device and it really does reset the box. | ||
| In that crash window /sys/class/watchdog/watchdog0/state still reads | ||
| "active", so this is the only way is_armed() can report the truth | ||
| instead of a dangerous False until systemd restarts the daemon (which | ||
| then re-adopts the live watchdog). Reading sysfs does not open | ||
| /dev/watchdog0, so it never contends with the daemon. | ||
| """ | ||
| try: | ||
| with open(os.path.join(self.sysfs_path, "state")) as f: | ||
| return f.read().strip() == "active" | ||
| except OSError: | ||
| return False | ||
|
|
||
| def is_armed(self): | ||
| """ | ||
| Retrieves the armed state of the hardware watchdog | ||
|
|
||
| Returns: | ||
| A boolean, True if watchdog is armed, False if not | ||
| """ | ||
| resp = self._request("is_armed") | ||
| if resp is None: | ||
| # Daemon unreachable; fall back to the read-only sysfs state. | ||
| return self._sysfs_is_armed() | ||
| return bool(resp.get("result", False)) | ||
|
|
||
| def arm(self, seconds): | ||
| """ | ||
| Arm the hardware watchdog with a timeout of <seconds> seconds | ||
|
|
||
| Args: | ||
| seconds: Timeout value in seconds | ||
|
|
||
| Returns: | ||
| An integer specifying the actual number of seconds the watchdog | ||
| was armed with. On failure returns -1. | ||
| """ | ||
| resp = self._request("arm", seconds=seconds) | ||
| if resp is None or "result" not in resp: | ||
| return -1 | ||
| return int(resp["result"]) | ||
|
|
||
| def disarm(self): | ||
| """ | ||
| Disarm the hardware watchdog | ||
|
|
||
| Returns: | ||
| A boolean, True if watchdog is disarmed successfully, False if not | ||
| """ | ||
| resp = self._request("disarm") | ||
| if resp is None: | ||
| return False | ||
| return bool(resp.get("result", False)) | ||
|
|
||
| def get_remaining_time(self): | ||
| """ | ||
| Get the number of seconds remaining on the watchdog timer | ||
|
|
||
| Returns: | ||
| An integer specifying the number of seconds remaining on the | ||
| watchdog timer. If the watchdog is not armed, returns -1. | ||
| """ | ||
| resp = self._request("get_remaining_time") | ||
| if resp is None or "result" not in resp: | ||
| return -1 | ||
| return int(resp["result"]) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,151 @@ | ||
| ''' | ||
| Test BMCWatchdog module | ||
| ''' | ||
|
|
||
| import json | ||
| import socket | ||
| import sys | ||
| from unittest import mock | ||
|
|
||
| import pytest | ||
|
|
||
| try: | ||
| from sonic_py_common import logger | ||
| except ImportError: | ||
| sys.modules['sonic_py_common'] = mock.MagicMock() | ||
| sys.modules['sonic_py_common.logger'] = mock.MagicMock() | ||
|
|
||
| from sonic_platform_base.bmc_watchdog import BMCWatchdog | ||
|
|
||
|
|
||
| def make_sock_mock(response): | ||
| ''' | ||
| Build a mock socket whose recv() returns the JSON-encoded `response`. | ||
| Returns (sock_mock, sent) where `sent` collects sendall() payloads. | ||
| ''' | ||
| sock = mock.MagicMock() | ||
| sent = [] | ||
| sock.sendall.side_effect = lambda data: sent.append(data) | ||
| if response is None: | ||
| sock.recv.return_value = b"" | ||
| else: | ||
| sock.recv.return_value = (json.dumps(response) + "\n").encode() | ||
| return sock, sent | ||
|
|
||
|
|
||
| class TestBMCWatchdog: | ||
| def test_init_defaults(self): | ||
| wd = BMCWatchdog() | ||
| assert wd.socket_path == "/run/hw-watchdog-mgrd.sock" | ||
| assert wd.sysfs_path == "/sys/class/watchdog/watchdog0/" | ||
|
|
||
| def test_init_custom_paths(self): | ||
| wd = BMCWatchdog(socket_path="/tmp/wd.sock", sysfs_path="/tmp/wd/") | ||
| assert wd.socket_path == "/tmp/wd.sock" | ||
| assert wd.sysfs_path == "/tmp/wd/" | ||
|
|
||
| @mock.patch("sonic_platform_base.bmc_watchdog.socket.socket") | ||
| def test_request_sends_command(self, mock_socket): | ||
| sock, sent = make_sock_mock({"result": True}) | ||
| mock_socket.return_value = sock | ||
| wd = BMCWatchdog(socket_path="/tmp/wd.sock") | ||
|
|
||
| resp = wd._request("arm", seconds=30) | ||
|
|
||
| assert resp == {"result": True} | ||
| sock.connect.assert_called_once_with("/tmp/wd.sock") | ||
| assert json.loads(sent[0].decode().strip()) == {"cmd": "arm", "seconds": 30} | ||
|
|
||
| @mock.patch("sonic_platform_base.bmc_watchdog.socket.socket") | ||
| def test_request_connection_error_returns_none(self, mock_socket): | ||
| sock = mock.MagicMock() | ||
| sock.connect.side_effect = OSError("no daemon") | ||
| mock_socket.return_value = sock | ||
| wd = BMCWatchdog() | ||
| assert wd._request("is_armed") is None | ||
|
|
||
| @mock.patch("sonic_platform_base.bmc_watchdog.socket.socket") | ||
| def test_request_invalid_json_returns_none(self, mock_socket): | ||
| sock = mock.MagicMock() | ||
| sock.recv.return_value = b"not-json\n" | ||
| mock_socket.return_value = sock | ||
| wd = BMCWatchdog() | ||
| assert wd._request("is_armed") is None | ||
|
|
||
| @mock.patch("sonic_platform_base.bmc_watchdog.socket.socket") | ||
| def test_arm(self, mock_socket): | ||
| sock, _ = make_sock_mock({"result": 30}) | ||
| mock_socket.return_value = sock | ||
| assert BMCWatchdog().arm(30) == 30 | ||
|
|
||
| @mock.patch("sonic_platform_base.bmc_watchdog.socket.socket") | ||
| def test_arm_failure_returns_minus_one(self, mock_socket): | ||
| sock, _ = make_sock_mock({"error": "bad"}) | ||
| mock_socket.return_value = sock | ||
| assert BMCWatchdog().arm(30) == -1 | ||
|
|
||
| @mock.patch("sonic_platform_base.bmc_watchdog.socket.socket") | ||
| def test_arm_daemon_unreachable_returns_minus_one(self, mock_socket): | ||
| sock = mock.MagicMock() | ||
| sock.connect.side_effect = OSError("no daemon") | ||
| mock_socket.return_value = sock | ||
| assert BMCWatchdog().arm(30) == -1 | ||
|
|
||
| @mock.patch("sonic_platform_base.bmc_watchdog.socket.socket") | ||
| def test_disarm(self, mock_socket): | ||
| sock, _ = make_sock_mock({"result": True}) | ||
| mock_socket.return_value = sock | ||
| assert BMCWatchdog().disarm() is True | ||
|
|
||
| @mock.patch("sonic_platform_base.bmc_watchdog.socket.socket") | ||
| def test_disarm_daemon_unreachable_returns_false(self, mock_socket): | ||
| sock = mock.MagicMock() | ||
| sock.connect.side_effect = OSError("no daemon") | ||
| mock_socket.return_value = sock | ||
| assert BMCWatchdog().disarm() is False | ||
|
|
||
| @mock.patch("sonic_platform_base.bmc_watchdog.socket.socket") | ||
| def test_is_armed(self, mock_socket): | ||
| sock, _ = make_sock_mock({"result": True}) | ||
| mock_socket.return_value = sock | ||
| assert BMCWatchdog().is_armed() is True | ||
|
|
||
| @mock.patch("sonic_platform_base.bmc_watchdog.socket.socket") | ||
| def test_get_remaining_time(self, mock_socket): | ||
| sock, _ = make_sock_mock({"result": 42}) | ||
| mock_socket.return_value = sock | ||
| assert BMCWatchdog().get_remaining_time() == 42 | ||
|
|
||
| @mock.patch("sonic_platform_base.bmc_watchdog.socket.socket") | ||
| def test_get_remaining_time_daemon_unreachable(self, mock_socket): | ||
| sock = mock.MagicMock() | ||
| sock.connect.side_effect = OSError("no daemon") | ||
| mock_socket.return_value = sock | ||
| assert BMCWatchdog().get_remaining_time() == -1 | ||
|
|
||
| @mock.patch("sonic_platform_base.bmc_watchdog.socket.socket") | ||
| def test_is_armed_falls_back_to_sysfs_active(self, mock_socket): | ||
| sock = mock.MagicMock() | ||
| sock.connect.side_effect = OSError("no daemon") | ||
| mock_socket.return_value = sock | ||
| wd = BMCWatchdog() | ||
| with mock.patch("builtins.open", mock.mock_open(read_data="active\n")): | ||
| assert wd.is_armed() is True | ||
|
|
||
| @mock.patch("sonic_platform_base.bmc_watchdog.socket.socket") | ||
| def test_is_armed_falls_back_to_sysfs_inactive(self, mock_socket): | ||
| sock = mock.MagicMock() | ||
| sock.connect.side_effect = OSError("no daemon") | ||
| mock_socket.return_value = sock | ||
| wd = BMCWatchdog() | ||
| with mock.patch("builtins.open", mock.mock_open(read_data="inactive\n")): | ||
| assert wd.is_armed() is False | ||
|
|
||
| @mock.patch("sonic_platform_base.bmc_watchdog.socket.socket") | ||
| def test_is_armed_sysfs_missing_returns_false(self, mock_socket): | ||
| sock = mock.MagicMock() | ||
| sock.connect.side_effect = OSError("no daemon") | ||
| mock_socket.return_value = sock | ||
| wd = BMCWatchdog() | ||
| with mock.patch("builtins.open", side_effect=OSError("missing")): | ||
| assert wd.is_armed() is False |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.