Recover from interface recreation - #70
Merged
Merged
Conversation
The name is an interface's stable identity; the index is not, since a recreated interface keeps the name and gets a new number. Reidentify re-resolves it and reports whether the interface was repointed, parked, unchanged, or unresolvable, so a caller can tell recreation from absence. if_nametoindex is not a pure lookup -- glibc opens a socket inside it -- so under fd pressure it returns 0 for a live interface. Reading that as absence would clear the addresses and shut the egress gate on something healthy, so the resource errnos report Unresolved and leave the identity untouched. An interface that is absent and stays absent reports Unchanged, which keeps a retry loop from logging or working on every pass. Joining a group now refuses index 0 rather than letting the kernel read it as "any interface".
A capture holds an fd, so nothing in userland notices its interface leaving; only the kernel knows. Attached asks it -- the bound ifindex via getsockname on Linux, BIOCGDLT on the BSDs, which fails outright once a descriptor detaches. The probe is not a corner case. Linux gives a recreated interface a fresh index, but macOS hands back the one it had, so there the index is unchanged and the probe is the only thing that reports the capture is detached. Rebind re-runs the attach path on the same fd, so registrations keyed by it stay valid. Linux consumes the ENETDOWN the kernel parked when the old interface died, and the BSDs reset the batch cursor the re-attach invalidated, re-reading the framing in case the interface came back as a different link type.
Receive returned nullopt for three outcomes: an empty queue, a frame we do not forward, and a read that failed. The drain has to tell the third from the others, since a failed read is often the first sign of an interface destroyed under the capture, so Receive now returns the reason. The reason is an enum rather than the codebase's Error: the drain switches on it instead of printing it, and building an Error for the would-block that ends every drain would allocate a string per readable event. The socket still logs the errno where it knows it.
A destroyed and recreated interface kept its old index, and its capture stayed bound to a dead kernel object, so the link went quiet until a restart. The reconcile re-resolves every interface, not just the one an event named: a returning interface reports a new index, which matches nothing we hold. It probes each capture first -- one getsockname against three syscalls for a name lookup -- and resolves when the probe fails or the drain asked for that interface. The second case is what catches a rename, which keeps both the index and the capture. Two timers back it up. The backstop is the only channel that does not wait on the kernel to say something; at 30s it rides a wakeup the poll already makes, costing one probe per capture. The retry runs each second and only while a repair is outstanding, since a rebind that failed on a transient error has nothing coming to finish it. The address monitor now reports a whole drain at once instead of once per index, into a fixed-size list that degrades to refresh-all when a drain names more interfaces than it holds.
Both resolvers bailed out on a resource failure and returned an empty address set, which Refresh assigned -- so a transient EMFILE blanked a live interface and shut its egress gate. They now report whether the enumeration ran at all, and Refresh keeps what it had when it did not. An interface that genuinely has no addresses still resolves to an empty set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An interface that is destroyed and recreated -- a VLAN re-added, a bridge member
re-attached, a USB NIC re-plugged -- kept its old kernel index, and its capture
socket stayed bound to the dead kernel object. The daemon went quiet on that link
until a restart.
What recovers it
Interface::Reidentify()re-resolves the name to a kernel index and reports whetherthe interface was repointed, parked, unchanged, or unresolvable. Interface is the
only thing that caches the index -- every consumer reads
Index()at use -- sore-pointing it fixes all of them.
LinkSocket::Attached()/Rebind()cover the kernel-side binding, which does notfollow the interface across a recreation. Rebind reuses the same fd, so dispatcher
registrations keyed by it stay valid.
Application::ReconcileInterfaces()ties them together, on three triggers:indexes instead of once per index
The pass re-resolves every interface, not only the ones an event named: a returning
interface reports a new index, which matches nothing we currently hold. It probes
each capture before resolving, since a probe is one syscall where a name lookup is
three, and resolves anyway for an interface the drain named -- which is what catches
a rename, as that keeps both the index and the capture.
Measured platform difference
Linux gives a recreated interface a fresh index; macOS hands back the one it had. So
comparing indexes alone misses the swap on macOS, and the attachment probe is a
primary signal rather than a backstop. The root-gated tests destroy and recreate a
real veth/feth and assert recovery under both behaviours.
Two "could not run" defects fixed on the way
if_nametoindexopens a socket internally, and the address enumerators open netlink/ AF_INET6 sockets. Under fd or memory pressure each returned "nothing", which read
as "this interface is gone" and cleared its addresses -- shutting the egress gate on
a perfectly healthy link. Both now separate a lookup that could not run from one that
ran and found nothing, and keep what they had in the first case.
Known gap: DIAL proxy state
An interface recreated with the same IPv4 address leaves existing DIAL listeners and
connections pinned to the dead interface:
SO_BINDTODEVICEresolves the name to anifindex at setsockopt time and
IP_BOUND_IFtakes the index outright, whileDialProxy's staleness check compares only addresses and so never fires.
This PR makes that gap more visible rather than introducing it -- previously a
recreated interface killed the capture and nothing flowed at all, whereas now
everything else recovers and DIAL is the one thing degraded until a restart.
Evicting state that depends on a replaced interface (DIAL proxies, SSDP sessions,
port reservations) is the next change.
Testing
909 unit tests native (Debug, ASan/UBSan) and 896 in docker, including 19 root-gated
tests that exercise real interface destroy/recreate. Every commit builds standalone.