Make idleness explicit in ObjectDiffusion (V2) mini-protocol

[amesgen]

Motivated by IntersectMBO/ouroboros-consensus#1669 (comment)

FTR, an alternative would be to make idling explicit in the protocol, ie add a MsgAwaitReply (without any payload) just like in ChainSync as a reply by the outbound side in response to a blocking request, and to only idle in that case.

More concretely, we can change the protocol state machine from

graph LR
    start((start)) --> StInit[StInit]
    StInit -->|MsgInit| StIdle[StIdle]

    StIdle -->|MsgRequestObjs| StObjs[StObjs]
    StObjs -->|MsgReplyObjs| StIdle

    StIdle -->|MsgDone| StDone[StDone]

    StIdle -->|MsgRequestObjIdsNonBlocking| StObjIdsNonBlocking[StObjIdsNonBlocking]
    StObjIdsNonBlocking -->|MsgReplyObjIds| StIdle

    StIdle -->|MsgRequestObjIdsBlocking| StObjIdsBlocking[StObjIdsBlocking]
    StObjIdsBlocking -->|MsgReplyObjIds| StIdle

    classDef inboundState fill:#c7f9cc,stroke:#2d6a4f,stroke-width:2px,color:#000;
    classDef doneState fill:#ffffff,stroke:#000,stroke-width:2px,color:#000;
    classDef outboundState fill:#e7f0ff,stroke:#4361ee,stroke-width:2px,color:#000;

    class StInit,StIdle inboundState;
    class StDone doneState;
    class StObjs,StObjIdsNonBlocking,StObjIdsBlocking outboundState;

Loading

to

graph LR
    start((start)) --> StInit[StInit]
    StInit -->|MsgInit| StIdle[StIdle]

    StIdle -->|MsgRequestObjs| StObjs[StObjs]
    StObjs -->|MsgReplyObjs| StIdle

    StIdle -->|MsgDone| StDone[StDone]

    StIdle -->|MsgRequestObjIdsNonBlocking| StObjIdsNonBlocking[StObjIdsNonBlocking]
    StObjIdsNonBlocking -->|MsgReplyObjIds| StIdle    

    StIdle -->|MsgRequestObjIdsBlocking| StObjIdsCanAwait[StObjIdsCanAwait]
    StObjIdsCanAwait -->|MsgReplyObjIds| StIdle
    StObjIdsCanAwait -->|MsgAwaitReply| StObjIdsMustReply[StObjIdsMustReply]
    StObjIdsMustReply -->|MsgReplyObjIds| StIdle

    classDef inboundState fill:#c7f9cc,stroke:#2d6a4f,stroke-width:2px,color:#000;
    classDef doneState fill:#ffffff,stroke:#000,stroke-width:2px,color:#000;
    classDef outboundState fill:#e7f0ff,stroke:#4361ee,stroke-width:2px,color:#000;

    class StInit,StIdle inboundState;
    class StDone doneState;
    class StObjs,StObjIdsNonBlocking,StObjIdsCanAwait,StObjIdsMustReply outboundState;

Loading

Here, StObjIdsCanAwait has a short timeout (as the outbound side can always quickly reply, either with MsgReplyObjIds if it has more IDs, and with MsgAwaitReply otherwise), while StObjIdsMustReply has a long timeout.

Depends on #140

[agustinmista]

As #140 is close to being merged, we should implement this for ObjectDiffusion V2 instead.