Wire Protocol Migration: RPC to StyreneProtocol

Summary: Consolidating JSON RPC and binary StyreneProtocol into unified wire format v2. LXMF integration via FIELD_CUSTOM_TYPE, 16-byte request correlation, 256 message types.

Date: 2026-01-31 Status: Approved for Implementation Context: Consolidating duplicate wire protocol implementations in styrened

Executive Summary

styrened currently has two overlapping wire protocol systems:

RPC System (rpc/messages.py, rpc/client.py, rpc/server.py) - JSON-based, active
StyreneProtocol (protocols/styrene.py, models/styrene_wire.py) - Binary msgpack, newer

This document plans the migration from RPC to StyreneProtocol to achieve:

Backward compatibility with non-Styrene LXMF clients (NomadNet, Sideband, MeshChat)
Proper LXMF protocol encapsulation via FIELD_CUSTOM_TYPE/FIELD_CUSTOM_DATA
Single wire format for all Styrene messages

Philosophical Alignment

LXMF Already Identifies Senders

Per the LXMF specification, every message includes:

Source hash (16 bytes) - mandatory
Ed25519 signature - cryptographic proof of sender

LXMF messages are NOT anonymous by design. The request_id correlation token we add:

Is NOT an identity (random bytes, no structure)
Is ephemeral (changes per request)
Is encrypted in transit (inside LXMF payload)
Reveals nothing about the sender beyond what LXMF already requires

This aligns with Reticulum’s philosophy: initiator anonymity is possible at the RNS layer, but LXMF is explicitly a signed messaging protocol.

Current State

RPC System (to be deprecated)

Location: styrened/rpc/

Wire Format:

{
  "type": "status_request",
  "request_id": "uuid-v4",
  "protocol": "rpc"
}

Issues:

JSON blob visible to non-Styrene clients (confusing)
No LXMF FIELD_CUSTOM_TYPE usage
No versioning
Duplicates functionality in StyreneProtocol

StyreneProtocol (target architecture)

Location: styrened/protocols/styrene.py, styrened/models/styrene_wire.py

Wire Format:

LXMF Fields:
  FIELD_CUSTOM_TYPE = b"styrene.io"
  FIELD_CUSTOM_DATA = [PREFIX][VERSION:1][TYPE:1][PAYLOAD:msgpack]
  content = "[styrene.io:STATUS_REQUEST]"  (human-readable fallback)

Where PREFIX = b"styrene.io:" (11 bytes)

Advantages:

Non-Styrene clients see readable content: [styrene.io:STATUS_REQUEST]
Proper LXMF protocol encapsulation
Versioned for future compatibility
Binary msgpack payload (compact, matches RNS/LXMF internals)

Migration Plan

Phase 1: Extend StyreneMessageType

Replace message type enum in models/styrene_wire.py with expanded allocation:

class StyreneMessageType(IntEnum):
    """Message types for Styrene protocol.

    Allocation:
    - 0x00:       Reserved (invalid)
    - 0x01-0x0F:  Control/keepalive (15 types)
    - 0x10-0x1F:  Status/query (16 types)
    - 0x20-0x2F:  Content/chat (16 types)
    - 0x30-0x3F:  Network/discovery (16 types)
    - 0x40-0x5F:  RPC commands (32 types)
    - 0x60-0x7F:  RPC responses (32 types)
    - 0x80-0x9F:  Hub services (32 types)
    - 0xA0-0xBF:  Pub/Sub (32 types)
    - 0xC0-0xDF:  Reserved/future (32 types)
    - 0xE0-0xFE:  Application-specific (31 types)
    - 0xFF:       Error
    """

    # Control (0x01-0x0F)
    PING = 0x01
    PONG = 0x02
    HEARTBEAT = 0x03

    # Status (0x10-0x1F)
    STATUS_REQUEST = 0x10
    STATUS_RESPONSE = 0x11
    CAPABILITIES_REQUEST = 0x12
    CAPABILITIES_RESPONSE = 0x13

    # Content (0x20-0x2F)
    CHAT = 0x20
    CHAT_ACK = 0x21
    FILE_OFFER = 0x22
    FILE_ACCEPT = 0x23
    FILE_CHUNK = 0x24

    # Network (0x30-0x3F)
    ANNOUNCE = 0x30
    ANNOUNCE_ACK = 0x31
    PEER_REQUEST = 0x32
    PEER_RESPONSE = 0x33

    # RPC Commands (0x40-0x5F)
    EXEC = 0x40
    REBOOT = 0x41
    CONFIG_UPDATE = 0x42
    SERVICE_CONTROL = 0x43

    # RPC Responses (0x60-0x7F)
    EXEC_RESULT = 0x60
    REBOOT_RESULT = 0x61
    CONFIG_RESULT = 0x62
    SERVICE_RESULT = 0x63

    # Hub Services (0x80-0x9F)
    REGISTRY_QUERY = 0x80
    REGISTRY_RESPONSE = 0x81
    FLEET_STATUS_REQUEST = 0x82
    FLEET_STATUS_RESPONSE = 0x83

    # Pub/Sub (0xA0-0xBF)
    SUBSCRIBE = 0xA0
    UNSUBSCRIBE = 0xA1
    PUBLISH = 0xA2
    SUBSCRIPTION_ACK = 0xA3

    # Error
    ERROR = 0xFF

This allocation provides:

160 reserved types (0xC0-0xDF) for future expansion
31 application-specific types (0xE0-0xFE) for custom extensions
Clear semantic groupings with room to grow

Phase 2: Add Request Correlation

The RPC system uses request_id for matching responses to requests.

Decision: Header-level correlation with random bytes

Wire format v2:
[PREFIX][VERSION:1][TYPE:1][REQUEST_ID:16][PAYLOAD:N]

Where REQUEST_ID = 16 random bytes (os.urandom(16))

Why random bytes instead of UUID:

No structure to fingerprint (UUID has version/variant bits)
More bandwidth-efficient (16 bytes vs 36 character string)
Equally unique for correlation purposes
Aligns with Reticulum’s use of random hashes

For fire-and-forget messages (ANNOUNCE, CHAT without ACK):

Use 16 zero bytes to indicate “no correlation expected”
Recipient ignores request_id field

This requires incrementing STYRENE_VERSION to 2 and updating StyreneEnvelope.

Phase 3: Create Payload Schemas

Define msgpack payload schemas for each message type:

# EXEC (0x12)
{
    "command": str,      # Command to execute
    "args": list[str],   # Command arguments
}

# EXEC_RESULT (0x13)
{
    "exit_code": int,
    "stdout": str,
    "stderr": str,
}

# REBOOT (0x14)
{
    "delay": int,        # Seconds to delay (0 = immediate)
}

# REBOOT_RESULT (0x15)
{
    "success": bool,
    "message": str,
    "scheduled_time": float | None,
}

# CONFIG_UPDATE (0x16)
{
    "updates": dict[str, Any],
}

# CONFIG_RESULT (0x17)
{
    "success": bool,
    "message": str,
    "updated_keys": list[str],
}

# ERROR (0xFF)
{
    "code": int,
    "message": str,
}

Phase 4: Refactor RPCClient

Update rpc/client.py to use StyreneProtocol.send_typed_message():

class RPCClient:
    def __init__(self, styrene_protocol: StyreneProtocol):
        self._protocol = styrene_protocol
        self.pending_requests: dict[str, PendingRequest] = {}

    async def call_status(self, destination: str, timeout: float = 10.0) -> StatusResponse:
        request_id = uuid4().bytes

        await self._protocol.send_typed_message(
            destination=destination,
            message_type=StyreneMessageType.STATUS_REQUEST,
            request_id=request_id,
            payload=b"",
        )

        return await self._wait_for_response(request_id, timeout)

Phase 5: Refactor RPCServer

Update rpc/server.py to handle StyreneProtocol messages:

class RPCServer:
    def __init__(self, styrene_protocol: StyreneProtocol):
        self._protocol = styrene_protocol
        # Register handler for Styrene RPC message types
        styrene_protocol.register_handler(
            StyreneMessageType.STATUS_REQUEST,
            self._handle_status_request
        )
        styrene_protocol.register_handler(
            StyreneMessageType.EXEC,
            self._handle_exec
        )
        # ... etc

Phase 6: Deprecate Old RPC Messages

Mark rpc/messages.py dataclasses as deprecated
Update imports in TUI to use new path
Remove after one release cycle

Wire Format v2 Specification

┌─────────────────────────────────────────────────────────────┐
│                    Styrene Wire Format v2                    │
├─────────────────────────────────────────────────────────────┤
│ FIELD_CUSTOM_TYPE: b"styrene.io"                            │
├─────────────────────────────────────────────────────────────┤
│ FIELD_CUSTOM_DATA:                                          │
│   ┌───────────┬─────────┬──────┬────────────┬─────────────┐ │
│   │  PREFIX   │ VERSION │ TYPE │ REQUEST_ID │   PAYLOAD   │ │
│   │ 11 bytes  │ 1 byte  │1 byte│  16 bytes  │  N bytes    │ │
│   │styrene.io:│   0x02  │ enum │  random    │  msgpack    │ │
│   └───────────┴─────────┴──────┴────────────┴─────────────┘ │
├─────────────────────────────────────────────────────────────┤
│ content: "[styrene.io:TYPE_NAME]" (human-readable fallback) │
└─────────────────────────────────────────────────────────────┘

Total header overhead: 11 + 1 + 1 + 16 = 29 bytes

REQUEST_ID Semantics

Value	Meaning
16 random bytes	Expects correlated response
16 zero bytes	Fire-and-forget, no response expected

Backward Compatibility

Version 1 messages (no REQUEST_ID) remain parseable
Decoder checks version byte and adjusts parsing accordingly
v1 messages cannot participate in request/response correlation
v1 clients receive v1 responses (no REQUEST_ID in header)

Implementation Order

styrened changes:
- Add new message types to StyreneMessageType
- Update StyreneEnvelope for v2 wire format with request_id
- Add payload encode/decode helpers for each message type
- Refactor RPCClient to use StyreneProtocol
- Refactor RPCServer to handle Styrene messages
- Add request_id parameter to StyreneProtocol.send_typed_message()
- Add handler registration system to StyreneProtocol
- Bridge RPCServer handlers via StyreneProtocol.register_handler()
- Update tests
TUI changes:
- Update README examples with new StyreneProtocol API
- Update actual code imports
- Test RPC functionality end-to-end
Cleanup:
- Mark old rpc/messages.py deprecated (with deprecation warnings)
- Remove deprecated request classes (StatusRequest, ExecCommand, etc.)
- Remove deprecated deserialize_message() function
- Remove deprecated handle_message() from RPCServer
- Clean up rpc/init.py exports

Testing Strategy

Unit Tests

Wire format encode/decode round-trip
All message types serialize correctly
Version compatibility (v1 vs v2)
Request correlation matching

Integration Tests

RPC client/server over LXMF
Timeout handling
Error propagation

Compatibility Tests

Non-Styrene clients see readable content field
v1 messages still parse correctly

Design Decisions (Resolved)

REQUEST_ID format: 16 random bytes (os.urandom(16)), not UUID string
- No fingerprinting risk from UUID structure
- Bandwidth efficient
- Zero bytes = fire-and-forget
Error handling: ERROR (0xFF) includes original request_id for correlation
Message type allocation: Expanded to 256 values with semantic ranges
- Hub services: 0x80-0x9F (32 types)
- Pub/Sub: 0xA0-0xBF (32 types)
- Reserved: 0xC0-0xDF (32 types)
- App-specific: 0xE0-0xFE (31 types)
Philosophical alignment: Confirmed kosher with Reticulum principles
- LXMF already requires sender identification (source hash + signature)
- REQUEST_ID is correlation token, not identity
- Encrypted in transit, ephemeral per request

References

LXMF Protocol - Custom fields documentation
Reticulum Manual - Identity philosophy
styrened wire format - Current implementation
RPC messages - To be deprecated