CRDT Sync

https://github.com/muni-town/roomy-services/blob/main/roomy-syncserver/main.ts

The code shows a thoughtful approach to building a CRDT sync server with some particularly interesting architectural decisions:

Most Notable Design Choices:

1. AT Protocol Integration - Using decentralized identity (DIDs) instead of traditional user accounts is forward-thinking and aligns with web3 principles
2. Two-Stage Authentication - The JWT → short-lived token → WebSocket flow is clever for security while maintaining performance
3. Binary WebSocket Protocol - Smart choice for CRDT operations that need high throughput and low latency

The Good: The separation of concerns is clean, the security model is well-thought-out, and the use of Deno KV provides a simple but effective storage solution.

The Concerning: The missing DID caching (which they acknowledge) could become a major bottleneck, and the error handling is quite generic which could make debugging difficult in production.

Overall, this feels like a solid MVP architecture that could scale with some additional production hardening around caching, monitoring, and connection management.

CRDT Sync Server Architecture Review

Summary

This is a Deno-based CRDT synchronization server that provides real-time collaborative data sync capabilities. It integrates with the AT Protocol (ATProto) ecosystem for identity management and uses WebSockets for real-time communication. The server acts as a relay/coordination point for CRDT-based collaborative applications.

Key Components

Core Technologies

• Runtime: Deno with native KV storage
• Routing: itty-router with CORS support
• CRDT Engine: @muni-town/leaf (SuperPeer1)
• Identity: AT Protocol DID resolution and JWT verification
• Transport: WebSocket with binary protocol
• Storage: Deno KV with blob storage adapter

Authentication Flow

1. JWT Validation: Uses ATProto JWT tokens verified against DID documents
2. Token Exchange: Converts JWT to short-lived WebSocket tokens (30s TTL)
3. WebSocket Auth: Uses Sec-WebSocket-Protocol header for token passing

Architectural Decisions Analysis

✅ Strengths

1. Decentralized Identity Integration

• Leveraging AT Protocol DIDs provides robust, decentralized identity
• No need to manage user accounts or passwords
• Interoperable with the broader ATProto ecosystem

2. Security Design

• JWT verification with proper DID resolution
• Short-lived tokens (30s) minimize exposure window
• Clear separation between HTTP auth and WebSocket auth

3. Binary Protocol Efficiency

• WebSocket binary transport reduces overhead
• Direct ArrayBuffer handling for performance
• Appropriate for high-frequency CRDT operations

4. Clean Separation of Concerns

• Authentication middleware cleanly separates auth logic
• CRDT operations abstracted through SuperPeer1BinaryWrapper
• Storage abstraction allows different backends

5. Development Ergonomics

• Simple environment-based configuration
• Dev token bypass for testing
• Clear error responses

⚠️ Areas of Concern

1. Token Storage Architecture

const tokenDid = (await kv.get<string>(["tokens", token])).value;

• Storing tokens in same KV store as application data
• No token cleanup mechanism beyond TTL
• Potential for token enumeration if KV keys are predictable

2. Error Handling

} catch (e) {
  console.error("Error validating JWT:", e);
  return error(403, "Could not validate authorization JWT.");
}

• Generic error responses may hide useful debugging info
• No distinction between malformed vs expired tokens
• Console logging in production code

3. DID Resolution Performance

// TODO: add a DID cache using Deno KV
const idResolver = new IdResolver();

• No caching of DID documents (acknowledged TODO)
• Repeated network calls for same DIDs
• Potential latency and rate limiting issues

4. Resource Management

socket.addEventListener("close", () => {
  backend.cleanup();
});

• Limited connection lifecycle management
• No explicit connection limits or rate limiting
• Potential memory leaks if cleanup fails

5. Configuration Validation

• Missing validation for required environment variables
• No graceful degradation options
• Hard failure on missing DID config

🔄 Suggested Improvements

1. Implement DID Caching

// Cache DID documents with TTL
const didCache = new Map();
async function getCachedSigningKey(did: string) {
  // Implementation with cache-aside pattern
}

2. Enhanced Token Security

// Use cryptographically secure token generation
// Implement token revocation
// Separate token storage namespace

3. Connection Management

// Add connection limits per DID
// Implement heartbeat/keepalive
// Graceful shutdown handling

4. Observability

// Structured logging
// Metrics collection
// Health check endpoints

5. Error Granularity

// Distinguish between different error types
// Provide actionable error messages
// Rate limit error responses

Production Readiness Assessment

Ready ✅

• Core CRDT functionality
• Basic security model
• WebSocket transport
• AT Protocol integration

Needs Work ⚠️

• Performance optimization (DID caching)
• Production error handling
• Connection lifecycle management
• Monitoring and observability

Missing ❌

• Rate limiting
• Connection pooling
• Horizontal scaling considerations
• Comprehensive logging

Overall Assessment

This is a well-architected foundation for a CRDT sync server with thoughtful integration of modern decentralized identity systems. The core design decisions are sound, particularly the choice to use AT Protocol for identity and binary WebSocket transport for efficiency.

The main areas for improvement are around production hardening: caching, error handling, resource management, and observability. The TODO comment about DID caching shows awareness of the key performance bottleneck.

For a prototype or early-stage system, this architecture provides a solid foundation. For production deployment, addressing the identified concerns around performance, error handling, and resource management would be essential.