walrus-s3-gateway

Problem Statement

Walrus S3 Gateway is a high-performance, S3-compatible API gateway that provides seamless integration with the Walrus decentralized storage network. Built in Rust, it enables developers to use familiar S3 APIs while leveraging the robust, distributed storage capabilities of the Walrus blockchain-based storage system.Key Features:S3 API Compatibility: Full support for standard S3 operations (GET, PUT, DELETE, LIST)Client-Side Signing: Secure authentication using SUI blockchain-based cryptographic signaturesDecentralized Storage: Interfaces with Walrus network for distributed, fault-tolerant storageHigh Performance: Async request handling with configurable timeouts and concurrent operationsSecurity-First: Fine-grained permission management and signature verificationProduction Ready: TLS support, CORS handling, and comprehensive error managementArchitecture: The gateway acts as a bridge between traditional S3 client applications and the Walrus decentralized storage network. It implements a unique client-side signing workflow where clients maintain control of their private keys while the gateway validates and processes storage operations on the blockchain.Use Cases: Web3 Applications: Decentralized apps requiring censorship-resistant storage Data Archival: Long-term storage with blockchain guarantees Multi-Cloud Strategy: Hybrid storage solutions combining traditional and decentralized storage Developer Tools: Easy migration from S3 to decentralized storage without code changes The project is part of the broader Walrus ecosystem, providing enterprise-grade decentralized storage solutions with the convenience of familiar S3 APIs.

Solution

Walrus S3 Gateway is built using a modern, high-performance technology stack centered around Rust for maximum performance and memory safety:Rust - Core language providing zero-cost abstractions and memory safetyTokio - Async runtime enabling concurrent request handling without blockingHyper - High-performance HTTP server implementationSerde - Type-safe serialization/deserialization for configuration and dataSUI SDK - Blockchain integration for cryptographic signature verificationThe most hacky and notable aspect is our unique client-side signing workflow that bridges traditional S3 APIs with blockchain security.How it works:Client sends standard S3 PUT requestGateway responds with HTTP 202 + unsigned transaction template (breaking S3 convention)Client signs transaction locally with their SUI walletClient submits signed transaction via dedicated endpointGateway validates and executes on Walrus networkThis approach is particularly hacky because:We hijack the standard S3 PUT flow to return 202 instead of 200We embed blockchain transaction templates in HTTP responsesWe maintain S3 compatibility while adding blockchain security layer