Cloud Media Hub
Media upload and storage service that streams files straight to S3 and catalogs them in MongoDB.
Distributed storage backend, Sep 2026
A distributed file-storage service: Express streams multipart uploads directly to S3 without buffering in memory, MongoDB stores the metadata, and a proxy route serves objects from a private bucket.
Problem
Media uploads break naïve servers. Buffering files in memory exhausts the Node.js heap under concurrent uploads, storing binaries in the database bloats it, and a private bucket blocks the browser from showing previews.
Why it was worth building
Storing binaries in S3 and metadata in a database, then streaming between them, underpins almost every product that handles user files. This project builds that from first principles, with a written design document.
Architecture
- Browsermultipart/form-data
- ExpressPOST /upload
- Multer filterimage/* only, 25 MB cap
- multer-s3 streamUUID key, piped to S3
- S3private bucket
- MongoDBinsertMany after S3 ack
- BrowserGET /media/:id
- Express proxylooks up key in MongoDB
- S3 GetObjectserver IAM credentials
- Stream to clientcorrect Content-Type
Technical decisions
S3 for binaries, MongoDB for metadata
Object storage is cheap, durable and CDN-friendly. The database holds only what you query (name, size, MIME type, key, timestamps). The design doc compares this against GridFS and local disk.
Commit metadata only after S3 acknowledges
MongoDB records are written with insertMany once S3 confirms the upload, so a dropped connection mid-upload cannot leave orphaned database rows.
A streaming proxy for private buckets
With Block Public Access on, direct S3 URLs fail in the browser. GET /media/:id streams the object through the server’s credentials, and the UI falls back to it automatically.
Swappable storage engine
STORAGE_TYPE=s3 or local lets the app run offline in development and CI without AWS credentials or cloud cost.
Challenges
Memory under concurrent uploads
Replacing memory storage with multer-s3 streaming keeps memory use independent of file size, because chunks flow from the socket to S3 with backpressure.
Untrusted file input
Files are renamed to random UUIDv4 keys (no path traversal or collisions), non-image MIME types are rejected before upload starts, and size is capped at 25 MB.
What I built
- REST API: POST /upload, GET /images, GET /media/:id, DELETE /images (batch) and /images/:id, GET /api/config.
- Batch deletes use S3 DeleteObjects plus a single MongoDB delete.
- Vanilla JS client with drag-and-drop staging, multi-select delete and a lightbox preview.
- ARCHITECTURE.md with high-level and low-level design, sequence diagrams and API contracts.
Results
- Upload memory use does not grow with file size.
- Runs identically against S3 or local disk behind one environment flag.
Takeaways
- Where you commit the metadata write decides whether failures leave garbage behind.
- Writing the design doc first made the trade-offs (GridFS vs S3 vs disk) explicit instead of accidental.
What I'd do next
- Presigned URLs so browsers upload directly to S3 and the server never touches the bytes.
- CloudFront in front of the bucket, and a Lambda trigger for thumbnails.
- Authentication on upload and delete.