runtime/rants

Project

pdfbox

active

Serverless multi-tenant file storage on AWS, all in Terraform. Presigned uploads, event-driven confirmation and idempotent writes.

go · aws · terraform · s3 · sqs · dynamodb

What it is

Serverless multi-tenant PDF storage on AWS, written in Go and provisioned with Terraform, where file bytes never pass through the backend.

The API writes metadata to DynamoDB and hands the browser a short-lived S3 presigned URL; the browser then uploads or downloads directly against S3. Lambda handles kilobytes of JSON whether the user uploads 12 KB or 50 MB.

Architecture

Lambda (Go) behind an API Gateway HTTP API, DynamoDB for metadata, S3 for objects, S3 notifications into SQS with a DLQ, IAM roles and JWT (HS256) for identity, all provisioned with Terraform.

The upload flow:

  1. The client calls POST /files with filename, size and MIME type.
  2. The API writes a pending record keyed by the JWT’s user_id and returns a presigned PUT URL, valid for 15 minutes, with Content-Type and the exact Content-Length in the signature.
  3. The browser uploads straight to S3.
  4. S3 emits s3:ObjectCreated:* to an SQS queue.
  5. A worker Lambda checks the object’s real size against the record and the 50 MB ceiling, then marks the record uploaded, or rejected and deletes the object.

What I learned

The README’s design decisions, in short:

  • Presigned URLs instead of proxying. Proxying through Lambda would cap uploads at Lambda’s 6 MB request payload limit, well under the 50 MB the product allows.
  • The confirmation comes from S3, not the client. The client can close the tab, lose connectivity or confirm without uploading anything; only S3 knows whether the object exists.
  • Idempotency, because SQS is at-least-once. The worker treats a duplicate event as success, not as an error.
  • Conditional writes instead of read-then-write. Every DynamoDB write carries a ConditionExpression, so the check is atomic with the write.
  • Authorization is structural. The JWT’s user claim builds the partition key, so another user’s data is never in hand to be checked.
  • No dual-write problem. The backend writes only to DynamoDB; S3 publishes the event atomically with the object write.

What’s next

From the README’s “Not yet implemented” list:

  • The reconciliation Lambda and its EventBridge Scheduler rule (cmd/event-bridge exists but is empty).
  • The DynamoDB TTL that purges trashed records after a retention window.
  • A restore-from-trash route.