Technical Design TDD-0002 — File Storage System¶
| Field | Value |
|---|---|
| ID | TDD-0002 |
| Title | File Storage System |
| Status | Draft |
| Owner | Team |
| Audience | contributors · integrators |
| Created | 2025-07-14 |
| Last Updated | 2025-07-14 |
| Reviewers | — |
| Approvers | — |
| Tracking | — |
| Related | — |
1. Summary¶
This document describes the design for a file storage system within the loryth-cms service. The system allows users to upload, download, and delete files associated with specific owners (users or groups). Files are organized into storage buckets that are owned by specific OwnerIDs. The system manages file references, tracks usage statistics, and implements different file management strategies based on whether files are managed or unmanaged.
2. Context and Problem Statement¶
The loryth-cms service requires a robust file storage system to allow users to store and retrieve various assets. Currently, there is no standardized approach for file management within the system, leading to inconsistent implementations across different features. This design aims to provide a unified approach to file storage that addresses common requirements while allowing for future extensibility.
2.1 Goals¶
Functional Requirements¶
- Users must be able to upload files to buckets associated with an owner ID (user or group)
- Users must be able to download files by file hash from appropriate buckets
- Users must be able to perform partial uploads and downloads of large files with resume capability
- Users must be able to create, list, and manage storage buckets they have access to
- Users must be able to delete files they have access to
- The system must maintain file references and track which owners have access to which files
- The system must distinguish between managed and unmanaged files with different deletion behaviors
- The system must track statistics about total file size and count for each owner and bucket
Non-functional Requirements¶
- Performance: File uploads and downloads should be efficient, even for large files
- Security: Files should only be accessible to authorized users
- Scalability: The system should handle a growing number of files and users
- Reliability: File operations should be transactional to prevent data corruption
- Resilience: The system should support resumable uploads and downloads to handle network interruptions
2.2 Out of Scope¶
Not defined.
2.3 Assumptions and Constraints¶
- Database system for metadata storage (using existing project database)
- File storage backend (local filesystem initially, with cloud storage options for production)
- Authentication and authorization services for access control
- Background job processing for managed file cleanup
2.4 Open Questions¶
- Should we implement file versioning in the initial release or leave it for a future enhancement?
- What is the appropriate grace period duration for managed files?
- Should we implement file type restrictions or virus scanning as part of the upload process?
- How should we handle large file uploads to prevent timeouts and improve user experience?
- Should buckets have default versus custom settings for file management (managed vs. unmanaged)?
- What should be the default bucket creation limits per owner?
- Should we implement bucket templates for common use cases (e.g., image bucket, document bucket)?
- How should we handle bucket permissions for collaborative scenarios?
- Should files be transferable between buckets, and if so, how should references be maintained?
3. Design Overview¶
The file storage system consists of several components:
- Storage Service: Core service responsible for storing and retrieving file data
- File Metadata Store: Database for tracking file metadata, references, and statistics
- Access Control Layer: Component that enforces access permissions
- Statistics Tracker: Component that maintains usage statistics for owners
graph TD
Client[Client Application] --> API[File Storage API]
API --> Upload[Upload Handler]
API --> MultipartUpload[Multipart Upload Handler]
API --> Download[Download Handler]
API --> PartialDownload[Partial Download Handler]
API --> Delete[Delete Handler]
API --> BucketMgmt[Bucket Management]
Upload --> ACL[Access Control Layer]
MultipartUpload --> ACL
Download --> ACL
PartialDownload --> ACL
Delete --> ACL
BucketMgmt --> ACL
ACL --> MetadataService[File Metadata Service]
MetadataService --> DB[(Metadata Database)]
ACL --> StorageService[Storage Service]
StorageService --> BucketManager[Bucket Manager]
BucketManager --> FileSystem[(File System Storage)]
StorageService --> ChunkManager[Chunk Manager]
ChunkManager --> FileSystem
MetadataService --> StatsTracker[Statistics Tracker]
StatsTracker --> DB3.1 Key Components¶
| Component | Purpose |
|---|---|
| Storage Service | Core service responsible for storing and retrieving file data |
| File Metadata Store | Database for tracking file metadata, references, and statistics |
| Access Control Layer | Component that enforces access permissions |
| Statistics Tracker | Component that maintains usage statistics for owners |
| Bucket Manager | Manages bucket creation, listing, and deletion |
| Chunk Manager | Handles multipart upload and partial download operations |
3.2 Architecture¶
Data Model¶
Bucket¶
interface Bucket {
id: string; // Unique identifier
name: string; // Bucket name (e.g., "profile-images", "documents")
ownerId: string; // Owner (user or group) ID that owns this bucket
description?: string; // Optional description of bucket contents
createdAt: Date; // When the bucket was created
createdBy: string; // User who created the bucket
settings?: { // Optional bucket-specific settings
maxFileSize?: number; // Maximum file size in bytes
allowedMimeTypes?: string[]; // List of allowed MIME types
};
}
File¶
interface File {
id: string; // Unique identifier
bucketId: string; // ID of the bucket containing this file
hash: string; // Content hash for deduplication and access
name: string; // Original filename
size: number; // File size in bytes
mimeType: string; // MIME type
createdAt: Date; // Creation timestamp
isManaged: boolean; // Whether the file is managed
deleteAt?: Date; // Scheduled deletion time for managed files
status: 'complete' | 'partial'; // Whether the file is complete or in progress
uploadId?: string; // ID for tracking multi-part uploads
chunks?: FileChunk[]; // Chunks for partially uploaded files
}
FileChunk¶
interface FileChunk {
id: string; // Unique identifier
fileId: string; // Reference to the parent file
partNumber: number; // Sequence number of the chunk
startByte: number; // Starting byte position in the complete file
endByte: number; // Ending byte position in the complete file
size: number; // Size of this chunk in bytes
etag: string; // ETag/hash of this chunk for verification
uploadedAt: Date; // When this chunk was uploaded
}
FileReference¶
interface FileReference {
id: string; // Unique identifier
fileId: string; // Reference to the file
bucketId: string; // ID of the bucket containing the file
ownerId: string; // Owner (user or group) ID
createdAt: Date; // When the reference was created
createdBy: string; // User who created the reference
}
OwnerStats¶
interface OwnerStats {
ownerId: string; // Owner (user or group) ID
bucketId?: string; // Optional bucket ID for bucket-specific stats
totalSize: number; // Total size of all files
totalCount: number; // Total number of files
updatedAt: Date; // Last update timestamp
}
3.3 Interface Impact (Before → After)¶
Not applicable — new system addition.
4. Implementation Details¶
4.1 Core Contracts and APIs¶
openapi: 3.0.3
info:
title: File Storage API
description: API for managing file storage buckets and files
version: 1.0.0
paths:
/api/buckets:
post:
summary: Create a new bucket
operationId: createBucket
tags:
- buckets
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- name
- ownerId
properties:
name:
type: string
description: Bucket name
example: "profile-images"
ownerId:
type: string
description: ID of the owner (user or group)
example: "user-123"
description:
type: string
description: Optional description of bucket contents
example: "User profile images"
settings:
type: object
properties:
maxFileSize:
type: integer
description: Maximum file size in bytes
example: 5242880
allowedMimeTypes:
type: array
description: List of allowed MIME types
items:
type: string
example: "image/jpeg"
responses:
'201':
description: Bucket created successfully
content:
application/json:
schema:
type: object
properties:
bucketId:
type: string
description: ID of the created bucket
name:
type: string
description: Bucket name
ownerId:
type: string
description: Owner ID
'400':
description: Invalid request
'401':
description: Unauthorized
'403':
description: Forbidden
get:
summary: List buckets
operationId: listBuckets
tags:
- buckets
parameters:
- name: ownerId
in: query
description: Filter buckets by owner ID
required: false
schema:
type: string
responses:
'200':
description: List of buckets
content:
application/json:
schema:
type: object
properties:
buckets:
type: array
items:
type: object
properties:
id:
type: string
description: Bucket ID
name:
type: string
description: Bucket name
ownerId:
type: string
description: Owner ID
description:
type: string
description: Bucket description
createdAt:
type: string
format: date-time
description: Creation timestamp
'401':
description: Unauthorized
/api/buckets/{bucketId}:
delete:
summary: Delete a bucket
operationId: deleteBucket
tags:
- buckets
parameters:
- name: bucketId
in: path
description: ID of the bucket to delete
required: true
schema:
type: string
- name: force
in: query
description: Whether to force deletion of non-empty bucket
required: false
schema:
type: boolean
default: false
responses:
'200':
description: Bucket deleted successfully
content:
application/json:
schema:
type: object
properties:
success:
type: boolean
description: Whether deletion was successful
message:
type: string
description: Optional message (e.g., for errors)
'400':
description: Invalid request
'401':
description: Unauthorized
'403':
description: Forbidden
'404':
description: Bucket not found
/api/buckets/{bucketId}/files:
post:
summary: Upload a file to a bucket
operationId: uploadFile
tags:
- files
parameters:
- name: bucketId
in: path
description: ID of the bucket to upload to
required: true
schema:
type: string
requestBody:
required: true
content:
multipart/form-data:
schema:
type: object
required:
- file
properties:
file:
type: string
format: binary
description: File content
isManaged:
type: boolean
description: Whether the file should be managed
default: false
metadata:
type: object
description: Optional additional metadata
/api/buckets/{bucketId}/multipart:
post:
summary: Initialize a multipart upload
operationId: initMultipartUpload
tags:
- files
parameters:
- name: bucketId
in: path
description: ID of the bucket to upload to
required: true
schema:
type: string
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- fileName
- fileSize
- mimeType
properties:
fileName:
type: string
description: Name of the file
fileSize:
type: integer
description: Total size of the file in bytes
mimeType:
type: string
description: MIME type of the file
isManaged:
type: boolean
description: Whether the file should be managed
default: false
metadata:
type: object
description: Optional additional metadata
responses:
'201':
description: Multipart upload initialized
content:
application/json:
schema:
type: object
properties:
uploadId:
type: string
description: ID for the multipart upload session
fileId:
type: string
description: ID of the file being uploaded
bucketId:
type: string
description: ID of the bucket
partSize:
type: integer
description: Recommended part size in bytes
expiresAt:
type: string
format: date-time
description: When this upload session expires if not completed
'400':
description: Invalid request
'401':
description: Unauthorized
'403':
description: Forbidden
'404':
description: Bucket not found
/api/buckets/{bucketId}/multipart/{uploadId}/parts:
post:
summary: Upload a part for a multipart upload
operationId: uploadPart
tags:
- files
parameters:
- name: bucketId
in: path
description: ID of the bucket
required: true
schema:
type: string
- name: uploadId
in: path
description: ID of the multipart upload
required: true
schema:
type: string
- name: partNumber
in: query
description: Part number (1-based)
required: true
schema:
type: integer
minimum: 1
requestBody:
required: true
content:
application/octet-stream:
schema:
type: string
format: binary
description: Part content
responses:
'200':
description: Part uploaded successfully
content:
application/json:
schema:
type: object
properties:
etag:
type: string
description: ETag for the uploaded part
partNumber:
type: integer
description: Part number
size:
type: integer
description: Size of the part in bytes
'400':
description: Invalid request
'401':
description: Unauthorized
'403':
description: Forbidden
'404':
description: Upload not found
'409':
description: Conflict with existing part
/api/buckets/{bucketId}/multipart/{uploadId}:
get:
summary: Get multipart upload status
operationId: getMultipartStatus
tags:
- files
parameters:
- name: bucketId
in: path
description: ID of the bucket
required: true
schema:
type: string
- name: uploadId
in: path
description: ID of the multipart upload
required: true
schema:
type: string
responses:
'200':
description: Multipart upload status
content:
application/json:
schema:
type: object
properties:
uploadId:
type: string
description: ID of the multipart upload
fileId:
type: string
description: ID of the file being uploaded
fileName:
type: string
description: Name of the file
fileSize:
type: integer
description: Total size of the file in bytes
uploadedSize:
type: integer
description: Total size of uploaded parts in bytes
parts:
type: array
items:
type: object
properties:
partNumber:
type: integer
description: Part number
size:
type: integer
description: Size of the part in bytes
etag:
type: string
description: ETag of the part
uploadedAt:
type: string
format: date-time
description: When the part was uploaded
status:
type: string
enum: [ in-progress, complete, aborted ]
description: Status of the multipart upload
expiresAt:
type: string
format: date-time
description: When this upload expires if not completed
'401':
description: Unauthorized
'403':
description: Forbidden
'404':
description: Upload not found
post:
summary: Complete a multipart upload
operationId: completeMultipartUpload
tags:
- files
parameters:
- name: bucketId
in: path
description: ID of the bucket
required: true
schema:
type: string
- name: uploadId
in: path
description: ID of the multipart upload
required: true
schema:
type: string
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- parts
properties:
parts:
type: array
items:
type: object
required:
- partNumber
- etag
properties:
partNumber:
type: integer
description: Part number
etag:
type: string
description: ETag of the part
responses:
'200':
description: Multipart upload completed successfully
content:
application/json:
schema:
type: object
properties:
fileId:
type: string
description: ID of the uploaded file
fileHash:
type: string
description: Hash of the file content
bucketId:
type: string
description: ID of the bucket containing the file
referenceId:
type: string
description: ID of the created file reference
'400':
description: Invalid request or parts missing
'401':
description: Unauthorized
'403':
description: Forbidden
'404':
description: Upload not found
delete:
summary: Abort a multipart upload
operationId: abortMultipartUpload
tags:
- files
parameters:
- name: bucketId
in: path
description: ID of the bucket
required: true
schema:
type: string
- name: uploadId
in: path
description: ID of the multipart upload
required: true
schema:
type: string
responses:
'200':
description: Multipart upload aborted successfully
content:
application/json:
schema:
type: object
properties:
success:
type: boolean
description: Whether abortion was successful
'401':
description: Unauthorized
'403':
description: Forbidden
'404':
description: Upload not found
responses:
'201':
description: File uploaded successfully
content:
application/json:
schema:
type: object
properties:
fileId:
type: string
description: ID of the uploaded file
fileHash:
type: string
description: Hash of the file content
bucketId:
type: string
description: ID of the bucket containing the file
referenceId:
type: string
description: ID of the created file reference
'400':
description: Invalid request
'401':
description: Unauthorized
'403':
description: Forbidden
'404':
description: Bucket not found
'413':
description: File too large
get:
summary: List files in a bucket
operationId: listFiles
tags:
- files
parameters:
- name: bucketId
in: path
description: ID of the bucket to list files from
required: true
schema:
type: string
- name: offset
in: query
description: Pagination offset
required: false
schema:
type: integer
default: 0
- name: limit
in: query
description: Maximum number of results to return
required: false
schema:
type: integer
default: 20
- name: mimeType
in: query
description: Filter by MIME type
required: false
schema:
type: string
- name: nameContains
in: query
description: Filter by name
required: false
schema:
type: string
responses:
'200':
description: List of files
content:
application/json:
schema:
type: object
properties:
files:
type: array
items:
type: object
properties:
id:
type: string
description: File ID
name:
type: string
description: Filename
hash:
type: string
description: File hash
size:
type: integer
description: File size in bytes
mimeType:
type: string
description: MIME type
createdAt:
type: string
format: date-time
description: Creation timestamp
totalCount:
type: integer
description: Total number of files matching filters
'401':
description: Unauthorized
'403':
description: Forbidden
'404':
description: Bucket not found
/api/files/{fileHash}:
get:
summary: Download a file by hash
operationId: downloadFile
tags:
- files
parameters:
- name: fileHash
in: path
description: Hash of the file to download
required: true
schema:
type: string
- name: bucketId
in: query
description: Optional bucket ID (if hash exists in multiple buckets)
required: false
schema:
type: string
- name: range
in: header
description: Range of bytes to download (RFC 7233 format, e.g., 'bytes=0-1023')
required: false
schema:
type: string
responses:
'200':
description: Complete file content
content:
application/octet-stream:
schema:
type: string
format: binary
headers:
Content-Disposition:
schema:
type: string
description: Attachment filename
Content-Type:
schema:
type: string
description: MIME type of the file
Content-Length:
schema:
type: integer
description: Size of the response in bytes
X-File-Size:
schema:
type: integer
description: Total file size in bytes
X-Bucket-Id:
schema:
type: string
description: ID of the bucket containing the file
'206':
description: Partial file content
content:
application/octet-stream:
schema:
type: string
format: binary
headers:
Content-Disposition:
schema:
type: string
description: Attachment filename
Content-Type:
schema:
type: string
description: MIME type of the file
Content-Range:
schema:
type: string
description: Range of bytes returned (e.g., 'bytes 0-1023/146515')
Content-Length:
schema:
type: integer
description: Size of the partial content in bytes
X-File-Size:
schema:
type: integer
description: Total file size in bytes
X-Bucket-Id:
schema:
type: string
description: ID of the bucket containing the file
Accept-Ranges:
schema:
type: string
description:
Indicates server supports range requests (value: 'bytes')
'401':
description: Unauthorized
'403':
description: Forbidden
'404':
description: File not found
/api/buckets/{bucketId}/files/{fileHash}:
delete:
summary: Delete a file from a bucket
operationId: deleteFile
tags:
- files
parameters:
- name: bucketId
in: path
description: ID of the bucket containing the file
required: true
schema:
type: string
- name: fileHash
in: path
description: Hash of the file to delete
required: true
schema:
type: string
responses:
'200':
description: File deleted successfully
content:
application/json:
schema:
type: object
properties:
success:
type: boolean
description: Whether deletion was successful
message:
type: string
description: Optional message (e.g., for errors)
'401':
description: Unauthorized
'403':
description: Forbidden
'404':
description: File or bucket not found
components:
schemas:
Bucket:
type: object
properties:
id:
type: string
description: Unique identifier
name:
type: string
description: Bucket name (e.g., "profile-images", "documents")
ownerId:
type: string
description: Owner (user or group) ID that owns this bucket
description:
type: string
description: Optional description of bucket contents
createdAt:
type: string
format: date-time
description: When the bucket was created
createdBy:
type: string
description: User who created the bucket
settings:
type: object
properties:
maxFileSize:
type: integer
description: Maximum file size in bytes
allowedMimeTypes:
type: array
items:
type: string
description: List of allowed MIME types
File:
type: object
properties:
id:
type: string
description: Unique identifier
bucketId:
type: string
description: ID of the bucket containing this file
hash:
type: string
description: Content hash for deduplication and access
name:
type: string
description: Original filename
size:
type: integer
description: File size in bytes
mimeType:
type: string
description: MIME type
createdAt:
type: string
format: date-time
description: Creation timestamp
isManaged:
type: boolean
description: Whether the file is managed
deleteAt:
type: string
format: date-time
description: Scheduled deletion time for managed files
securitySchemes:
BearerAuth:
type: http
scheme: bearer
bearerFormat: JWT
security:
- BearerAuth: [ ]
4.2 Component Architecture¶
File Management Logic¶
Unmanaged Files¶
Unmanaged files are deleted when:
- A user explicitly deletes the file from their own references
- The system detects that all references to the file have been removed
Managed Files¶
Managed files follow a different lifecycle:
- When all references are removed, the file is marked for deletion with a grace period
- If new references are added during the grace period, the deletion is canceled
- If the grace period expires without new references, the file is permanently deleted
stateDiagram-v2
[*] --> Active: File uploaded
Active --> MarkedForDeletion: All references removed
MarkedForDeletion --> Active: New reference added
MarkedForDeletion --> [*]: Grace period expiredAccess Control¶
Access control rules for the initial implementation:
- Users can create buckets for their own user ID or for groups they belong to
- Users can upload files to buckets owned by their user ID or by groups they belong to
- Users can download files from buckets if they are the owner or belong to the group that owns the bucket
- Users can delete files from buckets owned by their user ID or by groups they belong to
- Only bucket owners can delete buckets
Buckets provide an additional layer of access control, allowing for more organized management of file permissions. More granular access control (such as read-only or write-only permissions) will be implemented in a future phase.
Statistics Tracking¶
The system maintains statistics at both the owner level and the bucket level:
Owner-level Statistics¶
- Total File Size: The sum of the sizes of all files referenced by the owner across all buckets
- Total File Count: The number of files referenced by the owner across all buckets
- Bucket Count: The number of buckets owned by this owner
Bucket-level Statistics¶
- Total File Size: The sum of the sizes of all files in the bucket
- Total File Count: The number of files in the bucket
- Last Activity: Timestamp of the last file operation in the bucket
These statistics are updated transactionally with file operations:
- When a new file is uploaded to a bucket, both owner and bucket statistics are incremented
- When a file is deleted from a bucket, both owner and bucket statistics are decremented
- When a bucket is created or deleted, the owner's bucket count is updated
7. Test Matrix¶
7.1 Unit Coverage¶
- Test individual components like the Storage Service and Metadata Service
7.2 Integration Coverage¶
- Test the interaction between components, especially file reference management
- E2E tests: Validate the complete flow from upload to download and deletion
- Performance tests: Measure upload/download speeds and database operation performance
7.3 Regression Coverage¶
- Security tests: Verify access control mechanisms
9. Rollout and Migration¶
9.1 Phased Plan¶
Phase 1: Core Storage and Bucket Functionality
- Implement bucket creation, listing, and deletion mechanisms
- Implement basic file storage and retrieval mechanisms within buckets
- Create database schema for buckets, file metadata, and references
- Implement file deduplication based on content hash
- Implement multipart upload and partial download functionality for large files
- Build basic bucket management, upload, download, and delete APIs
Phase 2: File Management and Statistics
- Implement managed vs. unmanaged file logic
- Add grace period handling for managed files
- Implement statistics tracking for both owners and buckets
- Add background job for cleaning up expired managed files
- Implement bucket quotas and size limitations
Phase 3: Access Control and Integration
- Implement group-based access control for buckets
- Integrate with authentication system
- Add API endpoints for querying file and bucket metadata
- Implement client-side components for bucket and file management
- Add support for bucket-specific configuration options (allowed file types, etc.)
10. Risks and Mitigations¶
| Risk | Impact | Likelihood | Mitigation |
|---|---|---|---|
| File storage scaling issues | High | Medium | Design for horizontal scaling; consider cloud storage options |
| Data corruption during file operations | High | Low | Implement transactional operations and validation checks |
| Performance degradation with large files | Medium | Medium | Implement chunked uploads/downloads and progress tracking |
| Orphaned files consuming storage | Medium | Medium | Implement periodic audit jobs to detect and clean up orphaned files |
| Bucket proliferation causing management overhead | Medium | High | Implement bucket quotas per owner; provide tools for bucket lifecycle management |
| Inconsistent statistics between owners and buckets | Medium | Medium | Implement reconciliation jobs to ensure statistics consistency |
| Bucket deletion affecting multiple users | High | Low | Require confirmation for bucket deletion; provide restore mechanism |
Alternatives Considered¶
-
Third-party File Storage Service
- Description: Using an existing service like AWS S3 or Google Cloud Storage
- Pros:
- Proven reliability and scalability
- Built-in features like versioning
- Cons:
- Vendor lock-in
- Additional cost
- More complex integration with our permission model
- Why not chosen: Need for tight integration with our custom permission model and metadata requirements
-
Document Database for File Storage
- Description: Using a document database to store both files and metadata
- Pros:
- Simplified architecture
- Transactional operations for files and metadata
- Cons:
- Performance issues with large files
- Database size scaling challenges
- Why not chosen: Separation of concerns and better performance with dedicated file storage
12. Changelog¶
| Date | Change |
|---|---|
| 2025-07-14 | Initial version |
| 2025-07-14 | Added partial upload/download capabilities |
References¶
- Relevant internal services: loryth-cms
- Related documentation: Authentication service documentation, Group management documentation