I’m trying to design storage for app files across on-prem and cloud. What architecture would you suggest and what should I avoid?

A good default architecture for app-file storage across on-prem and cloud is to separate file storage from the application layer and use a single logical storage interface that can be backed by different physical systems.

Recommended architecture

1. Use an object-storage-first design

For most app files, prefer object storage rather than shared filesystem storage.

• Cloud: S3 / Azure Blob / GCS
• On-prem: MinIO, Ceph RGW, or a vendor S3-compatible object store

Why:

• Easier to scale
• Cheaper for large binary/file payloads
• Better fit for cloud-native apps
• Easier replication and lifecycle management
• Works well across environments if you standardize on S3-style APIs

2. Keep metadata in a database

Store file metadata separately from the file bytes:

• file ID
• original filename
• content type
• size
• checksum
• owner/tenant
• permissions
• storage location/key
• upload state/version

This lets you:

• search/filter files
• track ownership and retention
• support versioning
• migrate storage backends without redesigning the app

3. Put a storage abstraction behind the app

Define a storage service interface in your app, for example:

• upload
• download
• delete
• copy
• getSignedURL
• head/metadata

Then implement:

• cloud object storage adapter
• on-prem object storage adapter
• optional local dev adapter

This avoids hard-coding assumptions into your business logic.

4. Use direct-to-storage uploads where possible

For web/mobile clients, have the app server issue pre-signed URLs so clients upload/download directly to the storage system.

Benefits:

• less load on app servers
• faster uploads
• simpler scaling
• fewer bottlenecks for large files

5. Add an async processing pipeline

If files need scanning, conversion, thumbnailing, OCR, or indexing:

• upload file
• write metadata record as “pending”
• enqueue processing job
• worker validates/scans/transforms
• update metadata to “ready” or “rejected”

This is much more robust than doing everything inline in the request path.

6. Use lifecycle, retention, and tiering policies

Plan for:

• automatic cleanup of temporary files
• archival of old files
• retention windows
• legal hold if needed
• version expiration
• cold storage for infrequently accessed data

7. Encrypt and audit

Make sure you have:

• encryption at rest
• TLS in transit
• per-tenant access controls
• audit logs for access/downloads
• key management strategy
• malware scanning if users upload content

---

A practical reference architecture

Client → uploads via pre-signed URL
→ Object storage
→ Metadata DB
→ Async worker pipeline
→ optional CDN for read-heavy public content

For on-prem/cloud portability:

• Use S3-compatible API everywhere if possible
• Keep app code oblivious to the backend vendor
• Store only object keys in the DB, not raw paths tied to a filesystem

---

What to avoid

1. Avoid storing files as database blobs for general app storage

Unless files are small and tightly coupled to transactional data, avoid putting file bytes directly in a relational DB.

Why:

• DB backups get huge
• performance and replication suffer
• expensive scaling
• harder to serve files efficiently
• operational complexity grows quickly

Use DB only for metadata, not the file body, in most cases.

2. Avoid local disk as the primary system of record

Local files on app servers are fragile:

• fail on instance loss or autoscaling
• hard to replicate
• poor fit for HA
• backups become tricky
• problematic in containers/Kubernetes

Local disk is fine only for:

• temporary caches
• transient processing
• dev/test

3. Avoid mixing concerns in one shared filesystem namespace

A giant shared NAS/CIFS/NFS tree can become a bottleneck and operational pain point.

Common issues:

• locking/consistency headaches
• scaling limits
• permission sprawl
• difficult multi-region support
• app coupling to path structure

If you must use shared filesystem semantics, keep it for specific use cases only.

4. Avoid tight coupling to cloud-specific features

If portability matters, don’t rely heavily on:

• vendor-specific object metadata behaviors
• proprietary eventing semantics
• custom ACL models
• path-like assumptions that only work in one cloud

Abstract where practical, especially around uploads/downloads and lifecycle policies.

5. Avoid synchronous processing on upload

Don’t scan, transform, and index big files in the user request path unless files are tiny and latency is not important.

This causes:

• timeouts
• poor UX
• fragile retries
• scaling issues

6. Avoid “one bucket/container for everything” without structure

You can centralize, but still separate logically by:

• environment
• tenant/classification
• sensitivity
• retention policy
• region
• workload type

This helps with security and lifecycle management.

7. Avoid relying on filenames as identifiers

Users rename files. Same names can repeat. Use immutable IDs or object keys instead.

---

Decision guide

Choose object storage if:

• files are user uploads, media, docs, exports, backups
• you need cloud/on-prem portability
• scale and durability matter
• files are mostly read/write as whole objects

Choose filesystem/NFS if:

• you need POSIX semantics
• apps expect shared file locks or atomic rename behavior
• legacy software requires a mounted path

Choose DB blobs only if:

• files are very small
• strict transactional coupling is required
• the number of files is modest
• operational simplicity outweighs scale concerns

---

Good default pattern

If you want a sane enterprise default, I’d recommend:

• S3-compatible object storage in both on-prem and cloud
• DB for metadata
• pre-signed URLs for client transfers
• async workers for validation/transforms
• CDN for public or frequently downloaded content
• lifecycle/retention policies
• encryption + audit logging

If you want, I can also sketch:

1. a reference diagram,
2. a cloud/on-prem comparison, or
3. a decision matrix for object storage vs NFS vs DB blobs.