feat(storage): S3-совместимое хранилище файлов (SeaweedFS/MinIO) и миграция uploads
- storage.js: абстракция хранилища с драйверами local и s3 (AWS SDK v3), ключи объектов совпадают с текущими путями /uploads/<файл>, поэтому схема БД и URL не меняются - docker-compose.yml: сервис s3 (SeaweedFS, том s3-data, API только на loopback), переменные S3_*/STORAGE_*, restart unless-stopped для app и db - docker-compose.minio.yml: оверрайд S3-сервиса на MinIO (образ из своего зеркала) - server.js/worker.js: чтение и запись файлов только через storage (отдача /uploads/*, миниатюры, share-файлы, zip-отчёты, enhance/apply/rollback, photo-worker), автосоздание бакета, глобальная персистенция загрузок multer - бэкап/восстановление и scripts/backup.sh, restore.sh — через scripts/storage-sync.js - scripts/migrate-to-s3.js: идемпотентная миграция uploads/ в бакет (--dry-run, --verify-only, --delete-local) - админка: блок «Хранилище» в системной информации - .env.example, README.md, AGENTS.md: описание драйверов, переменных и перехода на S3
This commit is contained in:
@@ -8,9 +8,9 @@ This document defines how AI agents should work with the WhatIDo codebase. Follo
|
||||
|
||||
**WhatIDo** — Accounting system for an educational center: attendance journal, student project works, group gallery, detached files, and public showcase pages (share links).
|
||||
|
||||
- **Stack**: Node.js 20 + Express, PostgreSQL 16, Docker Compose, Tailscale (Serve/Funnel)
|
||||
- **Architecture**: Single Express server (`server.js`) + static frontend in `public/`
|
||||
- **Deployment**: Docker Compose (app + db + tailscale), bind-mounted uploads, named volume for Postgres data
|
||||
- **Stack**: Node.js 20 + Express, PostgreSQL 16, Docker Compose, S3-совместимое хранилище файлов, Tailscale (Serve/Funnel)
|
||||
- **Architecture**: Single Express server (`server.js`) + storage abstraction (`storage.js`) + static frontend in `public/`
|
||||
- **Deployment**: Docker Compose (app + db + s3 + tailscale), bind-mounted uploads, named volumes for Postgres and S3 data
|
||||
- **Auth**: Admin-only via `X-Admin-Token` header (value = `ADMIN_PASSWORD` env var). No user sessions.
|
||||
|
||||
---
|
||||
@@ -34,9 +34,18 @@ This document defines how AI agents should work with the WhatIDo codebase. Follo
|
||||
### 3. File Uploads
|
||||
- **Multer configs**: `upload` (images only), `adminUpload` (wider allowed ext), `uploadBackup` (restore)
|
||||
- **Limits**: 10 MB/file, 30 MB total per entry
|
||||
- **Storage**: `uploads/` bind-mounted to host, filenames = `timestamp-random.ext`
|
||||
- **Staging**: Multer always writes to `uploads/` (`timestamp-random.ext`); a global `res.on('finish')` hook persists each uploaded file through `storage.persist` on successful responses (only when `STORAGE_DRIVER=s3`)
|
||||
- **HEIC**: Auto-converted to JPEG via `heic-convert`
|
||||
- **Cleanup**: `safeUnlink` / `sweepOrphanedUploads` — never delete outside `uploads/`
|
||||
- **Cleanup**: `safeUnlink` / `sweepOrphanedUploads` — never delete outside `uploads/` or the configured bucket
|
||||
|
||||
### 3a. Storage (`storage.js`)
|
||||
- **Drivers**: `local` (default, files in `uploads/`) and `s3` (S3-compatible: SeaweedFS by default, MinIO via `docker-compose.minio.yml`)
|
||||
- **Keys are stable**: DB stores `/uploads/<name>`; S3 object keys are the same `<name>` (plus `.originals/<name>`). Never change key format — it would break existing DB rows and URLs
|
||||
- **API**: `put`, `putFile`, `head`, `exists`, `sizeOf`, `getStream`, `getBuffer`, `del`, `copyObject`, `listAll`, `localize`, `persist`, `streamTo`, `downloadAll`, `uploadTree`, `ensureBucket`, `usage`, `pruneCache`
|
||||
- **Rules**: never call `fs.*` on `uploads/` directly in request/worker code — use `storage.*`. `safeUnlink` is the only deletion helper (local + remote, idempotent)
|
||||
- **Read path**: `STORAGE_LOCAL_FALLBACK=1` prefers a local file when it still exists (covers in-flight uploads and partial migration); otherwise the app streams the object from S3
|
||||
- **Cache**: `.thumbs` (WebP miniatures) and `.cache` (originals localized for sharp/zip) live inside `uploads/` and are pruned hourly (`STORAGE_CACHE_MAX_AGE_HOURS`)
|
||||
- **Never publish the S3 API port**: only `127.0.0.1` on the host, file access stays behind app auth/rate limits
|
||||
|
||||
### 4. API Patterns
|
||||
- **Admin routes**: `requireAdmin` middleware (checks `X-Admin-Token`)
|
||||
@@ -102,6 +111,15 @@ This document defines how AI agents should work with the WhatIDo codebase. Follo
|
||||
- Update Multer `fileFilter` functions
|
||||
- Keep `MAX_TOTAL_UPLOAD_BYTES` and per-file limit in sync
|
||||
|
||||
### Migrate files to S3 / switch storage driver
|
||||
1. `docker compose up -d s3`
|
||||
2. `docker compose exec -T app node scripts/migrate-to-s3.js --dry-run` then without the flag (idempotent, size-checked, keeps local files)
|
||||
3. `docker compose exec -T app node scripts/migrate-to-s3.js --verify-only`
|
||||
4. Set `STORAGE_DRIVER=s3` in `.env`, `docker compose up -d app`
|
||||
5. After verification: `docker compose exec -T app node scripts/migrate-to-s3.js --delete-local`
|
||||
- Rollback: `STORAGE_DRIVER=local` + `docker compose up -d app`
|
||||
- Do not run `--delete-local` before the app serves reads from S3 and the verification passes
|
||||
|
||||
---
|
||||
|
||||
## Testing & Verification
|
||||
@@ -121,6 +139,11 @@ curl -H "X-Admin-Token: $ADMIN_PASSWORD" http://localhost:3003/api/groups
|
||||
# Run backup/restore scripts
|
||||
./scripts/backup.sh
|
||||
./scripts/restore.sh backups/whatido-backup-<date>.tar.gz
|
||||
|
||||
# Storage checks
|
||||
docker compose up -d s3
|
||||
docker compose exec -T app node scripts/migrate-to-s3.js --dry-run
|
||||
docker compose exec -T app node scripts/migrate-to-s3.js --verify-only
|
||||
```
|
||||
|
||||
---
|
||||
@@ -128,6 +151,7 @@ curl -H "X-Admin-Token: $ADMIN_PASSWORD" http://localhost:3003/api/groups
|
||||
## Security Checklist (before any change)
|
||||
- [ ] No SQL interpolation — only `$1`, `$2`...
|
||||
- [ ] Upload path validation via `isSafeUploadPath` / `safeUnlink`
|
||||
- [ ] File I/O через `storage.*`, ключи объектов не выходят за пределы бакета/`uploads/`
|
||||
- [ ] Rate limiter on new public routes
|
||||
- [ ] Admin routes behind `requireAdmin`
|
||||
- [ ] No secrets in code — only via env vars
|
||||
@@ -141,15 +165,19 @@ curl -H "X-Admin-Token: $ADMIN_PASSWORD" http://localhost:3003/api/groups
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `server.js` | Entire backend (Express, routes, DB, uploads, backup) |
|
||||
| `worker.js` | Background AI auto-check worker for entry messages |
|
||||
| `storage.js` | Storage abstraction: `local` and `s3` drivers, key normalization, cache/thumb helpers |
|
||||
| `worker.js` | Background AI auto-check worker for entry messages + photo enhance worker |
|
||||
| `db/init.sql` | Initial schema (runs on fresh DB) |
|
||||
| `db/migration.sql` | Idempotent migrations for existing DBs |
|
||||
| `docker-compose.yml` | Service definitions (app, db, tailscale) |
|
||||
| `docker-compose.yml` | Service definitions (app, db, s3, tailscale) |
|
||||
| `docker-compose.minio.yml` | Override: S3 service backed by MinIO instead of SeaweedFS |
|
||||
| `Dockerfile` | App image build |
|
||||
| `public/*.html` | Frontend pages |
|
||||
| `public/admin.js` | Shared frontend logic |
|
||||
| `scripts/backup.sh` | Host-level backup script |
|
||||
| `scripts/restore.sh` | Host-level restore script |
|
||||
| `scripts/backup.sh` | Host-level backup script (DB dump + storage export) |
|
||||
| `scripts/restore.sh` | Host-level restore script (DB dump + storage import) |
|
||||
| `scripts/storage-sync.js` | Export/import all storage objects (used by backup/restore) |
|
||||
| `scripts/migrate-to-s3.js` | One-off/idempotent migration `uploads/` -> S3 bucket |
|
||||
| `scripts/deploy.sh` | Deploy script (pull master, build image with commit version, restart app) |
|
||||
| `start-tailscale.sh` | Tailscale container entrypoint |
|
||||
| `.env.example` | Env var template |
|
||||
@@ -160,6 +188,9 @@ curl -H "X-Admin-Token: $ADMIN_PASSWORD" http://localhost:3003/api/groups
|
||||
|
||||
- ❌ Add dependencies without updating `package.json` and rebuilding
|
||||
- ❌ Write files outside `uploads/` or `certs/`
|
||||
- ❌ Touch `uploads/` with `fs.*` in request/worker code — use `storage.*` (files may live only in S3)
|
||||
- ❌ Run `migrate-to-s3.js --delete-local` before verification and cutover
|
||||
- ❌ Expose the S3 API port publicly (only `127.0.0.1` in compose)
|
||||
- ❌ Commit `.env`, `certs/`, `uploads/`, `backups/`, `node_modules/`
|
||||
- ❌ Expose DB port (5432) outside docker network
|
||||
- ❌ Use `eval`, `Function` constructor, or dynamic code execution
|
||||
@@ -182,6 +213,11 @@ docker compose logs -f app
|
||||
# DB shell
|
||||
docker compose exec db psql -U app -d whereldo
|
||||
|
||||
# S3 storage status and migration verification
|
||||
docker compose up -d s3
|
||||
docker compose exec -T app node scripts/migrate-to-s3.js --verify-only
|
||||
docker compose exec -T app node scripts/migrate-to-s3.js --delete-local
|
||||
|
||||
# Tailscale status
|
||||
docker exec -it whatido-tailscale-1 tailscale status
|
||||
|
||||
|
||||
Reference in New Issue
Block a user