WhatIDo/AGENTS.md

AGENT.md — Developer Agent Guidelines for WhatIDo

This document defines how AI agents should work with the WhatIDo codebase. Follow these rules strictly.

---

Project Overview

WhatIDo — Accounting system for an educational center: attendance journal, student project works, group gallery, detached files, and public showcase pages (share links).

• Stack: Node.js 20 + Express, PostgreSQL 16, Redis 7, Docker Compose, S3-совместимое хранилище файлов, Tailscale (Serve/Funnel)
• Architecture: Single Express server (server.js) + storage abstraction (storage.js) + cache/pub-sub abstraction (redis.js) + static frontend in public/
• Deployment: Docker Compose (app + db + s3 + tailscale), bind-mounted uploads, named volumes for Postgres and S3 data
• Auth: Admin-only via X-Admin-Token header (value = ADMIN_PASSWORD env var). No user sessions.

---

Development Rules

1. Code Style

• No comments unless explicitly requested
• ES modules not used — CommonJS (require) throughout
• Error handling: try/catch with explicit status codes, no global error handler
• Validation: Inline helper functions (reqInt, reqStr, optInt, etc.) — use them
• Security first: All uploads validated, path traversal blocked, rate limits on public routes

2. Database

• Schema: Defined in db/init.sql (runs on first container start)
• Migrations: db/migration.sql for existing DBs — update both when changing schema
• Connection: Single Pool from pg, DATABASE_URL from env
• Queries: Parameterized only ($1, $2...), never string interpolation
• Transactions: Use client.query('BEGIN') / COMMIT / ROLLBACK for multi-statement ops

3. File Uploads

• Multer configs: upload (images only), adminUpload (wider allowed ext), uploadBackup (restore)
• Limits: 10 MB/file, 30 MB total per entry
• Staging: Multer always writes to uploads/ (timestamp-random.ext); a global res.on('finish') hook persists each uploaded file through storage.persist on successful responses (only when STORAGE_DRIVER=s3)
• HEIC: Auto-converted to JPEG via heic-convert
• Cleanup: safeUnlink / sweepOrphanedUploads — never delete outside uploads/ or the configured bucket

3a. Storage (storage.js)

• Drivers: local (default, files in uploads/) and s3 (S3-compatible: SeaweedFS by default, MinIO via docker-compose.minio.yml)
• Keys are stable: DB stores /uploads/<name>; S3 object keys are the same <name> (plus .originals/<name>). Never change key format — it would break existing DB rows and URLs
• API: put, putFile, head, exists, sizeOf, getStream, getBuffer, del, copyObject, listAll, localize, persist, streamTo, downloadAll, uploadTree, ensureBucket, usage, pruneCache
• Rules: never call fs.* on uploads/ directly in request/worker code — use storage.*. safeUnlink is the only deletion helper (local + remote, idempotent)
• Read path: STORAGE_LOCAL_FALLBACK=1 prefers a local file when it still exists (covers in-flight uploads and partial migration); otherwise the app streams the object from S3
• Cache: .thumbs (WebP miniatures) and .cache (originals localized for sharp/zip) live inside uploads/ and are pruned hourly (STORAGE_CACHE_MAX_AGE_HOURS)
• Never publish the S3 API port: only 127.0.0.1 on the host, file access stays behind app auth/rate limits

3b. Redis (redis.js)

• Единственная точка доступа: createRedis({ url, prefix }) — все операции кэша/счётчиков/pub-sub идут через неё
• API: get, set, del, dropPrefix, dropMatch, clear, wrap, incr, publish, on, rateLimitStore, info, connect, close
• Graceful fallback — обязательное требование: при недоступном Redis все операции уходят в in-memory backend с той же семантикой. Приложение обязано стартовать и работать без Redis
• Первое подключение ограничено по времени (REDIS_CONNECT_TIMEOUT_MS, 5 с): node-redis не отклоняет connect() при недоступном сервере, а повторяет попытки бесконечно — без таймаута старт приложения зависнет навсегда
• Переподключение: node-redis переподключается сам; по событию ready подписки и subscriber-клиент восстанавливаются (ensureSubscriber). Не пересоздавать subscriber через destroy() — это гонка с внутренним teardown node-redis
• Ключи: get/set сами добавляют namespace (REDIS_PREFIX, по умолчанию whatido), dropPrefix/dropMatch тоже. В rateLimitStore префикс добавляется один раз в base — не применяйте fullKey повторно
• scanDelete: курсор SCAN в node-redis v5 обязан быть строкой, числовой 0 вызовет TypeError. Возвращаемое значение курсора — тоже строка, сравнивайте с '0'
• resetTime в rateLimitStore.increment обязан быть Date — express-rate-limit v8 вызывает resetTime.getTime()
• Пабликация всегда отдаёт подписчикам строку (JSON), независимо от бэкенда — иначе fallback и Redis расходятся по формату
• Инвалидация — по префиксу (SCAN + DEL), точечного удаления по ключу избегайте
• Ключевые пространства: setting:, groups:, students:, entries:, stats:, dashboard:, share:payload:, public-settings, system-info, session:, ban:, fail:, rl:
• Сессии: loadUserByToken кэширует пользователя на 30 с. Любая мутация users / sessions / user_branches обязана вызывать invalidateSessions() или удалять session:<token>, иначе деактивированный пользователь сохранит доступ
• Секреты: пароль только в REDIS_URL / REDIS_PASSWORD, порт 6379 публикуется лишь на 127.0.0.1

4. API Patterns

• Admin routes: requireAdmin middleware (checks X-Admin-Token)
• Public routes: apiLimiter (300/15min), entryLimiter (10/15min), fileLimiter (300/15min) — все на cache.rateLimitStore(...), не на MemoryStore
• Responses: JSON, { error: 'message' } on failure, data directly on success
• Pagination: limit / offset query params, return { items, total } or { entries, total }
• Filters: group_id, date_from, date_to, student_name, search, deleted

5. Frontend (public/)

• Vanilla HTML/CSS/JS, no build step
• Each page = single HTML file + shared admin.js / admin.css
• API calls via fetch with X-Admin-Token from localStorage
• Share pages (share.html, links.html) work without auth

6. Docker / Compose

• Dockerfile: Node 20 Alpine, installs deps, generates self-signed TLS cert
• docker-compose.yml: сервисы db, app, redis, s3 (+ опционально tailscale, cloudflared, text-corrector, photo-ai)
  • db: postgres:16-alpine, healthcheck, init.sql mounted
  • redis: redis:7-alpine, --requirepass, AOF, maxmemory + allkeys-lru, healthcheck, том redis-data, порт только на 127.0.0.1
  • app: builds from Dockerfile, exposes 3003/3443, mounts uploads
  • tailscale: host network, NET_ADMIN, runs start-tailscale.sh (funnel to 127.0.0.1:3443)
• Env vars (required): ADMIN_PASSWORD, DB_PASSWORD, REDIS_PASSWORD
• Env vars (optional): REDIS_PREFIX (default whatido), REDIS_MAXMEMORY (default 256mb), REDIS_CONNECT_TIMEOUT_MS (default 5000)
• Port 443 on host must be free (tailscale listens directly)

7. Tailscale Publication

• No external IP / port forwarding needed
• Access: https://whatido.<tailnet>.ts.net (inside tailnet + internet via Funnel)
• First run: docker exec -it whatido-tailscale-1 tailscale up --hostname=whatido → authorize in browser
• Enable Serve/Funnel in Tailscale admin console for the node
• Cert: app generates self-signed cert at build (certs/cert.pem), mounted into tailscale container

8. Backup / Restore

• Admin UI: /api/backup (download tar.gz), /api/restore (upload tar.gz)
• Scripts: scripts/backup.sh, scripts/restore.sh (host-level)
• Backup format: data.json (all tables) + uploads/ directory
• Restore validates all data, resets sequences, sweeps orphans

---

Common Tasks

Add a new API endpoint

1. Add route in server.js (group with related routes)
2. Use requireAdmin for admin, apiLimiter/fileLimiter for public
3. Validate input with helper functions
4. Use parameterized queries, transactions if multi-table
5. Call logAudit(req, 'action.name', { ... }) for mutations
6. Return JSON, handle errors with appropriate status codes

Add a database column/table

1. Update db/init.sql (CREATE TABLE / ALTER TABLE)
2. Update db/migration.sql (idempotent ALTERs)
3. Update server.js queries that SELECT/INSERT the table
4. Test: docker compose down && docker compose up -d --build

Add a frontend page

1. Create public/newpage.html (copy structure from existing)
2. Link in public/admin.html navigation if admin page
3. Use admin.js utilities: api(), requireAuth(), formatDate(), etc.
4. No build step — just refresh browser

Modify file upload rules

• Edit BLOCKED_EXT, ALLOWED_IMAGE_EXT, ADMIN_ALLOWED_EXT constants
• Update Multer fileFilter functions
• Keep MAX_TOTAL_UPLOAD_BYTES and per-file limit in sync

Migrate files to S3 / switch storage driver

1. docker compose up -d s3
2. docker compose exec -T app node scripts/migrate-to-s3.js --dry-run then without the flag (idempotent, size-checked, keeps local files)
3. docker compose exec -T app node scripts/migrate-to-s3.js --verify-only
4. Set STORAGE_DRIVER=s3 in .env, docker compose up -d app
5. After verification: docker compose exec -T app node scripts/migrate-to-s3.js --delete-local

• Rollback: STORAGE_DRIVER=local + docker compose up -d app
• Do not run --delete-local before the app serves reads from S3 and the verification passes

---

Testing & Verification

No automated test suite exists. Verify manually:

# Start stack
docker compose up -d --build

# Check logs
docker compose logs -f app

# Test API (replace TOKEN)
curl -H "X-Admin-Token: $ADMIN_PASSWORD" http://localhost:3003/api/groups

# Run backup/restore scripts
./scripts/backup.sh
./scripts/restore.sh backups/whatido-backup-<date>.tar.gz

# Storage checks
docker compose up -d s3
docker compose exec -T app node scripts/migrate-to-s3.js --dry-run
docker compose exec -T app node scripts/migrate-to-s3.js --verify-only

# Redis checks
node redis.selftest.js                                  # unit + degradation, needs redis on 127.0.0.1:6379
node api.smoketest.js                                   # e2e, needs running stack
docker compose exec redis redis-cli -a "$REDIS_PASSWORD" --no-auth-warning INFO
docker compose stop redis && node api.smoketest.js      # app must keep working in-memory
docker compose start redis                              # app reconnects on its own

Verify Redis state through GET /api/system-info → cache (driver, ready, hits, misses, fallbackOps, used_memory_human, keys).

---

Security Checklist (before any change)

• No SQL interpolation — only $1, $2...
• Upload path validation via isSafeUploadPath / safeUnlink
• File I/O через storage.*, ключи объектов не выходят за пределы бакета/uploads/
• Rate limiter on new public routes
• Admin routes behind requireAdmin
• No secrets in code — only via env vars
• Helmet headers present (already global)
• CORS disabled (no cors middleware)

---

File Map (key files)

File	Purpose
server.js	Entire backend (Express, routes, DB, uploads, backup)
storage.js	Storage abstraction: local and s3 drivers, key normalization, cache/thumb helpers
redis.js	Redis abstraction: cache, counters, rate-limit store, pub/sub, in-memory fallback
redis.selftest.js	Self-tests for redis.js, including behaviour with Redis unavailable
api.smoketest.js	End-to-end API smoke test against a running stack
worker.js	Background AI auto-check worker for entry messages + photo enhance worker
db/init.sql	Initial schema (runs on fresh DB)
db/migration.sql	Idempotent migrations for existing DBs
docker-compose.yml	Service definitions (app, db, s3, tailscale)
docker-compose.minio.yml	Override: S3 service backed by MinIO instead of SeaweedFS
Dockerfile	App image build
public/*.html	Frontend pages
public/admin.js	Shared frontend logic
scripts/backup.sh	Host-level backup script (DB dump + storage export)
scripts/restore.sh	Host-level restore script (DB dump + storage import)
scripts/storage-sync.js	Export/import all storage objects (used by backup/restore)
scripts/migrate-to-s3.js	One-off/idempotent migration uploads/ -> S3 bucket
scripts/deploy.sh	Deploy script (pull master, build image with commit version, restart app)
start-tailscale.sh	Tailscale container entrypoint
.env.example	Env var template

---

Do Not

• ❌ Add dependencies without updating package.json and rebuilding
• ❌ Write files outside uploads/ or certs/
• ❌ Touch uploads/ with fs.* in request/worker code — use storage.* (files may live only in S3)
• ❌ Run migrate-to-s3.js --delete-local before verification and cutover
• ❌ Expose the S3 API port publicly (only 127.0.0.1 in compose)
• ❌ Expose the Redis port publicly (only 127.0.0.1 in compose)
• ❌ Call fs.*/pg directly for cache, counters or pub/sub — use redis.js
• ❌ Make Redis a hard dependency: any new Redis-backed path must keep the in-memory fallback
• ❌ await client.connect() without a timeout — it never rejects while Redis is unreachable
• ❌ Cache authorization-relevant data without an invalidation path on the mutation
• ❌ Commit .env, certs/, uploads/, backups/, node_modules/
• ❌ Expose DB port (5432) outside docker network
• ❌ Use eval, Function constructor, or dynamic code execution
• ❌ Add comments to code (this file excepted)

---

Quick Commands

# Full rebuild
docker compose down && docker compose up -d --build

# Обновление на сервере (pull master + сборка образа с версией коммита + перезапуск app)
./scripts/deploy.sh

# App logs
docker compose logs -f app

# DB shell
docker compose exec db psql -U app -d whereldo

# Redis status and cache keys
docker compose exec redis redis-cli -a "$REDIS_PASSWORD" --no-auth-warning DBSIZE
docker compose exec redis redis-cli -a "$REDIS_PASSWORD" --no-auth-warning KEYS 'whatido:*'

# S3 storage status and migration verification
docker compose up -d s3
docker compose exec -T app node scripts/migrate-to-s3.js --verify-only
docker compose exec -T app node scripts/migrate-to-s3.js --delete-local

# Tailscale status
docker exec -it whatido-tailscale-1 tailscale status

# Manual funnel restart
docker exec whatido-tailscale-1 tailscale funnel --bg --yes https://127.0.0.1:3443