Unified

NAME:
   singularity run unified - Run unified auto-preparation service (workflow orchestration + worker management)

USAGE:
   singularity run unified [command options]

DESCRIPTION:
   The unified service combines workflow orchestration and worker lifecycle management.
     
   It automatically:
   - Manages dataset worker lifecycle (start/stop workers based on job availability)
   - Orchestrates job progression (scan → pack → daggen → deals)
   - Scales workers up/down based on job queue
   - Handles automatic deal creation when preparations complete

   This is the recommended way to run fully automated data preparation.

OPTIONS:
   --min-workers value               Minimum number of workers to keep running (default: 1)
   --max-workers value               Maximum number of workers to run (default: 5)
   --scale-up-threshold value        Number of ready jobs to trigger worker scale-up (default: 5)
   --scale-down-threshold value      Number of ready jobs below which to scale down workers (default: 2)
   --check-interval value            How often to check for scaling and workflow progression (default: 30s)
   --worker-idle-timeout value       How long a worker can be idle before shutdown (0 = never) (default: 5m0s)
   --disable-auto-scaling            Disable automatic worker scaling (default: false)
   --disable-workflow-orchestration  Disable automatic job progression (default: false)
   --disable-auto-deals              Disable automatic deal creation (default: false)
   --disable-scan-to-pack            Disable automatic scan → pack transitions (default: false)
   --disable-pack-to-daggen          Disable automatic pack → daggen transitions (default: false)
   --disable-daggen-to-deals         Disable automatic daggen → deals transitions (default: false)
   --help, -h                        show help

Last updated

Was this helpful?