Get Started

Follow these steps to set up and start using Singularity.

1. Initialize the Database

If you're using Singularity for the first time, you'll need to initialize the database. This step is required only once.

singularity admin init

2. Connect to Storage Systems

Singularity partners with RClone to provide seamless integration with over 40 different storage systems. These storage systems can play two main roles:

• Source Storage: This is where the dataset is currently stored and where Singularity will source data from for preparation.

• Output Storage: This is the destination where Singularity will store the CAR (Content Addressable Archive) files after processing. Choose a storage system appropriate for your needs and connect it with Singularity to start preparing your datasets.

2a. Add a local file system

The most command storage system is the local file system. To add a folder as a source storage to singularity:

singularity storage create local --name "my-source" --path "/mnt/dataset/folder"

2b. Add a S3 data source

Any S3 compatible storage system can be used, including AWS S3, MinIO, etc. Below is an example for public dataset

singularity storage create s3 aws --name "my-source" --path "public-dataset-test"

3. Create a preparation

singularity prep create --source "my-source" --name "my-prep"

4. Run the preparation worker

singularity prep start-scan my-prep my-source
singularity run dataset-worker

5. Check the preparation status and result

singularity prep status my-prep
singularity prep list-pieces my-prep