✨
Singularity
English
English
  • Overview
    • What is Singularity
    • V1 or V2
  • Installation
    • Download binaries
    • Install via docker
    • Built from source
    • Deploy to production
    • Version upgrade
  • Data Preparation
    • Get Started
    • Performance Tuning
  • Content Distribution
    • Distribute CAR files
  • Deal Making
    • Create a deal schedule
  • Topics
    • Inline Preparation
    • Benchmark
  • 💻CLI Reference
    • Menu
    • Ez Prep
    • Version
    • Admin
      • Init
      • Reset
      • Migrate Dataset
      • Migrate Schedule
    • Download
    • Extract Car
    • Deal
      • Schedule
        • Create
        • List
        • Update
        • Pause
        • Resume
        • Remove
      • Send Manual
      • List
    • Run
      • Api
      • Dataset Worker
      • Content Provider
      • Deal Tracker
      • Deal Pusher
      • Download Server
    • Wallet
      • Import
      • List
      • Remove
    • Storage
      • Create
        • Acd
        • Azureblob
        • B2
        • Box
        • Drive
        • Dropbox
        • Fichier
        • Filefabric
        • Ftp
        • Google Cloud Storage
        • Gphotos
        • Hdfs
        • Hidrive
        • Http
        • Internetarchive
        • Jottacloud
        • Koofr / Digi Storage
          • Digistorage
          • Koofr / Digi Storage
          • Other
        • Local
        • Mailru
        • Mega
        • Netstorage
        • Onedrive
        • Oos
          • Env_auth
          • Instance_principal_auth
          • No_auth
          • Resource_principal_auth
          • User_principal_auth
        • Opendrive
        • Pcloud
        • Premiumizeme
        • Putio
        • Qingstor
        • AWS S3 and compliant
          • Aws
          • Alibaba
          • Arvancloud
          • Ceph
          • Chinamobile
          • Cloudflare
          • Digitalocean
          • Dreamhost
          • Huaweiobs
          • Ibmcos
          • Idrive
          • Ionos
          • Liara
          • Lyvecloud
          • Minio
          • Netease
          • Other
          • Qiniu
          • Rackcorp
          • Scaleway
          • Seaweedfs
          • Stackpath
          • Storj
          • Tencentcos
          • Wasabi
        • Seafile
        • Sftp
        • Sharefile
        • Sia
        • Smb
        • Storj
          • Existing
          • New
        • Sugarsync
        • Swift
        • Union
        • Uptobox
        • Webdav
        • Yandex
        • Zoho
      • Explore
      • List
      • Remove
      • Update
        • Acd
        • Azureblob
        • B2
        • Box
        • Drive
        • Dropbox
        • Fichier
        • Filefabric
        • Ftp
        • Google Cloud Storage
        • Gphotos
        • Hdfs
        • Hidrive
        • Http
        • Internetarchive
        • Jottacloud
        • Koofr / Digi Storage
          • Digistorage
          • Koofr / Digi Storage
          • Other
        • Local
        • Mailru
        • Mega
        • Netstorage
        • Onedrive
        • Oos
          • Env_auth
          • Instance_principal_auth
          • No_auth
          • Resource_principal_auth
          • User_principal_auth
        • Opendrive
        • Pcloud
        • Premiumizeme
        • Putio
        • Qingstor
        • AWS S3 and compliant
          • Aws
          • Alibaba
          • Arvancloud
          • Ceph
          • Chinamobile
          • Cloudflare
          • Digitalocean
          • Dreamhost
          • Huaweiobs
          • Ibmcos
          • Idrive
          • Ionos
          • Liara
          • Lyvecloud
          • Minio
          • Netease
          • Other
          • Qiniu
          • Rackcorp
          • Scaleway
          • Seaweedfs
          • Stackpath
          • Storj
          • Tencentcos
          • Wasabi
        • Seafile
        • Sftp
        • Sharefile
        • Sia
        • Smb
        • Storj
          • Existing
          • New
        • Sugarsync
        • Swift
        • Union
        • Uptobox
        • Webdav
        • Yandex
        • Zoho
      • Rename
    • Prep
      • Create
      • List
      • Status
      • Rename
      • Attach Source
      • Attach Output
      • Detach Output
      • Start Scan
      • Pause Scan
      • Start Pack
      • Pause Pack
      • Start Daggen
      • Pause Daggen
      • List Pieces
      • Add Piece
      • Explore
      • Attach Wallet
      • List Wallets
      • Detach Wallet
      • Remove
  • 🌐Web API Reference
    • Admin
    • Deal Schedule
    • Deal
    • File
    • Job
    • Piece
    • Preparation
    • Storage
    • Wallet Association
    • Wallet
    • Specification
  • ❓FAQ
    • Database is locked
Powered by GitBook
On this page
  • 1. Initialize the Database
  • 2. Connect to Storage Systems
  • 2a. Add a local file system
  • 2b. Add a S3 data source
  • 3. Create a preparation
  • 4. Run the preparation worker
  • 5. Check the preparation status and result

Was this helpful?

Edit on GitHub
  1. Data Preparation

Get Started

Follow these steps to set up and start using Singularity.

1. Initialize the Database

If you're using Singularity for the first time, you'll need to initialize the database. This step is required only once.

singularity admin init

2. Connect to Storage Systems

Singularity partners with RClone to provide seamless integration with over 40 different storage systems. These storage systems can play two main roles:

  • Source Storage: This is where the dataset is currently stored and where Singularity will source data from for preparation.

  • Output Storage: This is the destination where Singularity will store the CAR (Content Addressable Archive) files after processing. Choose a storage system appropriate for your needs and connect it with Singularity to start preparing your datasets.

2a. Add a local file system

The most command storage system is the local file system. To add a folder as a source storage to singularity:

singularity storage create local --name "my-source" --path "/mnt/dataset/folder"

2b. Add a S3 data source

Any S3 compatible storage system can be used, including AWS S3, MinIO, etc. Below is an example for public dataset

singularity storage create s3 aws --name "my-source" --path "public-dataset-test"

3. Create a preparation

singularity prep create --source "my-source" --name "my-prep"

4. Run the preparation worker

singularity prep start-scan my-prep my-source
singularity run dataset-worker

5. Check the preparation status and result

singularity prep status my-prep
singularity prep list-pieces my-prep
PreviousVersion upgradeNextPerformance Tuning

Last updated 1 year ago

Was this helpful?