Azureblob
NAME:
singularity storage create azureblob - Microsoft Azure Blob Storage
USAGE:
singularity storage create azureblob [command options] [arguments...]
DESCRIPTION:
--account
Azure Storage Account Name.
Set this to the Azure Storage Account Name in use.
Leave blank to use SAS URL or Emulator, otherwise it needs to be set.
If this is blank and if env_auth is set it will be read from the
environment variable `AZURE_STORAGE_ACCOUNT_NAME` if possible.
--env-auth
Read credentials from runtime (environment variables, CLI or MSI).
See the [authentication docs](/azureblob#authentication) for full info.
--key
Storage Account Shared Key.
Leave blank to use SAS URL or Emulator.
--sas-url
SAS URL for container level access only.
Leave blank if using account/key or Emulator.
--tenant
ID of the service principal's tenant. Also called its directory ID.
Set this if using
- Service principal with client secret
- Service principal with certificate
- User with username and password
--client-id
The ID of the client in use.
Set this if using
- Service principal with client secret
- Service principal with certificate
- User with username and password
--client-secret
One of the service principal's client secrets
Set this if using
- Service principal with client secret
--client-certificate-path
Path to a PEM or PKCS12 certificate file including the private key.
Set this if using
- Service principal with certificate
--client-certificate-password
Password for the certificate file (optional).
Optionally set this if using
- Service principal with certificate
And the certificate has a password.
--client-send-certificate-chain
Send the certificate chain when using certificate auth.
Specifies whether an authentication request will include an x5c header
to support subject name / issuer based authentication. When set to
true, authentication requests include the x5c header.
Optionally set this if using
- Service principal with certificate
--username
User name (usually an email address)
Set this if using
- User with username and password
--password
The user's password
Set this if using
- User with username and password
--service-principal-file
Path to file containing credentials for use with a service principal.
Leave blank normally. Needed only if you want to use a service principal instead of interactive login.
$ az ad sp create-for-rbac --name "<name>" \
--role "Storage Blob Data Owner" \
--scopes "/subscriptions/<subscription>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>/blobServices/default/containers/<container>" \
> azure-principal.json
See ["Create an Azure service principal"](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) and ["Assign an Azure role for access to blob data"](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli) pages for more details.
It may be more convenient to put the credentials directly into the
rclone config file under the `client_id`, `tenant` and `client_secret`
keys instead of setting `service_principal_file`.
--use-msi
Use a managed service identity to authenticate (only works in Azure).
When true, use a [managed service identity](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/)
to authenticate to Azure Storage instead of a SAS token or account key.
If the VM(SS) on which this program is running has a system-assigned identity, it will
be used by default. If the resource has no system-assigned but exactly one user-assigned identity,
the user-assigned identity will be used by default. If the resource has multiple user-assigned
identities, the identity to use must be explicitly specified using exactly one of the msi_object_id,
msi_client_id, or msi_mi_res_id parameters.
--msi-object-id
Object ID of the user-assigned MSI to use, if any.
Leave blank if msi_client_id or msi_mi_res_id specified.
--msi-client-id
Object ID of the user-assigned MSI to use, if any.
Leave blank if msi_object_id or msi_mi_res_id specified.
--msi-mi-res-id
Azure resource ID of the user-assigned MSI to use, if any.
Leave blank if msi_client_id or msi_object_id specified.
--use-emulator
Uses local storage emulator if provided as 'true'.
Leave blank if using real azure storage endpoint.
--endpoint
Endpoint for the service.
Leave blank normally.
--upload-cutoff
Cutoff for switching to chunked upload (<= 256 MiB) (deprecated).
--chunk-size
Upload chunk size.
Note that this is stored in memory and there may be up to
"--transfers" * "--azureblob-upload-concurrency" chunks stored at once
in memory.
--upload-concurrency
Concurrency for multipart uploads.
This is the number of chunks of the same file that are uploaded
concurrently.
If you are uploading small numbers of large files over high-speed
links and these uploads do not fully utilize your bandwidth, then
increasing this may help to speed up the transfers.
In tests, upload speed increases almost linearly with upload
concurrency. For example to fill a gigabit pipe it may be necessary to
raise this to 64. Note that this will use more memory.
Note that chunks are stored in memory and there may be up to
"--transfers" * "--azureblob-upload-concurrency" chunks stored at once
in memory.
--list-chunk
Size of blob list.
This sets the number of blobs requested in each listing chunk. Default
is the maximum, 5000. "List blobs" requests are permitted 2 minutes
per megabyte to complete. If an operation is taking longer than 2
minutes per megabyte on average, it will time out (
[source](https://docs.microsoft.com/en-us/rest/api/storageservices/setting-timeouts-for-blob-service-operations#exceptions-to-default-timeout-interval)
). This can be used to limit the number of blobs items to return, to
avoid the time out.
--access-tier
Access tier of blob: hot, cool or archive.
Archived blobs can be restored by setting access tier to hot or
cool. Leave blank if you intend to use default access tier, which is
set at account level
If there is no "access tier" specified, rclone doesn't apply any tier.
rclone performs "Set Tier" operation on blobs while uploading, if objects
are not modified, specifying "access tier" to new one will have no effect.
If blobs are in "archive tier" at remote, trying to perform data transfer
operations from remote will not be allowed. User should first restore by
tiering blob to "Hot" or "Cool".
--archive-tier-delete
Delete archive tier blobs before overwriting.
Archive tier blobs cannot be updated. So without this flag, if you
attempt to update an archive tier blob, then rclone will produce the
error:
can't update archive tier blob without --azureblob-archive-tier-delete
With this flag set then before rclone attempts to overwrite an archive
tier blob, it will delete the existing blob before uploading its
replacement. This has the potential for data loss if the upload fails
(unlike updating a normal blob) and also may cost more since deleting
archive tier blobs early may be chargable.
--disable-checksum
Don't store MD5 checksum with object metadata.
Normally rclone will calculate the MD5 checksum of the input before
uploading it so it can add it to metadata on the object. This is great
for data integrity checking but can cause long delays for large files
to start uploading.
--memory-pool-flush-time
How often internal memory buffer pools will be flushed.
Uploads which requires additional buffers (f.e multipart) will use memory pool for allocations.
This option controls how often unused buffers will be removed from the pool.
--memory-pool-use-mmap
Whether to use mmap buffers in internal memory pool.
--encoding
The encoding for the backend.
See the [encoding section in the overview](/overview/#encoding) for more info.
--public-access
Public access level of a container: blob or container.
Examples:
| <unset> | The container and its blobs can be accessed only with an authorized request.
| | It's a default value.
| blob | Blob data within this container can be read via anonymous request.
| container | Allow full public read access for container and blob data.
--no-check-container
If set, don't attempt to check the container exists or create it.
This can be useful when trying to minimise the number of transactions
rclone does if you know the container exists already.
--no-head-object
If set, do not do HEAD before GET when getting objects.
OPTIONS:
--account value Azure Storage Account Name. [$ACCOUNT]
--client-certificate-password value Password for the certificate file (optional). [$CLIENT_CERTIFICATE_PASSWORD]
--client-certificate-path value Path to a PEM or PKCS12 certificate file including the private key. [$CLIENT_CERTIFICATE_PATH]
--client-id value The ID of the client in use. [$CLIENT_ID]
--client-secret value One of the service principal's client secrets [$CLIENT_SECRET]
--env-auth Read credentials from runtime (environment variables, CLI or MSI). (default: false) [$ENV_AUTH]
--help, -h show help
--key value Storage Account Shared Key. [$KEY]
--sas-url value SAS URL for container level access only. [$SAS_URL]
--tenant value ID of the service principal's tenant. Also called its directory ID. [$TENANT]
Advanced
--access-tier value Access tier of blob: hot, cool or archive. [$ACCESS_TIER]
--archive-tier-delete Delete archive tier blobs before overwriting. (default: false) [$ARCHIVE_TIER_DELETE]
--chunk-size value Upload chunk size. (default: "4Mi") [$CHUNK_SIZE]
--client-send-certificate-chain Send the certificate chain when using certificate auth. (default: false) [$CLIENT_SEND_CERTIFICATE_CHAIN]
--disable-checksum Don't store MD5 checksum with object metadata. (default: false) [$DISABLE_CHECKSUM]
--encoding value The encoding for the backend. (default: "Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8") [$ENCODING]
--endpoint value Endpoint for the service. [$ENDPOINT]
--list-chunk value Size of blob list. (default: 5000) [$LIST_CHUNK]
--memory-pool-flush-time value How often internal memory buffer pools will be flushed. (default: "1m0s") [$MEMORY_POOL_FLUSH_TIME]
--memory-pool-use-mmap Whether to use mmap buffers in internal memory pool. (default: false) [$MEMORY_POOL_USE_MMAP]
--msi-client-id value Object ID of the user-assigned MSI to use, if any. [$MSI_CLIENT_ID]
--msi-mi-res-id value Azure resource ID of the user-assigned MSI to use, if any. [$MSI_MI_RES_ID]
--msi-object-id value Object ID of the user-assigned MSI to use, if any. [$MSI_OBJECT_ID]
--no-check-container If set, don't attempt to check the container exists or create it. (default: false) [$NO_CHECK_CONTAINER]
--no-head-object If set, do not do HEAD before GET when getting objects. (default: false) [$NO_HEAD_OBJECT]
--password value The user's password [$PASSWORD]
--public-access value Public access level of a container: blob or container. [$PUBLIC_ACCESS]
--service-principal-file value Path to file containing credentials for use with a service principal. [$SERVICE_PRINCIPAL_FILE]
--upload-concurrency value Concurrency for multipart uploads. (default: 16) [$UPLOAD_CONCURRENCY]
--upload-cutoff value Cutoff for switching to chunked upload (<= 256 MiB) (deprecated). [$UPLOAD_CUTOFF]
--use-emulator Uses local storage emulator if provided as 'true'. (default: false) [$USE_EMULATOR]
--use-msi Use a managed service identity to authenticate (only works in Azure). (default: false) [$USE_MSI]
--username value User name (usually an email address) [$USERNAME]
Client Config
--client-ca-cert value Path to CA certificate used to verify servers
--client-cert value Path to Client SSL certificate (PEM) for mutual TLS auth
--client-connect-timeout value HTTP Client Connect timeout (default: 1m0s)
--client-expect-continue-timeout value Timeout when using expect / 100-continue in HTTP (default: 1s)
--client-header value [ --client-header value ] Set HTTP header for all transactions (i.e. key=value)
--client-insecure-skip-verify Do not verify the server SSL certificate (insecure) (default: false)
--client-key value Path to Client SSL private key (PEM) for mutual TLS auth
--client-no-gzip Don't set Accept-Encoding: gzip (default: false)
--client-scan-concurrency value Max number of concurrent listing requests when scanning data source (default: 1)
--client-timeout value IO idle timeout (default: 5m0s)
--client-use-server-mod-time Use server modified time if possible (default: false)
--client-user-agent value Set the user-agent to a specified string (default: rclone/v1.62.2-DEV)
General
--name value Name of the storage (default: Auto generated)
--path value Path of the storage
Retry Strategy
--client-low-level-retries value Maximum number of retries for low-level client errors (default: 10)
--client-retry-backoff value The constant delay backoff for retrying IO read errors (default: 1s)
--client-retry-backoff-exp value The exponential delay backoff for retrying IO read errors (default: 1.0)
--client-retry-delay value The initial delay before retrying IO read errors (default: 1s)
--client-retry-max value Max number of retries for IO read errors (default: 10)
--client-skip-inaccessible Skip inaccessible files when opening (default: false)
Last updated