Helpers
linc_convert.utils.io.zarr.helpers ¶
Helper functions for Zarr I/O.
auto_shard_size ¶
auto_shard_size(max_shape, itemsize, max_file_size=2 * 1024 ** 4, compression_ratio=2)
Find maximal shard size that ensures file size below cap.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
max_shape
|
list[int]
|
Maximum shape along each dimension. |
required |
itemsize
|
dtype or int
|
Data type, or data type size |
required |
max_file_size
|
int
|
Maximum file size (default: 2TB). S3 has a 5TB/file limit, but given that we use an estimated compression factor, we aim for 2TB to leave some leeway. |
2 * 1024 ** 4
|
compression_ratio
|
float
|
Estimated compression factor. I roughly found 2 for bosc-compressed LSM data, when compressing only across space and channels (5 channels). |
2
|
Returns:
| Name | Type | Description |
|---|---|---|
shard |
list[int]
|
Estimated shard size along each dimension. Returned shards are either max_shape or powers of two. |
fix_shard_chunk ¶
fix_shard_chunk(shard, chunk, shape)
Fix incompatibilities between chunk and shard size.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
shard
|
list[int]
|
|
required |
chunk
|
list[int]
|
|
required |
shape
|
list[int]
|
|
required |
Returns:
| Name | Type | Description |
|---|---|---|
shard |
list[int]
|
|
chunk |
list[int]
|
|