tensorplay@1.0.0.dev20260909

Wraps another sampler to yield a mini-batch of indices.

Parameters:

• sampler (Sampler or Iterable) – Base sampler. Can be any iterable object

• batch_size (int) – Size of mini-batch.

• drop_last (bool) – If True, the sampler will drop the last batch if its size would be less than batch_size

Dataset for chaining multiple IterableDataset s.

This class is useful to assemble different existing dataset streams. The chaining operation is done on-the-fly, so concatenating large-scale datasets with this class will be efficient.

Parameters:

datasets (iterable of IterableDataset) – datasets to be chained together

Dataset as a concatenation of multiple datasets.

This class is useful to assemble different existing datasets.

Parameters:

datasets (sequence) – List of datasets to be concatenated

Data loader combines a dataset and a sampler, and provides an iterable over the given dataset.

The DataLoader supports both map-style and iterable-style datasets with single- or multi-process loading, customizing loading order and optional automatic batching (collation).

Parameters:

• dataset (Dataset) – dataset from which to load the data.

• batch_size (int, optional) – how many samples per batch to load (default: 1).

• shuffle (bool, optional) – set to True to have the data reshuffled at every epoch (default: False).

• sampler (Sampler or Iterable, optional) – defines the strategy to draw samples from the dataset. Can be any Iterable with __len__ implemented. If specified, shuffle must not be specified.

• batch_sampler (Sampler or Iterable, optional) – like sampler, but returns a batch of indices at a time. Mutually exclusive with batch_size, shuffle, sampler, and drop_last.

• num_workers (int, optional) – how many subprocesses to use for data loading. 0 means that the data will be loaded in the main process. (default: 0)

• collate_fn (Callable, optional) – merges a list of samples to form a mini-batch of Tensor(s). Used when using batched loading from a map-style dataset.

• pin_memory (bool, optional) – If True, the data loader will copy Tensors into CUDA page-locked host memory before returning them.

• drop_last (bool, optional) – set to True to drop the last incomplete batch, if the dataset size is not divisible by the batch size. (default: False)

• timeout (numeric, optional) – if positive, the timeout value for collecting a batch from workers. Should always be non-negative. (default: 0)

• worker_init_fn (Callable, optional) – If not None, this will be called on each worker subprocess with the worker id (an int in [0, num_workers - 1]) as input, before data loading. (default: None)

• multiprocessing_context (str or context, optional) – start method or multiprocessing context used to spawn the workers, e.g., "fork" or "spawn". If None, the default context of the platform is used. (default: None)

• generator (Generator, optional) – If not None, this RNG will be used by RandomSampler to generate random indexes. (default: None)

• prefetch_factor (int, optional) – Number of batches loaded in advance by each worker. 2 means there will be a total of 2 * num_workers batches prefetched across all workers. (default: 2 when num_workers > 0; otherwise must be None)

• persistent_workers (bool, optional) – If True, the data loader will not shut down the worker processes after a dataset has been consumed once. This allows to maintain the workers Dataset instances alive. (default: False)

• pin_memory_device (str, optional) – Deprecated device spelling kept for accelerator for pinned host allocations. (default: "")

• in_order (bool, optional) – If False, the data loader will not enforce that batches returned from multiprocessing workers are provided in the order the sampler produced them. This enables faster delivery of batches that complete early, at the cost of batch order no longer being deterministic. (default: True)

• device (str, optional) – device to move batches to after collation.

An abstract class representing a Dataset.

All datasets that represent a map from keys to data samples should subclass it. All subclasses should overwrite __getitem__(), supporting fetching a data sample for a given key. Subclasses could also optionally overwrite __len__(), which is expected to return the size of the dataset by many Sampler implementations and the default options of DataLoader.

An iterable Dataset.

All datasets that represent an iterable of data samples should subclass it. Such form of datasets is particularly useful when data come from a stream.

All subclasses should overwrite __iter__(), which would return an iterator of samples in this dataset.

When a subclass is used with DataLoader, each item in the dataset will be yielded from the DataLoader iterator.

Samples elements randomly. If without replacement, then sample from a shuffled dataset. If with replacement, then user can specify num_samples to draw.

Parameters:

• data_source (Sized) – data source to sample from. Must implement __len__.

• replacement (bool) – samples are drawn on-demand with replacement if True, default=``False``.

• num_samples (int) – number of samples to draw, default=`len(dataset)`.

• generator (Generator) – Generator used in sampling.

Base class for all Samplers.

Every Sampler subclass has to provide an __iter__() method, providing a way to iterate over indices or lists of indices (batches) of dataset elements, and may provide a __len__() method that returns the length of the returned iterators.

Samples elements sequentially, always in the same order.

Parameters:

data_source (Sized) – data source to sample from. Must implement __len__.

Dataset as a stacking of multiple datasets.

This class is useful to assemble different parts of complex input data, given as datasets.

Example

>>> images = ImageDataset()
>>> texts = TextDataset()
>>> tuple_stack = StackDataset(images, texts)
>>> tuple_stack[0] == (images[0], texts[0])
>>> dict_stack = StackDataset(image=images, text=texts)
>>> dict_stack[0] == {"image": images[0], "text": texts[0]}

Parameters:

• *args (Dataset) – Datasets for stacking returned as tuple.

• **kwargs (Dataset) – Datasets for stacking returned as dict.

Subset of a dataset at specified indices.

Parameters:

• dataset (Dataset) – The whole Dataset

• indices (sequence) – Indices in the whole set selected for subset

Samples elements randomly from a given list of indices, without replacement.

Parameters:

• indices (sequence) – a sequence of indices

• generator (Generator) – Generator used in sampling.

Dataset wrapping tensors.

Each sample will be retrieved by indexing tensors along the first dimension.

Parameters:

*tensors (Tensor) – tensors that have the same size of the first dimension.

Samples elements from [0,..,len(weights)-1] with given probabilities (weights).

Parameters:

• weights (sequence) – a sequence of weights, not necessary summing up to one

• num_samples (int) – number of samples to draw

• replacement (bool) – if True, samples are drawn with replacement. If not, they are drawn without replacement, which means that when a sample index is drawn for a row, it cannot be drawn again for that row.

• generator (Generator) – Generator used in sampling.