data.buffer.buffer

buffer

Buffer

class ding.data.buffer.buffer.Buffer(size: int)

Buffer is an abstraction of device storage, third-party services or data structures, For example, memory queue, sum-tree, redis, or di-store.

abstract delete(index: str)

Overview:

Delete one data sample by index

Arguments:

• index (str): Index

abstract get(idx: int) → BufferedData

Overview:

Get item by subscript index

Arguments:

• idx (int): Subscript index

Returns:

• buffered_data (BufferedData): Item from buffer

abstract push(data: Any, meta: dict | None = None) → BufferedData

Overview:

Push data and it’s meta information in buffer.

Arguments:

• data (Any): The data which will be pushed into buffer.

• meta (dict): Meta information, e.g. priority, count, staleness.

Returns:

• buffered_data (BufferedData): The pushed data.

abstract sample(size: int | None = None, indices: List[str] | None = None, replace: bool = False, sample_range: slice | None = None, ignore_insufficient: bool = False, groupby: str | None = None, unroll_len: int | None = None) → List[BufferedData] | List[List[BufferedData]]

Overview:

Sample data with length size.

Arguments:

• size (Optional[int]): The number of the data that will be sampled.

• indices (Optional[List[str]]): Sample with multiple indices.

• replace (bool): If use replace is true, you may receive duplicated data from the buffer.

• sample_range (slice): Sample range slice.

• ignore_insufficient (bool): If ignore_insufficient is true, sampling more than buffer size

  with no repetition will not cause an exception.

• groupby (Optional[str]): Groupby key in meta, i.e. groupby=”episode”

• unroll_len (Optional[int]): Number of consecutive frames within a group.

Returns:

• sample_data (Union[List[BufferedData], List[List[BufferedData]]]):

  A list of data with length size, may be nested if groupby is set.

abstract update(index: str, data: Any | None = None, meta: dict | None = None) → bool

Overview:

Update data and meta by index

Arguments:

• index (str): Index of data.

• data (any): Pure data.

• meta (dict): Meta information.

Returns:

• success (bool): Success or not, if data with the index not exist in buffer, return false.

use(func: Callable) → Buffer

Overview:

Use algorithm middleware to modify the behavior of the buffer. Every middleware should be a callable function, it will receive three argument parts, including: 1. The buffer instance, you can use this instance to visit every thing of the buffer, including the storage. 2. The functions called by the user, there are three methods named push , sample and clear , so you can use these function name to decide which action to choose. 3. The remaining arguments passed by the user to the original function, will be passed in *args .

Each middleware handler should return two parts of the value, including: 1. The first value is done (True or False), if done==True, the middleware chain will stop immediately, no more middleware will be executed during this execution 2. The remaining values, will be passed to the next middleware or the default function in the buffer.

Arguments:

• func (Callable): The middleware handler

Returns:

• buffer (Buffer): The instance self

view() → Buffer

Overview:

A view is a new instance of buffer, with a deepcopy of every property except the storage. The storage is shared among all the buffer instances.

Returns:

• buffer (Buffer): The instance self