pyarrow.csv.CSVStreamingReader

class pyarrow.csv.CSVStreamingReader

Bases: pyarrow.lib.RecordBatchReader

An object that reads record batches incrementally from a CSV file.

Should not be instantiated directly by user code.

__init__(*args, **kwargs)

Initialize self. See help(type(self)) for accurate signature.

Methods

__init__(*args, **kwargs)	Initialize self.
from_batches(schema, batches)	Create RecordBatchReader from an iterable of batches.
get_next_batch(self)
read_all(self)	Read all record batches as a pyarrow.Table.
read_next_batch(self)	Read next RecordBatch from the stream.
read_pandas(self, **options)	Read contents of stream to a pandas.DataFrame.

Attributes

schema

static from_batches(schema, batches)

Create RecordBatchReader from an iterable of batches.

Parameters

• schema (Schema) – The shared schema of the record batches

• batches (Iterable[RecordBatch]) – The batches that this reader will return.

Returns

reader (RecordBatchReader)

get_next_batch(self)

read_all(self)

Read all record batches as a pyarrow.Table.

read_next_batch(self)

Read next RecordBatch from the stream.

Raises

StopIteration: – At end of stream.

read_pandas(self, **options)

Read contents of stream to a pandas.DataFrame.

Read all record batches as a pyarrow.Table then convert it to a pandas.DataFrame using Table.to_pandas.

Parameters

**options (arguments to forward to Table.to_pandas) –

Returns

df (pandas.DataFrame)

schema

pyarrow.csv.open_csv pyarrow.feather.read_feather