Utilities

Decimal Numbers

class arrow::Decimal128 : public arrow::BasicDecimal128

Represents a signed 128-bit integer in two’s complement.

Calculations wrap around and overflow is ignored. The max decimal precision that can be safely represented is 38 significant digits.

For a discussion of the algorithms, look at Knuth’s volume 2, Semi-numerical Algorithms section 4.3.1.

Adapted from the Apache ORC C++ implementation

The implementation is split into two parts :

  1. BasicDecimal128

    • can be safely compiled to IR without references to libstdc++.

  2. Decimal128

    • has additional functionality on top of BasicDecimal128 to deal with strings and streams.

Public Functions

constexpr Decimal128(const BasicDecimal128 &value) noexcept

constructor creates a Decimal128 from a BasicDecimal128.

Decimal128(const std::string &value)

Parse the number from a base 10 string representation.

constexpr Decimal128() noexcept

Empty constructor creates a Decimal128 with a value of 0.

Result<std::pair<Decimal128, Decimal128>> Divide(const Decimal128 &divisor) const

Divide this number by right and return the result.

This operation is not destructive. The answer rounds to zero. Signs work like: 21 / 5 -> 4, 1 -21 / 5 -> -4, -1 21 / -5 -> -4, 1 -21 / -5 -> 4, -1

Return

the pair of the quotient and the remainder

Parameters
  • [in] divisor: the number to divide by

std::string ToString(int32_t scale) const

Convert the Decimal128 value to a base 10 decimal string with the given scale.

std::string ToIntegerString() const

Convert the value to an integer string.

operator int64_t() const

Cast this value to an int64_t.

Result<Decimal128> Rescale(int32_t original_scale, int32_t new_scale) const

Convert Decimal128 from one scale to another.

template<typename T, typename = internal::EnableIfIsOneOf<T, int32_t, int64_t>>
Result<T> ToInteger() const

Convert to a signed integer.

template<typename T, typename = internal::EnableIfIsOneOf<T, int32_t, int64_t>>
Status ToInteger(T *out) const

Convert to a signed integer.

float ToFloat(int32_t scale) const

Convert to a floating-point number (scaled)

double ToDouble(int32_t scale) const

Convert to a floating-point number (scaled)

template<typename T>
T ToReal(int32_t scale) const

Convert to a floating-point number (scaled)

Public Static Functions

Status FromString(const util::string_view &s, Decimal128 *out, int32_t *precision, int32_t *scale = NULLPTR)

Convert a decimal string to a Decimal128 value, optionally including precision and scale if they’re passed in and not null.

Result<Decimal128> FromBigEndian(const uint8_t *data, int32_t length)

Convert from a big-endian byte representation.

The length must be between 1 and 16.

Return

error status if the length is an invalid value

template<>
struct ToRealConversion<double>
template<>
struct ToRealConversion<float>

Abstract Sequences

template<typename T>
class arrow::Iterator

A generic Iterator that can return errors.

Public Functions

template<typename Wrapped>
Iterator(Wrapped has_next)

Iterator may be constructed from any type which has a member function with signature Status Next(T*);.

The argument is moved or copied to the heap and kept in a unique_ptr<void>. Only its destructor and its Next method (which are stored in function pointers) are referenced after construction.

This approach is used to dodge MSVC linkage hell (ARROW-6244, ARROW-6558) when using an abstract template base class: instead of being inlined as usual for a template function the base’s virtual destructor will be exported, leading to multiple definition errors when linking to any other TU where the base is instantiated.

Result<T> Next()

Return the next element of the sequence, IterationTraits<T>::End() when the iteration is completed.

Calling this on a default constructed Iterator will result in undefined behavior.

template<typename Visitor>
Status Visit(Visitor &&visitor)

Pass each element of the sequence to a visitor.

Will return any error status returned by the visitor, terminating iteration.

bool Equals(const Iterator &other) const

Iterators will only compare equal if they are both null.

Equality comparability is required to make an Iterator of Iterators (to check for the end condition).

Result<std::vector<T>> ToVector()

Move every element of this iterator into a vector.

class RangeIterator
template<typename T>
class VectorIterator

Simple iterator which yields the elements of a std::vector.

Compression

enum arrow::Compression::type

Compression algorithm.

Values:

enumerator UNCOMPRESSED
enumerator SNAPPY
enumerator GZIP
enumerator BROTLI
enumerator ZSTD
enumerator LZ4
enumerator LZ4_FRAME
enumerator LZO
enumerator BZ2
enumerator LZ4_HADOOP
class arrow::util::Codec

Compression codec.

Public Functions

Result<int64_t> Decompress(int64_t input_len, const uint8_t *input, int64_t output_buffer_len, uint8_t *output_buffer) = 0

One-shot decompression function.

output_buffer_len must be correct and therefore be obtained in advance. The actual decompressed length is returned.

Note

One-shot decompression is not always compatible with streaming compression. Depending on the codec (e.g. LZ4), different formats may be used.

Result<int64_t> Compress(int64_t input_len, const uint8_t *input, int64_t output_buffer_len, uint8_t *output_buffer) = 0

One-shot compression function.

output_buffer_len must first have been computed using MaxCompressedLen(). The actual compressed length is returned.

Note

One-shot compression is not always compatible with streaming decompression. Depending on the codec (e.g. LZ4), different formats may be used.

Result<std::shared_ptr<Compressor>> MakeCompressor() = 0

Create a streaming compressor instance.

Result<std::shared_ptr<Decompressor>> MakeDecompressor() = 0

Create a streaming compressor instance.

Compression::type compression_type() const = 0

This Codec’s compression type.

const std::string &name() const

The name of this Codec’s compression type.

int compression_level() const

This Codec’s compression level, if applicable.

Public Static Functions

int UseDefaultCompressionLevel()

Return special value to indicate that a codec implementation should use its default compression level.

const std::string &GetCodecAsString(Compression::type t)

Return a string name for compression type.

Result<Compression::type> GetCompressionType(const std::string &name)

Return compression type for name (all upper case)

Result<std::unique_ptr<Codec>> Create(Compression::type codec, int compression_level = kUseDefaultCompressionLevel)

Create a codec for the given compression algorithm.

bool IsAvailable(Compression::type codec)

Return true if support for indicated codec has been enabled.

bool SupportsCompressionLevel(Compression::type codec)

Return true if indicated codec supports setting a compression level.

class arrow::util::Compressor

Streaming compressor interface.

Public Functions

Result<CompressResult> Compress(int64_t input_len, const uint8_t *input, int64_t output_len, uint8_t *output) = 0

Compress some input.

If bytes_read is 0 on return, then a larger output buffer should be supplied.

Result<FlushResult> Flush(int64_t output_len, uint8_t *output) = 0

Flush part of the compressed output.

If should_retry is true on return, Flush() should be called again with a larger buffer.

Result<EndResult> End(int64_t output_len, uint8_t *output) = 0

End compressing, doing whatever is necessary to end the stream.

If should_retry is true on return, End() should be called again with a larger buffer. Otherwise, the Compressor should not be used anymore.

End() implies Flush().

struct CompressResult
struct EndResult
struct FlushResult
class arrow::util::Decompressor

Streaming decompressor interface.

Public Functions

Result<DecompressResult> Decompress(int64_t input_len, const uint8_t *input, int64_t output_len, uint8_t *output) = 0

Decompress some input.

If need_more_output is true on return, a larger output buffer needs to be supplied.

bool IsFinished() = 0

Return whether the compressed stream is finished.

This is a heuristic. If true is returned, then it is guaranteed that the stream is finished. If false is returned, however, it may simply be that the underlying library isn’t able to provide the information.

Status Reset() = 0

Reinitialize decompressor, making it ready for a new compressed stream.

struct DecompressResult