Decoder has internal buffer. If consumed data isn't enough to
advance the state of the decoder, consumed data is buffered to
the internal buffer. It causes performance overhead.
Users can use this method to avoid creating small chunks. Record
batch data must be contiguous data. If users pass small chunks to
the decoder, the decoder needs concatenate small chunks
internally. It causes performance overhead.
This method is provided for users who want to optimize performance. Normal users don't need to use this method.
Here is an example usage for normal users:
garrow_stream_decoder_consume_buffer(decoder, buffer1); garrow_stream_decoder_consume_buffer(decoder, buffer2); garrow_stream_decoder_consume_buffer(decoder, buffer3);
Decoder has internal buffer. If consumed data isn't enough to advance the state of the decoder, consumed data is buffered to the internal buffer. It causes performance overhead.
If you pass garrow_stream_decoer_get_next_required_size() size data to each arrow.stream_decoder.StreamDecoder.consumeBytes/arrow.stream_decoder.StreamDecoder.consumeBuffer call, the decoder doesn't use its internal buffer. It improves performance.
Here is an example usage to avoid using internal buffer:
buffer1 = get_data(garrow_stream_decoder_get_next_required_size(decoder)); garrow_stream_decoder_consume_buffer(buffer1); buffer2 = get_data(garrow_stream_decoder_get_next_required_size(decoder)); garrow_stream_decoder_consume_buffer(buffer2);
Users can use this method to avoid creating small chunks. Record batch data must be contiguous data. If users pass small chunks to the decoder, the decoder needs concatenate small chunks internally. It causes performance overhead.
Here is an example usage to reduce small chunks:
GArrowResizablBuffer *buffer = garrow_resizable_buffer_new(1024, NULL); while ((small_chunk = get_data(&small_chunk_size))) { size_t current_buffer_size = garrow_buffer_get_size(GARROW_BUFFER(buffer)); garrow_resizable_buffer_resize(buffer, current_buffer_size + small_chunk_size, NULL); garrow_mutable_buffer_set_data(GARROW_MUTABLE_BUFFER(buffer), current_buffer_size, small_chunk, small_chunk_size, NULL); if (garrow_buffer_get_size(GARROW_BUFFER(buffer)) < garrow_stream_decoder_get_next_required_size(decoder)) { continue; } garrow_stream_decoder_consume_buffer(decoder, GARROW_BUFFER(buffer), NULL); g_object_unref(buffer); buffer = garrow_resizable_buffer_new(1024, NULL); } if (garrow_buffer_get_size(GARROW_BUFFER(buffer)) > 0) { garrow_stream_decoder_consume_buffer(decoder, GARROW_BUFFER(buffer), NULL); } g_object_unref(buffer);