large_file_splitter
下の方に日本語の説明があります
Overview
- The tool considers a file so large that it does not fit in memory as a single string and performs a split process of the string. The tool stores the result as separate files.
- under construction
Usage
import large_file_splitter
large_file_splitter.split(
"dummy_large_file.txt",
split_str = "SPLIT_MARK\r\n",
div_mode = "start",
output_filename_frame = "./output/div_%d.txt",
cache_size = 10 * 1024 * 1024
)
Example of usage (in the context of a for loop)
import large_file_splitter
for one_str in large_file_splitter.for_split(
"dummy_large_file.txt",
split_str = "SPLIT_MARK\r\n",
div_mode = "start",
cache_size = 1024
):
print(one_str)
概要
- メモリに乗らないほど巨大なファイルを一つの文字列とみなし、文字列のsplit処理を実施。その結果を別々のファイルとして格納するツール。
- 説明は執筆中です
使用例
import large_file_splitter
large_file_splitter.split(
"dummy_large_file.txt",
split_str = "SPLIT_MARK\r\n",
div_mode = "start",
output_filename_frame = "./output/div_%d.txt",
cache_size = 10 * 1024 * 1024
)
使用例 (for文脈での利用)
import large_file_splitter
for one_str in large_file_splitter.for_split(
"dummy_large_file.txt",
split_str = "SPLIT_MARK\r\n",
div_mode = "start",
cache_size = 1024
):
print(one_str)