j+ddlmZddlZddlZddlmZddlmZmZm Z m Z ddlm Z ddl mZddlmZGddZd Zhd Z dd Z dd Z dd ZddZy))SequenceN) _pandas_api)CodecTable concat_tablesschema)_feather) FeatherErrorc.eZdZdZddZddZdZd dZy) FeatherDataseta Encapsulates details of reading a list of Feather files. Parameters ---------- path_or_paths : List[str] A list of file names validate_schema : bool, default True Check that individual file schemas are all the same / compatible c ||_||_yN)pathsvalidate_schema)self path_or_pathsrs 6/tmp/pip-target-jj7clzyk/lib/python/pyarrow/feather.py__init__zFeatherDataset.__init__+s" .NcFt|jd|}|g|_|j|_|jddD]H}t||}|jr|j |||jj |Jt|jS)a, Read multiple feather files as a single pyarrow.Table Parameters ---------- columns : List[str] Names of columns to read from the file Returns ------- pyarrow.Table Content of the file as a table (of columns) rcolumnsN)_read_table_internalr_tablesrrvalidate_schemasappendr)rr_filpathtables r read_tablezFeatherDataset.read_table/s$DJJqM7Cv kk JJqrN 'D(w?E##%%dE2 LL   &  ' T\\**rc|jj|js(td|d|jd|jy)Nz Schema in z was different. z vs )requals ValueError)rpiecer s rrzFeatherDataset.validate_schemasHsN{{!!%,,/z%0B $ }Ju||nFG G0rcF|j|j|S)a Read multiple Parquet files as a single pandas DataFrame Parameters ---------- columns : List[str] Names of columns to read from the file use_threads : bool, default True Use multiple threads when converting to pandas Returns ------- pandas.DataFrame Content of the file as a pandas DataFrame (of columns) r) use_threads)r! to_pandas)rrr's r read_pandaszFeatherDataset.read_pandasMs* w/99#:% %r)Tr)NT)__name__ __module__ __qualname____doc__rr!rr)rrr r s /+2G %rr c|jdk(ry|jtjtjfvrt d|dt d|d|jd)NrzColumn 'zg' exceeds 2GB maximum capacity of a Feather binary column. This restriction may be lifted in the futurez ' of type zU was chunked on conversion to Arrow and cannot be currently written to Feather format) num_chunkstypeextbinarystringr$)namecols rcheck_chunked_overflowr7asw ~~ xxCJJL#**,//8D6*001 1 tfJsxxj1@ @  r>lz4zstd uncompressedc|dk(rtjdtdtjrDtj r4t |tjjr|j}tj|rp|dk(rd}n|dk(rd}n tdtj||}|dk(r;t|jj D]\}} ||} t#| | n|}|dk(rYt%|j&t%t)|j&kDr td | td |Dtd |t+j,d rd }n||t.vrtd|dt. t1j2||||||y#t4$rBt |t6r0 t9j:|#t8j<$rYwxYwwxYw)a9 Write a pandas.DataFrame to Feather format. Parameters ---------- df : pandas.DataFrame or pyarrow.Table Data to write out as Feather format. dest : str Local destination path. compression : string, default None Can be one of {"zstd", "lz4", "uncompressed"}. The default of None uses LZ4 for V2 files if it is available, otherwise uncompressed. compression_level : int, default None Use a compression level particular to the chosen compressor. If None use the default compression level chunksize : int, default None For V2 files, the internal maximum size of Arrow RecordBatch chunks when writing the Arrow IPC file format. None means use the default, which is currently 64K version : int, default 2 Feather file version. Version 2 is the current. Version 1 is the more limited legacy format. .. deprecated:: 25.0.0 Writing Feather V1 files is deprecated. Use the default ``version=2`` to write Arrow IPC files instead. rzFeather V1 files are deprecated as of 25.0.0 and support will be removed in a future version. Use the default version=2 to write Arrow IPC files instead. stacklevelFNz%Version value should either be 1 or 2)preserve_indexz'cannot serialize duplicate column namesz2Feather V1 files do not support compression optionz0Feather V1 files do not support chunksize option lz4_framer8z compression="z " not supported, must be one of ) compressioncompression_level chunksizeversion)warningswarnDeprecationWarningr have_pandas has_sparse isinstancepdSparseDataFrameto_dense is_data_framer$r from_pandas enumeraternamesr7len column_namessetr is_available_FEATHER_SUPPORTED_CODECSr write_feather Exceptionstrosremoveerror) dfdestrArBrCrDr?r ir5r6s rrWrWts:!|  -     " "2{~~==>B  $ a<"N \!NDE E!!"^D a<$U\\%7%78 24Ah&tS1 2!| u!! "SU-?-?)@%A AFG G  "&' '  &' '  5#5#5k#BK%!::}[M:''@&ACD D ud 1B)2G E  dC   $ 88    s0F11G< G! G<!G74G<6G77G<c Dt||||jdd|i|S)a Read a pandas.DataFrame from Feather format. To read as pyarrow.Table use feather.read_table. Parameters ---------- source : str file path, or file-like object You can use MemoryMappedFile as source, for explicitly use memory map. columns : sequence, optional Only read a specific set of columns. If not provided, all columns are read. use_threads : bool, default True Whether to parallelize reading using multiple threads. If false the restriction is used in the conversion to Pandas as well as in the reading from Feather format. memory_map : boolean, default False Use memory mapping when opening file on disk, when source is a str. **kwargs Additional keyword arguments passed on to `pyarrow.Table.to_pandas`. Returns ------- df : pandas.DataFrame The contents of the Feather file as a pandas.DataFrame r memory_mapr'r'r.)rr()sourcerr'rbkwargss r read_featherres<6 + J !!* N7B NFL NOrctj|||}|jdkrtjdt d||j St|ts-tdjt|j|Dcgc] }t|}}ttd|r|j|}nRttd|r|j!|}n*|Dcgc]}|j} }td|d | |jdkr|St#t%||k(r|S|j'|Scc}wcc}w) z Internal implementation for reading a Feather file as a pyarrow.Table. Emits a deprecation warning if the file is a legacy Feather V1 file. )use_memory_mapr'zFeather V1 files are deprecated as of 25.0.0 and support will be removed in a future version. Consider rewriting this file in the Arrow IPC file format (Feather V2).r=z&Columns must be a sequence but, got {}c|tk(Sr)intts rz&_read_table_internal.. s crc|tk(Sr)rYrks rrmz&_read_table_internal..s 18rz.Columns must be indices or names. Got columns z of types )r FeatherReaderrDrErFrGreadrJr TypeErrorformatr1r*allmap read_indices read_namessortedrTselect) rcrrbr'readercolumn column_typesr rlcolumn_type_namess rrrs`  # #z{DF~~  9   {{} gx (@W 6 679 9077VDL7L7 3!< 01##G, S#\ 2 3!!'*1=>AQZZ>>''.iz:K9LNO O~~ G  ( ||G$$%8 ?s E ?E%c t||||S)a Read a pyarrow.Table from Feather format Parameters ---------- source : str file path, or file-like object You can use MemoryMappedFile as source, for explicitly use memory map. columns : sequence, optional Only read a specific set of columns. If not provided, all columns are read. memory_map : boolean, default False Use memory mapping when opening file on disk, when source is a str use_threads : bool, default True Whether to parallelize reading using multiple threads. Returns ------- table : pyarrow.Table The contents of the Feather file as a pyarrow.Table ra)r)rcrrbr's rr!r!!s* +5,7 99r)NNNr<)NTF)NFT)collections.abcrrZrEpyarrow.pandas_compatr pyarrow.librrrrlibr2pyarrowr pyarrow._featherr r r7rVrWrerr!r.rrrsr&% -00)?%?%D  <AE*+\~48!O@;@%)+%\9r