o h?@sddlZddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z ddl ZddlmZzddlZWneyOe sMddlZYnwddlmZddlmZddl mZmZmZdd lmZdd lmZmZm Z e!d Z"Gd d d e#Z$ddZ%ddZ&ddZ'Gdddej(j)Z*Gdddej(j+Z,ddZ-Gdddej(j.Z/GdddeZ0GdddeZ1dS) N) lru_cache)chain) TYPE_CHECKINGLiteral)AbstractBufferedFile)AsyncFileSystem)DEFAULT_CALLBACK) filesystemopensplit_protocol)AsyncFileSystemWrapper) isfilelikemerge_offset_ranges other_pathszfsspec.referencecs$eZdZfddZddZZS)ReferenceNotReachablecstj|||_||_dSN)super__init__ referencetarget)selfrrargs __class__t/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/fsspec/implementations/reference.pyrs  zReferenceNotReachable.__init__cCsd|jd|jS)Nz Reference "z" failed to fetch target )rrrrrr__str__$zReferenceNotReachable.__str__)__name__ __module__ __qualname__rr __classcell__rrrrrs rcCstt|Sr)nextitervalues)drrr_first(sr'cCsN||}t|ttfr#t|dtr%|drt|ddS|dSdSdSNr)get isinstancelisttuplestrr )path referencesrefrrr_prot_in_references,s  r1cCsHt|tr t|||giSi}|D]}t||}||g|q|Sr)r*r-r1 setdefaultappend)pathsr/outr.protocolrrr_protocol_groups2s  r7c@eZdZddZdS)RefsValuesViewccs|jjD] }t|Vq|jjEdH|jD] }|j|}t |dkr7|j|dVq |j |EdHq dS)Nr/0) _mapping zmetadatar%jsondumpsencode_itemslistdir_get_chunk_sizeslen_generate_all_records)rvalfield chunk_sizesrrr__iter__=s  zRefsValuesView.__iter__Nrr r!rHrrrrr9< r9c@r8) RefsItemsViewcCst|j|jSr)zipr;keysr%rrrrrHJszRefsItemsView.__iter__NrIrrrrrKIrJrKcCsHd}d}t|ddd|dddD]\}}|||7}||9}q|S)Nr)rL)idxsizesrEmultisrrrravel_multi_indexNs &  rUc@seZdZdZeddZeddZ     d@d ed fd d ZddZ ddZ e dAddZ e ddZdBddZddZe dddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Zd,d-Zd.d/Ze d0d1Zd2d3ZdCd4d5ZdCd6d7Zd8d9Zd:d;Zdd?Z!dS)DLazyReferenceMapperaoThis interface can be used to read/write references from Parquet stores. It is not intended for other types of references. It can be used with Kerchunk's MultiZarrToZarr method to combine references into a parquet store. Examples of this use-case can be found here: https://fsspec.github.io/kerchunk/advanced.html?highlight=parquet#parquet-storagecC ddl}|Sr()numpy)rnprrrrY`zLazyReferenceMapper.npcCrWr()pandas)rpdrrrr\frZzLazyReferenceMapper.pdN fastparquetengine)r_pyarrowcCs||_i|_||_||_||_|jd|_|durtdn||_|j |p(|j|_ ddl m }|jdkr@|ddurBt ddSdS)a This instance will be writable, storing changes in memory until full partitions are accumulated or .flush() is called. To create an empty lazy store, use .create() Parameters ---------- root : str Root of parquet store fs : fsspec.AbstractFileSystem fsspec filesystem object, default is local filesystem. cache_size : int, default=128 Maximum size of LRU cache, where cache_size*record_size denotes the total number of references that can be loaded in memory at once. categorical_threshold : int Encode urls as pandas.Categorical to reduce memory footprint if the ratio of the number of unique urls to total number of refs for each variable is greater than or equal to this number. (default 10) engine: Literal["fastparquet","pyarrow"] Engine choice for reading parquet files. (default is "fastparquet") z/{field}/refs.{record}.parqNfiler) find_specraz)engine choice `pyarrow` is not installed.)rootrG cat_threshr` cache_sizeurlfsspecr fsunstrip_protocolout_rootimportlib.utilrc ImportError)rrdrirkrfcategorical_thresholdr`rcrrrrls!  zLazyReferenceMapper.__init__cCs"|dvr ||j|St|)N)r@ record_sizer<)setup__dict__AttributeErrorritemrrr __getattr__s zLazyReferenceMapper.__getattr__csji_jdjdgjd<tjd}|d_|d_t j dfdd}|_ dS)N/ .zmetadatarometadata)maxsizecshjj||d}tj|}zjj|jdfddj D}W|St y3d}Y|Sw)zcached parquet file loader)rFrecord)r`csi|] }||qSr)to_numpy.0cdfrr z@LazyReferenceMapper.setup..open_refs..N) rgformatioBytesIOricat_filer\ read_parquetr`columnsOSError)rFrzr.datarefsrrr open_refss z,LazyReferenceMapper.setup..open_refs) r@rirjoinrdr=loadsror<rrfr)rmetrrrrrps     zLazyReferenceMapper.setup'cKsi|d}|durtjj|fi|pi\}}||r#|j|dd|j|dd|d|dgt | t ||fi|S)aJMake empty parquet reference set First deletes the contents of the given directory, if it exists. Parameters ---------- root: str Directory to contain the output; will be created storage_options: dict | None For making the filesystem to use for writing is fs is None fs: FileSystem | None Filesystem for writing record_size: int Number of references per parquet file kwargs: passed to __init__ Returns ------- LazyReferenceMapper instance rxroNT recursiveexist_okrvrw) rhcore url_to_fsexistsrmmakedirspiperr=r>r?rV)rdstorage_optionsrirokwargsrrrrcreates   zLazyReferenceMapper.createcCsdd|jD}t|S)zList top-level directoriescss*|]}|ds|dddVqdS).zrvrNrN) startswithrsplitr}prrr s(z.LazyReferenceMapper.listdir..)r<set)rdirsrrrrAszLazyReferenceMapper.listdirTc sDdr dnd}fdd|D}|r]fddtdgdd jDd d jDD}|d urC||t|Sd d|D}fd d|D}t||dddS}tfddjDfddjD}fdd|D}|} |d urt |t | S |} ddt | | D} || S)zShortcut file listingsrvrcs(g|]}|rd|vr|qSrvrlstrip)r}r&pathdashrr sz*LazyReferenceMapper.ls..cs(h|]}|rd|vr|qSrr)r}frrr sz)LazyReferenceMapper.ls..rwcs|]}|VqdSrrr}namerrrrz)LazyReferenceMapper.ls..csrrrrrrrrrFcSsg|]}|dddqS) directoryrrtypesizerrrrrrsc <g|]}|dt|jvrtj|nj|dqSrbrrCr<r=r>r@rrrrr  cSs|dS)Nrr)rTrrrsz(LazyReferenceMapper.ls..keyc g|] }|dr|qSrrrr.rrr crrrrrrrrrc rrrrrrrr rcSs(g|]\}}|dr|d|ddqS)rrbrOrr)r}rrecrrrrs ) rstriprArr<r@updatesortedr_keys_in_fieldr+rDrL) rr.detaildirnamesrothersdirinfofileinforFrMrecsrecinfor)r.rrrlssN           zLazyReferenceMapper.lsc sP||jvr |j|S||jvrt|j|Sd|vs"||r&t||dd\}}||\}}|j ||fi d}|durIt|rM|S|dkrSdSz| ||Wnt t t fyo}zt||d}~wwgd}fdd |D} | d } | dur| S| ddurtd | dd ddgkr| ddS| dd S) zfGet the reference for one key Returns bytes, one-element list or three-element list. rvrNFNrr.offsetrrawcs$g|]}|vr|ndqSrrr|rrirrr>s$z5LazyReferenceMapper._load_one_key..rOz1This reference does not exist or has been deleted)r@r<r=r>r?_is_metaKeyErrorr_key_to_recordr)r ValueError TypeErrorFileNotFoundError) rrrF_rz chunk_sizemaybeexcr selectionrrrr _load_one_key"s>       z!LazyReferenceMapper._load_one_keyic Csj|dd\}}||}t|dkrdSdd|dD}t||}||j}||j}||t|fS)z3Details needed to construct a reference for one keyrvrNr)rrrcSg|]}t|qSr)intr|rrrrQz6LazyReferenceMapper._key_to_record...)rrBrCsplitrUro) rrrFchunkrG chunk_idx chunk_numberrzrrrrrJs     z"LazyReferenceMapper._key_to_recordcCsP||jvr#|j|d}ddt|d|dD}|pdg|j|<|j|S)z6The number of chunks along each axis for a given fieldz/.zarraycSsg|] \}}t||qSr)mathceil)r}rTr~rrrr[sz8LazyReferenceMapper._get_chunk_sizes..shapechunksrN)rGr<rL)rrFzarray size_ratiorrrrBWs  z$LazyReferenceMapper._get_chunk_sizescCsX|||}tt|}t|dkrdd|DSt|dkr%|dSdd|DS)z8The references for a given parquet file of a given fieldrcs|]}t|VqdSrr+r}trrrrgz7LazyReferenceMapper._generate_record..rNrcss0|]}|dst|ddn|dVqdS)rNrrrrrrms.)rr$rLr%rC)rrFrzritrrr_generate_recordas   z$LazyReferenceMapper._generate_recordccsRd}||D]}||9}qt||j}t|D] }|||EdHqdS)zJLoad all the references within a field by iterating over the parquet filesrNN)rBrrroranger)rrFnrecchrzrrrrDos  z)LazyReferenceMapper._generate_all_recordscCt|Sr)r9rrrrr%xzLazyReferenceMapper.valuescCrr)rKrrrritems{rzLazyReferenceMapper.itemscCrr)idrrrr__hash__~rzLazyReferenceMapper.__hash__cCs ||Sr)r)rrrrr __getitem__ zLazyReferenceMapper.__getitem__c Csd|vr7||s7|dd\}}||\}}}|j||fi}|||<t||jkr5|||dSdSt|drC| } n t |t rM| } n|} | |j|<t | } i|j|i| |j|<dS)NrvrNto_bytes)rrrr@r2rCrowritehasattrrdecoder*bytesr=rr<r)) rrvaluerFrrzrSrsubdictrE new_valuerrr __setitem__s       zLazyReferenceMapper.__setitem__cCs|dpd|vS)Nrz/.zrrrrrrszLazyReferenceMapper._is_metacCs||jvr |j|=dS||jvr|j|=dSd|vrM||sM|dd\}}||\}}}|j||fi}d||<t||jkrK|||dSdSd|j|<dS)NrvrN) r@r<rrrr2rCror)rrrFrrzrSrrrr __delitem__s    zLazyReferenceMapper.__delitem__cCsddl}ddl}ddl}|j||f}d} t||jkr.z|||} Wn ty-Ynw| rA| d} | d} | d} | d} n$|j|j|j dd} |j |jd d} |j |jd d} |j|j|j dd} | D]U\}}t |t rt| jd kr|d| jjvr| |d} |d| |<t|d kr|d | |<|d | |<qi|durd| |<d| |<d| |<d| |<qi|j|| |<qi|j| | | | d dd}|j|jpd |jkr|dd |d<ddd}ddg}|p|jd|d|d}|jj|p|jd|dd|jdkrddi}n|jdkr#d||d}nt|jd|j|f|j|p:t |jddddd||!|j"||fdS)NrFr.rrrO)dtypeint64categoryrNr)copyrutf8)rr.rvz/refs.z.parqTrrawrite_statisticsr_)statsobject_encoding has_nullsz not supportedrzstd)r`r compressionindex)# kerchunk.dfrXr[r@rCrorrfullnanzerosrr*r+r-r  categoriesadd_categoriesr _proc_raw DataFramer.countnuniquereastyperkrimkdirsr`NotImplementedError to_parquetgetattrclearpop)rrFrzbase_urlrkerchunkrYr\ partitionoriginalr4offsetsrQrawsjrrrrfndf_backend_kwargsrrrrs          "     zLazyReferenceMapper.writecCst|jD]}t|tr|\}}|j||||dqt|jD]}|dkr5d|vr5t|j||j|<q|j|j d}|j t | |jd<|j d|pU|jdg|jd|jdS)zOutput any modified or deleted keys Parameters ---------- base_url: str Location of the output )r+rrwrrrvN)r+r@r*r,rr=rr*r<ror)r>r?rirrrkr cache_clear)rr+rthingrFrzkrrrrflushs,  zLazyReferenceMapper.flushcCsdd}|D]}|dr|d7}q|t||7}q|t|j7}|tdd|jD7}|S)NrrrNcss|] }t|tsdVqdS)rNN)r*r,r}rrrrr/z.LazyReferenceMapper.__len__..) rArrprodrBrCr<sumr@)rr"rFrrr__len__%s   zLazyReferenceMapper.__len__ccsbt|j}||j|D] }t|tr|Vq|D]}||D] }||vr-|Vq$qdSr)rr<rr@r*r-rAr)rmetasbitrFr6rrrrH2s    zLazyReferenceMapper.__iter__cCs&z||WdStyYdSw)NTF)rrrsrrr __contains__?s   z LazyReferenceMapper.__contains__ccsf||}t|dkr|dVdStjdd|D}|D]}|dddd |DVqdS) z|List key names in given field Produces strings like "field/x.y" appropriate from the chunking of the array rr:Ncsrr)r)r}rSrrrrOrz5LazyReferenceMapper._keys_in_field..rvrcSrr)r-r|rrrrQrz6LazyReferenceMapper._keys_in_field..)rBrC itertoolsproductr)rrFrGindsindrrrrFs    z"LazyReferenceMapper._keys_in_field)NNr]r^r_)NNr)rTNN)"rr r!__doc__propertyrYr\rrrurp staticmethodrrrArrrrBrrDr%rrrr rr rr7r<rHr?rrrrrrVWsR   0   B(       U#  rVcs<eZdZdZdZdZ            dDfd d ZdEd d ZdEd dZdEddZ ddZ ddZ e fddZ dFddZdGddZdHddZddZdHd d!Zd"d#Zd$d%Zd&d'ZdId)d*ZdJd+d,Zd-d.Zd/d0Zd1d2ZdJd3d4ZdKfd5d6 Zd7d8Zd9d:Zd;d<ZdLd>d?Z dLd@dAZ!dBdCZ"Z#S)MReferenceFileSystemaView byte ranges of some other file as a file system Initial version: single file system target, which must support async, and must allow start and end args in _cat_file. Later versions may allow multiple arbitrary URLs for the targets. This FileSystem is read-only. It is designed to be used with async targets (for now). We do not get original file details from the target FS. Configuration is by passing a dict of references at init, or a URL to a JSON file containing the same; this dict can also contain concrete data for some set of paths. Reference dict format: {path0: bytes_data, path1: (target_url, offset, size)} https://github.com/fsspec/kerchunk/blob/main/README.md rFNT@Br]c stjd i|||_| |_| |_i|_i|_i|_| |_| |_ t |t rt d i|p.|p.id|i}t jj|fi|\}}||rrt j|dfi|}td|t|}Wdn1sfwY||| ntd|t||| d|_n||| t |t rdd|D|_d|jvrtd |jd<dS|durt |jtr|jd n|j}||j|<|dur|jD]'}t|r|}t j|\}}|r||jvrt|fi|pi}||j|<q|dur-|jD]6}t|r|}t |t r,|d r,t j|d \}}||jvr,t|fi|p"i}||j|<nq|rF||jvrFt|fi|p>i}||j|<|pLtd |jd<|jD] \}}|j!sit"||j#d |j|<qU|j#|j#Artt$d qUdS)a Parameters ---------- fo : dict or str The set of references to use for this instance, with a structure as above. If str referencing a JSON file, will use fsspec.open, in conjunction with target_options and target_protocol to open and parse JSON at this location. If a directory, then assume references are a set of parquet files to be loaded lazily. target : str For any references having target_url as None, this is the default file target to use ref_storage_args : dict If references is a str, use these kwargs for loading the JSON file. Deprecated: use target_options instead. target_protocol : str Used for loading the reference file, if it is a path. If None, protocol will be derived from the given path target_options : dict Extra FS options for loading the reference file ``fo``, if given as a path remote_protocol : str The protocol of the filesystem on which the references will be evaluated (unless fs is provided). If not given, will be derived from the first URL that has a protocol in the templates or in the references, in that order. remote_options : dict kwargs to go with remote_protocol fs : AbstractFileSystem | dict(str, (AbstractFileSystem | dict)) Directly provide a file system(s): - a single filesystem instance - a dict of protocol:filesystem, where each value is either a filesystem instance, or a dict of kwargs that can be used to create in instance for the given protocol If this is given, remote_options and remote_protocol are ignored. template_overrides : dict Swap out any templates in the references file with these - useful for testing. simple_templates: bool Whether templates can be processed with simple replace (True) or if jinja is needed (False, much slower). All reference sets produced by ``kerchunk`` are simple in this sense, but the spec allows for complex. max_gap, max_block: int For merging multiple concurrent requests to the same remote file. Neighboring byte ranges will only be merged when their inter-range gap is <= ``max_gap``. Default is 64KB. Set to 0 to only merge when it requires no extra bytes. Pass a negative number to disable merging, appropriate for local target files. Neighboring byte ranges will only be merged when the size of the aggregated range is <= ``max_block``. Default is 256MB. cache_size : int Maximum size of LRU cache, where cache_size*record_size denotes the total number of references that can be loaded in memory at once. Only used for lazily loaded references. kwargs : passed to parent class r6rbzRead reference from URL %sNz$Open lazy reference dict from URL %s)rirfcSs>i|]\}}|t|trtj|dddfi|n|qS):rNr)r*dictrhr r)r}r6optsrrrrs z0ReferenceFileSystem.__init__..rbr) asynchronouszDReference-FS's target filesystem must have same valueof asynchronousr)%rrrtemplate_overridessimple_templates templatesfss _dircachemax_gap max_blockr*r-rMrhrrisfiler loggerinfor=load_process_referencesrVr/rr r6r,r%callabler r+ async_implr rOr)rforref_storage_argstarget_protocoltarget_optionsremote_protocolremote_optionsrirPrQrUrVrfrdicref_fsfo2rtextr0r6rr6rrrrfsI               zReferenceFileSystem.__init__c Cs||}td|z|j|}Wnty%}zt||d}~wwt|tr/|}t |dr8| }t|t rYtd|d| drTt |dd}|ddfSt|dkrttd|d||d }||}}nB|\}} } td|d |d | d | | | } |dur|d kr| |}n| |}n| }|dur|d kr| |}n| |}n| }|dur|j}|||fS) Nzcat: rz Reference: z , type bytesbase64:rNz, whole file => rz => z , offset z, size )_strip_protocolrXdebugr/rrr*r-r?rrrrbase64 b64decoderCr) rr.startendpartrrgstart1end1start0rend0rrr _cat_commonsJ          "     zReferenceFileSystem._cat_commonc s~|j|||d\}}}t|tr|||St|\}} z|j|j|||dIdHWSty>} zt||| d} ~ wwN)rnro)rur*rr rS _cat_file Exceptionr rr.rnror part_or_urlrsrtr6rerrrrw3s     zReferenceFileSystem._cat_filec Ksv|j|||d\}}}t|tr|||St|\}} z |j|j|||dWSty:} zt||| d} ~ wwrv)rur*rr rSrrxrryrrrr?s    zReferenceFileSystem.cat_filecKs||j|<dS)z2Temporarily add binary data or reference as a fileNr/)rr.rrrrr pipe_fileIszReferenceFileSystem.pipe_filecsd||r tj|ddS||IdH}t|d}||WddS1s+wYdSNTrwb)isdirosrrwr r)rrpathlpathrrrrrr _get_fileMs   "zReferenceFileSystem._get_filecKs||r tj|ddS|j|fi|}|t|t|r&||nt|d }||Wdn1s;wY| t|dSr~) rrrrset_sizerCr rr absolute_update)rrrcallbackrrrrrrget_fileTs    zReferenceFileSystem.get_filec  s|rdj||d}tjddd}t||}|r)fdd|D}n|}t||D]\}} ||vrC|| ||q3dS)NrrrbT) auto_mkdircsg|] }|s|qSr)r)r}rrrrrhrz+ReferenceFileSystem.get..)r expand_pathrhr rcatrLr}) rrrrrritargetsrremotelocalrrrr)`s   zReferenceFileSystem.getraisec$ Kst|tr |r tt|tr|stdd|Drtt||j}i}|D]Y\}}|j|} ggggf\} } } } |D]I}z| |\}}}t|t tfsQWq=Wnt yq}z|dkr_|dkrg|||<WYd}~q=d}~ww| || || || |q=g}g}g}g}t }t| | | | D]-\}}}}t|t r|||<q|dur||| || || || |qt| | | | D]"\}}}}|dur||vr| || || || |qtt|t|t|d|j|jd\}}}| |||}t| | | | D]l\}}}}||vr!qt||||D]V\}}}}||krS|dus=|durSt|trI|||<q(|||||<q(||kr}||kr}||kr}t|trn|||<q(|||||pxd||<q(qq'|D]0\} }!t|!tr| |jvr|| }"t| |j| }#|"|#_|dkr|#|dkr|#|| <qt|dkrt|trd|vrt|S|S) Ncss|]}d|vVqdS)*Nrrrrrrrrz*ReferenceFileSystem.cat..romitT)sortrUrVrNr)r*r-r&r+anyr7r/rrSrurrr3rrLaddrrUrV cat_rangesrxrr __cause__rCr')$rr.ron_errorr proto_dictr5protor4riurlsstartsends valid_pathsrurTr{errurls2starts2ends2paths2 whole_files new_paths new_startsnew_ends bytes_outrYnsnebr6vexnew_exrrrros                          $zReferenceFileSystem.catcCsJ|dd}|dur||dS|dkr|j||ddStd|)NversionrN)rPz Unknown reference spec version: )r)_process_references0_process_references1r)rr/rPversrrrr[s z'ReferenceFileSystem._process_referencescCs&t|trdd|D}||_dS)z&Make reference dict for Spec Version 0cSs*i|]\}}|t|trt|n|qSr)r*rMr=r>)r}rrErrrrsz.N)r*rMrr/)rr/rrrrs  z(ReferenceFileSystem._process_references0csLjrjr ddli_|ditdfdd}|diD]m\}}t|t rI| drCt |ddj|<|j|<q)t|t rWt|j|<q)jr|d}d |vr{jrw|d d d d jdij}n||}t|d kr|gn||d |dgj|<q)|j|<q)j|dgdS)NrrRics|jdijSNr)TemplaterenderrR)rjinja2rrr _render_jinjasz?ReferenceFileSystem._process_references1.._render_jinjarzbase64:ri{{{z}}}rNrgenr)rQrRrr/_process_templatesr)rrr*r-rrlrmrMr=r>replacerrCr _process_gen)rr/rPrr6rrrrrrs8      , z(ReferenceFileSystem._process_references1csdi|_|jdur||j|D]\}}d|vr*ddl|ffdd |j|<q||j|<qdS)Nrrcs|jdi|Sr)rr)temprrrrrsz8ReferenceFileSystem._process_templates..)rRrPrrr)rtmpr6rrrrrs   z&ReferenceFileSystem._process_templatesc s i}|D]}dd|dDfddtjD}|D]k}ddl}||djd i||j}||djd i||j}d |vr{d |vr{t||d jd i||j} t||d jd i||j} || | g||<q!d |vd |vArt d |g||<q!q|S) Nc Ss@i|]\}}|t|tr|nt|dd|d|ddqS)rnrstopsteprN)r*r+rr))r}r6rrrrrsz4ReferenceFileSystem._process_gen.. dimensionsc3s"|] }tt|VqdSr)rMrLrM)r}r% dimensionrrr!s  z3ReferenceFileSystem._process_gen..rrrgrlengthz^Both 'offset' and 'length' are required for a reference generator entry if either is provided.r) rr@rAr%rrrrRrr) rgensr5rproductsprrrrgrrrrrrs6      z ReferenceFileSystem._process_genc Cs4dgi|_|j}|D]\}}t|ttfst|dr!t|}nt|dkr*d}n|\}}}d|vr;|dddnd}|}|g}|rZ||jvrZ| |}| ||rZ||jvsI| t ||ddD]#\} } | |jvsrJ| |jvsyJ|j|  | dddg|j| <qg|j| |d|dq dS) NrrrNrvrrrrb) dircacher/rr*rr-rrCr_parentr3reverserL) rrr.rprrparpar0subdirsparentchildrrr_dircache_from_items;s2           z(ReferenceFileSystem._dircache_from_itemsrKc Ks||\}}}t|trt|||St|\} } |dur4|dur4|j| j||f||d|St|||f||d|S)N) block_size cache_options) rur*rrrr rS_open ReferenceFile) rr.moderrrrzrsrtr6rrrrrYs0   zReferenceFileSystem._opencKstd|||}t|jtr+z|j||WSty"Ynwtd|d|j s2| | |}|dur?t||rC|Sdd|DS)Nzlist %s'z' is not a known keycSsg|]}|dqSrrr}orrrrrz*ReferenceFileSystem.ls..) rXrkrjr*r/rVrrrrr_ls_from_cache)rr.rrr5rrrrss"     zReferenceFileSystem.lscKs||p ||Sr)rrWrr.rrrrrrzReferenceFileSystem.existscsB|jr|jvSt|jtr|jvStfdd|jDS)Nc3s |] }|dVqdS)rvNrr8rrrrsz,ReferenceFileSystem.isdir..)rr*r/rVrArrr.rrrrs   zReferenceFileSystem.isdircCs ||jvSrr|rrrrrWrzReferenceFileSystem.isfilecs|j||fi|Sr)r)rr.rrrrr_lsszReferenceFileSystem._lsc sz|rtjf|||d|Sr$tfddjD}ntj}|r;js2fdd|DS|S)N)maxdepthwithdirsrc3s|] }|r|VqdSrrr}r6rrrrr9z+ReferenceFileSystem.find..csi|] }||dqS)r)rrrrrrz,ReferenceFileSystem.find..)rfindrjrr/rr)rr.rrrrrr)r.rrrs"  zReferenceFileSystem.findc s|j}|dur/t|ttfrdt|dSt|dkr'd|ddSdddg}n|d}fdd|D}|sFdd dS|d d durjt|jd \}}|j| |jd |d d <|d S) NrbrrNrTcsg|] }|dkr|qSrrrrrrrrz,ReferenceFileSystem.info..rrr) r/r)r*r-rrCrr rSr)rr.rr5out0protrrrrrYs    "zReferenceFileSystem.infocs ||Sr)rYrrrr_infos zReferenceFileSystem._infocs|j|d|jdSr)r/r*rr)rrrr_rm_files zReferenceFileSystem._rm_file overwritecs0|dkr ||r t||j|<|jdS)Nr)rFileExistsErrorr/rr))rr.rrrrrr _pipe_files  zReferenceFileSystem._pipe_filecs^|dkr ||r tt|d}||j|<Wdn1s#wY|jdS)NrrK)rrr readr/rr))rrrrrrrrr _put_files zReferenceFileSystem._put_filec Ksi}|jD]+\}}t|tr.z |d||<Wqty-dt|||<Yqw|||<qtj |dfi|}| t d|d WddS1sVwYdS)z+Write modified references into new locationasciirhrrN)rrN)r/rr*rrUnicodeDecodeErrorrl b64encoderhr rr=r>r?)rrgrr5r6rrrrr save_jsons   "zReferenceFileSystem.save_json) NNNNNNNNTrIrJr]rD)F)Frr)rKNN)T)NFF)r)$rr r!rEr6cachablerrurwrr}rrrr)rr[rrrrrrrrrrWrrrYrrrrrr"rrrrrHTsZ " ,     b !%     rHcsJeZdZ      dfdd Zedd Zfd d Zd d ZZS)rrKdefaultT readaheadNc sftj||f||||||d| |j|j\} |_|_t| \} } |jj| |_ | |_ d|_ dS)N)rr autocommitr cache_typer) rrrirur.rnror rSsrc_fssrc_path_f) rrir.rrrrrrrrzr6rrrrrs$    zReferenceFile.__init__cCsB|jdus |jjr|jj|jf|j|j|jdd|j|_|jS)Nnone)rrrr) rclosedrrrr blocksizerrrrrrrszReferenceFile.fcs|jdur |jtSr)rcloserrrrrrs   zReferenceFile.closecCs8||j}t||j|j}|j||j||Sr)rnminrorseekr)rrnrorrr _fetch_ranges  zReferenceFile._fetch_range)rKrTrNN) rr r!rrFrrrr"rrrrrs  r)2rl collectionsrr@loggingrr functoolsrrtypingrr fsspec.corerh fsspec.specrujsonr=rm fsspec.asynrfsspec.callbacksrr r r #fsspec.implementations.asyn_wrapperr fsspec.utilsr rr getLoggerrX RuntimeErrorrr'r1r7abc ValuesViewr9 ItemsViewrKrUMutableMappingrVrHrrrrrsX