o h:@s|ddlZddlZddlZddlZddlmZmZddlmZddl m Z Gddde Z GdddeZ Gd d d e ZdS) N) HTTPAdapterRetry)AbstractFileSystem)AbstractBufferedFilecs eZdZdZfddZZS)DatabricksExceptionz< Helper class for exceptions raised in this module. cst|||_||_dS)z Create a new DatabricksExceptionN)super__init__ error_codemessage)selfr r  __class__o/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/fsspec/implementations/dbfs.pyrs  zDatabricksException.__init__)__name__ __module__ __qualname____doc__r __classcell__rrr rr srcseZdZdZfddZd!ddZd!ddZd!d d Zd"d d Z d#ddZ d$ddZ ddZ d!ddZ ddZ ddZddZd%fdd ZZS)&DatabricksFileSystemz Get access to the Databricks filesystem implementation over HTTP. Can be used inside and outside of a databricks cluster. c sp||_||_t|_tddgdd|_|jdt|jd|jj dd|jit j d i|d S) a Create a new DatabricksFileSystem. Parameters ---------- instance: str The instance URL of the databricks cluster. For example for an Azure databricks cluster, this has the form adb-..azuredatabricks.net. token: str Your personal token. Find out more here: https://docs.databricks.com/dev-tools/api/latest/authentication.html g?)iiiiii)totalbackoff_factorstatus_forcelisthttps://) max_retries AuthorizationzBearer Nr) instancetokenrequestsSessionsessionrretriesmountrheadersupdaterr)r rrkwargsr rrrs zDatabricksFileSystem.__init__Tc Ks||}|sTsz+DatabricksFileSystem.ls..cSsg|]}|dqS)r4rr7rrrr:`s)_ls_from_cache _send_to_apirr FileNotFoundErrorr dircache)r r)detailr&outrer/rrrls;s(      zDatabricksFileSystem.lsc Cs|s-z|jddd|idtd|dty,}z |jdkr" WYd}~nd}~wwz |jd d d|idWntyQ}z |jd krLt|j|d}~ww|||dS) aE Create a given absolute path and all of its parents. Parameters ---------- path: str Absolute path to create exist_ok: bool If false, checks if the folder exists before creating it (and raises an Exception if this is the case) r'z get-statusr)r*zPath z already existsr.NpostmkdirsRESOURCE_ALREADY_EXISTS)r<FileExistsErrorrr r invalidate_cache_parent)r r)exist_okrBrrrmakedirsbs(     zDatabricksFileSystem.makedirscKs|st|j|fi|dS)a! Create a given absolute path and all of its parents. Parameters ---------- path: str Absolute path to create create_parents: bool Whether to create all parents or not. "False" is not implemented so far. N)NotImplementedErrorrE)r r)create_parentsr&rrrmkdirs zDatabricksFileSystem.mkdirFc Kszz |jdd||ddWn%ty2}z|jdkr"|j||d|jdkr-t|j|d}~ww|||dS)z Remove the file or folder at the given absolute path. Parameters ---------- path: str Absolute path what to remove recursive: bool Recursively delete all files in a folder. rDdelete)r) recursiver*PARTIAL_DELETEIO_ERRORN)r<rr rmOSErrorr rHrI)r r)rPr&rBrrrrSs      zDatabricksFileSystem.rmNc Ks|rt|rtz |jdd||ddWn#ty8}z|jdkr(t|j||jdkr3t|j|d}~ww||||||dS)a Move a source to a destination path. A note from the original [databricks API manual] (https://docs.databricks.com/dev-tools/api/latest/dbfs.html#move). When moving a large number of files the API call will time out after approximately 60s, potentially resulting in partially moved data. Therefore, for operations that move more than 10k files, we strongly discourage using the DBFS REST API. Parameters ---------- source_path: str From where to move (absolute path) destination_path: str To where to move (absolute path) recursive: bool Not implemented to far. maxdepth: Not implemented to far. rDmove) source_pathdestination_pathr*r.rFN) rLr<rr r=r rGrHrI)r rVrWrPmaxdepthr&rBrrrmvs(     zDatabricksFileSystem.mvrbdefaultcKst||f||d|S)z Overwrite the base class method to make sure to create a DBFile. All arguments are copied from the base method. Only the default blocksize is allowed. )mode block_size)DatabricksFile)r r)r\r]r&rrr_openszDatabricksFileSystem._openc Cs|dkr |jj}n|dkr|jj}ntd|tjd|jd|}|||d}z |W| St j y[}zz|j }Wn t yN|dwtdi||d}~ww) a Send the given json to the DBFS API using a get or post request (specified by the argument `method`). Parameters ---------- method: str Which http method to use for communication; "get" or "post". endpoint: str Where to send the request to (last part of the API URL) json: dict Dictionary of information to send rDr'zDo not understand method rz/api/2.0/dbfs/)r-Nr)r!rDr' ValueErrorurllibparseurljoinrraise_for_statusr HTTPErrorresponser- Exceptionr) r r+r,r- session_callurlrArBexception_jsonrrrr<s&      z!DatabricksFileSystem._send_to_apic CsRz|jdd||dd}|dWSty(}z |jdkr#t|j|d}~ww)a{ Internal function to create a handle, which can be used to write blocks of a file to DBFS. A handle has a unique identifier which needs to be passed whenever written during this transaction. The handle is active for 10 minutes - after that a new write transaction needs to be created. Make sure to close the handle after you are finished. Parameters ---------- path: str Absolute path for this file. overwrite: bool If a file already exist at this location, either overwrite it or raise an exception. rDcreate)r) overwriter*handlerFN)r<rr rGr )r r)rlrArBrrr_create_handles   z#DatabricksFileSystem._create_handlec CsLz |jddd|idWdSty%}z |jdkr t|j|d}~ww)z Close a handle, which was opened by :func:`_create_handle`. Parameters ---------- handle: str Which handle to close. rDclosermr*r.N)r<rr r=r )r rmrBrrr _close_handle0s   z"DatabricksFileSystem._close_handlec Csrt|}z|jdd||ddWdSty8}z|jdkr(t|j||jdkr3t|j|d}~ww)a Upload data to an already opened file handle (opened by :func:`_create_handle`). The maximal allowed data size is 1MB after conversion to base64. Remember to close the handle when you are finished. Parameters ---------- handle: str Which handle to upload data to. data: bytes Block of data to add to the handle. rDz add-blockrmdatar*r.MAX_BLOCK_SIZE_EXCEEDEDN) base64 b64encodedecoder<rr r=r r`)r rmrrrBrrr _add_dataAs     zDatabricksFileSystem._add_datac Cstz|jdd||||dd}t|dWSty9}z|jdkr)t|j||jdvr4t|j|d}~ww) a Download data in bytes from a given absolute path in a block from [start, start+length]. The maximum number of allowed bytes to read is 1MB. Parameters ---------- path: str Absolute path to download data from start: int Start position of the block end: int End position of the block r'read)r)offsetlengthr*rrr.)INVALID_PARAMETER_VALUEMAX_READ_SIZE_EXCEEDEDN)r<rt b64decoderr r=r r`)r r)startendrArBrrr _get_data_s    zDatabricksFileSystem._get_datacs2|dur |jn|j|dt|dSN)r>clearpoprrH)r r)r rrrH}s z%DatabricksFileSystem.invalidate_cache)TF)FN)rZr[r)rrrrrrCrKrNrSrYr_r<rnrprwrrHrrrr rrs    ' !  . )rcsTeZdZdZdZ     dfdd Zd d Zdd dZddZdddZ Z S)r^zH Helper class for files referenced in the DatabricksFileSystem. irZr[T readaheadNc  sX|dus|dkr |j}||jksJd|tj||f|||||p$id|dS)zr Create a new instance of the DatabricksFile. The blocksize needs to be the default one. Nr[z,Only the default block size is allowed, not )r\r] autocommit cache_type cache_options)DEFAULT_BLOCK_SIZErr) r fsr)r\r]rrrr&r rrrs"  zDatabricksFile.__init__cCs|j|j|_dS)z(Internal function to start a file uploadN)rrnr)rm)r rrr_initiate_uploadszDatabricksFile._initiate_uploadFcsj|jd|jfdd|tD}|D] }|jj|j|dq|r3|jj|jddSdS)z.rq)rmTN) bufferseekgetvalue_to_sized_blockslenrrwrmrp)r final data_chunks data_chunkrrr _upload_chunks    zDatabricksFile._upload_chunkcCs>d}||}|||D]\}}||jj|j||d7}q |S)z-Internal function to download a block of data)r)r~r)rrrr))r r~r return_bufferrz chunk_start chunk_endrrr _fetch_ranges zDatabricksFile._fetch_rangerccs@||}t|||jD]}|}t|||j}||fVq dS)zFHelper function to split a range from 0 to total_length into bloksizesN)range blocksizemin)r rzr~rr data_startdata_endrrrrs zDatabricksFile._to_sized_blocks)rZr[TrNr)r) rrrrrrrrrrrrrr rr^s"  r^)rtrarrequests.exceptionsrequests.adaptersrrfsspecr fsspec.specrrgrrr^rrrrs   n