o U h@s>dZddlZddlZddlZddlZddlZddlZddlmZ ddl Z ddl m Z ddl mZddlmZddlmZddlmZdd lmZdd lmZmZmZdd lmZmZmZmZm Z dd l!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.dd l/m0Z0ddl1m2Z2m3Z3m4Z4m5Z5ddl6m7Z7m8Z8ddl9m:Z:m;Z; GdddeZddZ?d5ddZ@GdddeAZBGdddZCGdd d ZDGd!d"d"e#ZEGd#d$d$e"ZFGd%d&d&eZGGd'd(d(e2ZHGd)d*d*e8ZIGd+d,d,e:ZJd6d.d/ZKd6d0d1ZLejMfd6d2d3ZNeOd4kreNdSdS)7z)Repo implementation atop OpenStack SWIFT.N) ConfigParserBytesIO)Optional) HTTPClient)GreenThreadsMissingObjectFinder) LRUSizeCache)INFODIRPACKDIRPackBasedObjectStore) S_ISGITLINKBlobCommitTagTree) PackPackData PackIndexerPackStreamCopier_compute_object_sizecompute_file_sha iter_sha1load_pack_index_fileread_pack_header unpack_objectwrite_pack_headerwrite_pack_index_v2write_pack_object) TCP_GIT_PORT)InfoRefsContainerread_info_refssplit_peeled_refswrite_info_refs) OBJECTDIRBaseRepo)Backend TCPGitServerc@seZdZddZdS)PackInfoMissingObjectFindercCs |jsdS|j\}}}||jvrnq|sg|j|}|dtjkr0||dddfgn&|dtjkrD|dd|dDn|dt jkrV||dddfg||j vrg||j |ddfg|j || d t |jd ||fS) NTrrFcSsg|]}t|qS)tuple).0ir*r*i/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/dulwich/contrib/swift.py msz4PackInfoMissingObjectFinder.next..zcounting objects:  )objects_to_sendpopsha_done object_store pack_info_getrtype_numadd_todorr_taggedaddprogresslen)selfshanameleafinfor*r*r.nextas(    z PackInfoMissingObjectFinder.nextN)__name__ __module__ __qualname__rBr*r*r*r.r(`s r(c Cst}|rz |||W|Sty||Y|Swd}|s;ztjd}Wnty:}ztd|d}~ww|}tj |sJtd|| ||S)zLoad configuration in global var CONF. Args: path: The path to the configuration file file: If provided read instead the file like object NDULWICH_SWIFT_CFGz(You need to specify a configuration filez"Unable to read configuration file ) r read_fileAttributeErrorreadfposenvironKeyError Exceptionpathisfileread)rNfileconfconfpathexcr*r*r. load_confws,    rUcCs8|| }t||WdS1swYdS)zRead a pack index file from Swift. Args: scon: a `SwiftConnector` instance filename: Path to the index file objectise Returns: a `PackIndexer` instance N) get_objectr)sconfilenamefr*r*r.swift_load_pack_indexs $rZcCst||}i}|D]I}|jtjkr |j|j|jf||j<q |jtjkr8dd| D}|j|f||j<q |jt jkrDd||j<q |jt jkrU|j|j df||j<q t t|S)NcSs,g|]\}}}t|s||t| fqSr*)r statS_ISDIR)r,nmsr*r*r.r/s z$pack_info_create..r0)r from_objects iterobjectsr7rparentstreeidritemsrrobjectzlibcompressjsondumps) pack_data pack_indexpackrAobjshasr*r*r.pack_info_creates       rpcCsF|s||}n|}|sdSztt|W|S|wN)rVriloadsrg decompressrPclose)rXrWrQrYr*r*r.load_pack_infos ruc@s eZdZdS)SwiftExceptionN)rCrDrEr*r*r*r.rvsrvc@s~eZdZdZdddZddZdd Zdeefd d Z dd d Z ddZ ddZ dddZ dddZdddZdddZdS)SwiftConnectorzDA Connector to swift that manage authentication and errors catching.returnNcCsn||_|jdd|_|jdvrtd|jdd|_|jdd|_|jdd|_|jddp4d |_|jdd p>d |_ |jdd pHd |_ |jdd pRd|_ |jddp\d|_ |jddpfd |_ |jddppd|_||_d}|jdkr|\|_|_n|\|_|_dt|ji}tjt|j|j ||j |j |d|_ttt|jj|j|_dS)zInitialize a SwiftConnector. Args: root: The swift container that will act as Git bare repository conf: A ConfigParser Object swiftauth_ver)12z.Wrong authentication version use either 1 or 2auth_urlusernamepassword concurrency http_timeouthttp_pool_length region_name RegionOne endpoint_type internalURL cache_length chunk_lengthi/i0r{ X-Auth-Token)r block_sizeconnection_timeoutnetwork_timeoutheadersN)rRgetrzNotImplementedErrorr}userrgetintrrrrrrrroot swift_auth_v1 storage_urltoken swift_auth_v2strrfrom_url httpclient posixpathjoinurlparserN base_path)r=rrRr token_headerr*r*r.__init__s>   zSwiftConnector.__init__cCs|jdd|_tj|j|j|jd}|j|jd}t|jj}|j d||d}|j dks3|j dkrFt d |jd |j d | d |d }|d}||fS)N;:rr)z X-Auth-Userz X-Auth-KeyGETr,zAUTH v1.0 request failed on  with error code  ()z X-Storage-Urlr) rreplacerrr}rrrrNrequest status_codervre)r=auth_httpclientrrNretrrr*r*r.rs$zSwiftConnector.swift_auth_v1c s<jd\__i}jjdjd|d<t|}ddi}tjjj j d}t jj }| ds?t |d}|jd |||d }|jd ksR|jd krjtd t||d|jd|dt|}|ddd}|dd} tdd| D} | d} tfdd| D} | j|fS)Nr)r~r)passwordCredentials tenantNameauthz Content-Typezapplication/jsonrtokensPOSTbodyrrrzAUTH v2.0 request failed on rrraccessrrdserviceCatalogcss |] }|ddkr|VqdS)typez object-storeNr*)r,o_storer*r*r. *sz/SwiftConnector.swift_auth_v2.. endpointsc3s"|] }|djkr|VqdS)regionN)r)r,endpr=r*r.r.s)rsplittenantrrirjrrr}rrrNendswithrrrrrvr get_base_urlrerrrPrBr) r= auth_dict auth_jsonrrrNr auth_ret_jsonrcatalogsr5rendpointr*rr.r sF   &  zSwiftConnector.swift_auth_v2cCsF|jd|j}|jdkrdS|jdks|jdkr!td|jdS)zYCheck that Swift container exist. Returns: True if exist or None it not HEADNrr$HEAD request failed with error code T)rrrrrvr=rr*r*r.test_root_exists3s  zSwiftConnector.test_root_existscCsD|s|jd|j}|jdks|jdkr td|jdSdS)zcCreate the Swift container. Raises: SwiftException: if unable to create PUTrr#PUT request failed with error code N)rrrrrrvrr*r*r. create_rootAs zSwiftConnector.create_rootcCs`d}|j|}|jd|}|jdkrdS|jdks|jdkr'td|j|}t|S)zRetrieve objects list in a container. Returns: A list of dict that describe objects or None if container does not exist z ?format=jsonrrNrr#GET request failed with error code )rrrrrvrPrirr)r=qsrNrcontentr*r*r.get_container_objectsNs    z$SwiftConnector.get_container_objectscCst|jd|}|jd|}|jdkrdS|jdks|jdkr'td|ji}|D] \}}|||<q-|S)zRetrieve object stat. Args: name: The object name Returns: A dict that describe the object or None if object does not exist /rrNrrr)rrrrrvrelower)r=r?rNr resp_headersheadervaluer*r*r.get_object_stat`s  zSwiftConnector.get_object_statcs|d|jd|dttifdd}z|}Wn ty2|}Ynw|jdks=|jdkrEtd|jd S) zPut an object. Args: name: The object name content: A file object Raises: SwiftException: if unable to create rrzContent-Lengthcsjjdd}|S)Nrr)rr)rdatarrNr=r*r._sendsz(SwiftConnector.put_object.._sendrrrN)seekrPrrr<rMrrv)r=r?rrrr*rr. put_objectus     zSwiftConnector.put_objectcCsi}|r d||d<|jd|}|jjd||d}|jdkr"dS|jdks,|jd kr4td |j|}|r<|St|S) a Retrieve an object. Args: name: The object name range: A string range like "0-10" to retrieve specified bytes in object content Returns: A file like instance or bytestring if range is specified zbytes=RangerrrrNrrr)rrrrrvrPr)r=r?rangerrNrrr*r*r.rVs   zSwiftConnector.get_objectcCsD|jd|}|jd|}|jdks|jdkr td|jdS)zDelete an object. Args: name: The object name Raises: SwiftException: if unable to delete rDELETErr&DELETE request failed with error code N)rrrrrv)r=r?rNrr*r*r. del_objects zSwiftConnector.del_objectcCsT|D] }||dq|jd|j}|jdks |jdkr(td|jdS)zDelete the root container by removing container content. Raises: SwiftException: if unable to delete r?rrrrN)rrrrrrrv)r=rnrr*r*r.del_roots  zSwiftConnector.del_rootrxNrq)rCrDrE__doc__rrrrboolrrrrrrVrrr*r*r*r.rws )'     rwc@s@eZdZdZdddZddddZd d Zdd d Zd dZdS)SwiftPackReaderacA SwiftPackReader that mimic read and sync method. The reader allows to read a specified amount of bytes from a given offset of a Swift object. A read offset is kept internally. The reader will read from Swift a specified amount of data to complete its internal buffer. chunk_length specify the amount of data to read from Swift. rxNcCs2||_||_||_d|_d|_d|_|jj|_dS)zInitialize a SwiftPackReader. Args: scon: a `SwiftConnector` instance filename: the pack filename pack_length: The size of the pack object rN)rWrX pack_lengthoffset base_offsetbuffr buff_length)r=rWrXrr*r*r.rszSwiftPackReader.__init__FcCsP|r|jd|_|j}t|j|j|j}|jj|j|d|d}||_dS)Nr-r)rrminrrWrVrXr)r=morerrrr*r*r._reads   zSwiftPackReader._readcCsr|j|}|j||jkr|j|jd}||_|S|t|jkr,|jdd||S|j|j|}||_|S)zRead a specified amount of Bytes form the pack object. Args: length: amount of bytes to read Returns: a bytestring NT)r)rrrrr<rrP)r=lengthendrr*r*r.rPs   zSwiftPackReader.readcCs||_|d|_dS)z[Seek to a specified offset. Args: offset: the offset to seek to rN)rrr)r=rr*r*r.rs zSwiftPackReader.seekcCs|jj|jddS)zSRead the checksum from the pack. Returns: the checksum bytestring z-20r)rWrVrXrr*r*r. read_checksum szSwiftPackReader.read_checksumr)F) rCrDrErrrrPrrr*r*r*r.rs   rc@s4eZdZdZd ddZddZdd Zd d d ZdS) SwiftPackDatazThe data contained in a packfile. We use the SwiftPackReader to read bytes from packs stored in Swift using the Range header feature of Swift. rxNcCsp||_||_d|_|j|j}t|d|_t|j|j|j}t|j\}|_ t d|jj t d|_ d|_dS)zInitialize a SwiftPackReader. Args: scon: a `SwiftConnector` instance filename: the pack filename zcontent-lengthi) compute_sizeN)rW _filename _header_sizerintrrrrP _num_objectsr rr _offset_cacherm)r=rWrXr pack_readerversionr*r*r.rs  zSwiftPackData.__init__cCsZ||jvr |j|S||jksJt|j|j|j}||t|j\}}|j | fSrq) rrrrWrrrrrP pack_type_num_obj)r=rrunpacked_r*r*r. get_object_at0s   zSwiftPackData.get_object_atcCst|j|j|j}|Srq)rrWrrr)r=rr*r*r.get_stored_checksum9sz!SwiftPackData.get_stored_checksumcCdSrqr*rr*r*r.rt=zSwiftPackData.closer)rCrDrErrrr rtr*r*r*r.rs   rcs.eZdZdZdfdd ZeddZZS) SwiftPackzA Git pack object. Same implementation as pack.Pack except that _idx_load and _data_load are bounded to Swift version of load_pack_index and PackData. rxNcsb|d_|d=tj|i|jd_d_fdd_fdd_fdd_dS)NrW.infoctjjSrq)ru_pack_info_pathrWr*rr*r.Oz$SwiftPack.__init__..crrq)rZrW _idx_pathr*rr*r.rPrcrrq)rrW _data_pathr*rr*r.rQr) rWsuperr _basenamer _pack_info_pack_info_load _idx_load _data_load)r=argskwargs __class__rr.rIs  zSwiftPack.__init__cCs|jdur ||_|jS)z The pack data object being used.N)rrrr*r*r. pack_infoSs  zSwiftPack.pack_infor)rCrDrErrpropertyr __classcell__r*r*rr.r As  r cseZdZdZdfdd ZddZdd Zd d Zefd d Z ddZ dddZ de fddZ dddZddZddZZS)SwiftObjectStorezA Swift Object Store. Allow to manage a bare Git repository from Openstack Swift. This object store only supports pack files and not loose objects. rxNcs2t||_|jj|_ttt|_d|_ dS)z^Open a Swift object store. Args: scon: A `SwiftConnector` instance N) rrrWrrrr$r pack_dir _alternates)r=rWrr*r.rbs   zSwiftObjectStore.__init__cCsL|j}dd|D}g}|D]}t||jd}||j|<||q|S)NcSs*g|]}|ddr|dddqS)r?.packr))rrr,or*r*r.r/ps  z7SwiftObjectStore._update_pack_cache..rW)rWrr  _pack_cacheappend)r=objects pack_filesrbasenamermr*r*r._update_pack_cachens   z#SwiftObjectStore._update_pack_cachecCsgS)z3Loose objects are not supported by this repository.r*rr*r*r._iter_loose_objects|sz$SwiftObjectStore._iter_loose_objectscCs&|jD] }||vr|j|SqdSrq)packsr)r=r>rmr*r*r.r6s zSwiftObjectStore.pack_info_getc sxfdd}t}t}g}|||r8|d}||vr$||n||vr6||||}|||s||fS)Nc sLjD] }||vr#z |j|d}W|Sty"gYSwqdS)Nr0)r/rrL)commitrmrbrr*r. _find_parentss  z:SwiftObjectStore._collect_ancestors.._find_parentsr)setextendr3r:) r=headscommonr1basescommitsqueueerbr*rr._collect_ancestorss       z#SwiftObjectStore._collect_ancestorscs(tfdd}ddd}||fS)zAdd a new pack to this object store. Returns: Fileobject to write to and a commit function to call when the pack is finished. csdtdd}|}|rZtjdtdd|D}t}t||| j |d j |d|| t |j d }||||SdS) Nrr))rQrXpack-cs|]}|dVqdSrNr*)r,entryr*r*r.rz.commit..r$.idxr')rrsorted_entriesrrr"rrrr rWrrtr check_length_and_checksum_add_cached_pack)rmentriesr,index final_packrYr=r*r.r0s&   z)SwiftObjectStore.add_pack..commitrxNcSr rqr*r*r*r*r.abortr z(SwiftObjectStore.add_pack..abortrr)r=r0rHr*rGr.add_packs  zSwiftObjectStore.add_packcCs||dfgdSrq) add_objects)r=rnr*r*r. add_objectszSwiftObjectStore.add_objectcCdSNFr*rr*r*r._pack_cache_staler z"SwiftObjectStore._pack_cache_stalecCr rqr*)r=r>r*r*r._get_loose_objectr z"SwiftObjectStore._get_loose_objectc Cs|tjdd\}}t|d}z%t||jd}t||||d}||||||W| t |S| t |w)zRead a thin pack. Read it from a stream and complete it in a temporary file. Then the pack and the corresponding index file are uploaded to Swift. tmp_pack_)prefixzw+b)resolve_ext_ref) delta_iter) tempfilemkstemprJfdopenrget_rawrverify_complete_thin_packrtunlink)r=read_all read_somefdrNrYindexercopierr*r*r. add_thin_packs   zSwiftObjectStore.add_thin_packc Cst|}|dt|t|t||t|dd}|dtj|D]%}t|dks5J| |\}} | } t ||| |d} | || | fq+| } || ||t|jdttdd|D} |j| d || d }t}t||| |j|||dtd |d }|dtd |}t||}||t|}| d }|j|||t| |jd}||| ||S)Nri)end_ofsr)r>r;csr<r=r*)r,r9r*r*r.rr?z7SwiftObjectStore._complete_thin_pack..r$r@r))rXrQr r') listrrr<ext_refsflushrrJSEEK_CURrWtellrr)digestwritesortrrr"fsdecoderrWrrrrrrprtr rBrC)r=rYrNr_r^rDnew_shaext_shar7rrcrc32pack_shapack_base_namerX index_filerkrlserialized_pack_infopack_info_filerFr*r*r.rYsP           z$SwiftObjectStore._complete_thin_packr)rCrDrErrr-r.r6r2r:rIrKrrNrOr`rYr r*r*rr.r![s  # r!csfeZdZdZdfdd ZddZddd Z    ddefd d Z ddefd d Z ddZ Z S)SwiftInfoRefsContainerz&Manage references in info/refs object.rxNcs<||_d|_||_|j|j}|std}t|dS)Nz info/refsr)rWrXstorerVrrr)r=rWrtrYrr*r.rszSwiftInfoRefsContainer.__init__cCsP|||j|j}|siSt|}t|\}}|dur&|||kr&dS|SrM)_check_refnamerWrVrXr!r")r=r?old_refrYrefspeeledr*r*r._load_check_ref(s   z&SwiftInfoRefsContainer._load_check_refcCs,t}|t||j|j|j|dSrq)r writelinesr#rtrWrrX)r=rwrYr*r*r. _write_refs4sz"SwiftInfoRefsContainer._write_refsc CsF|dkrdS|||}t|tsdS|||<||||j|<dS)z=Set a refname to new_ref only if it currently equals old_ref.rTFry isinstancedictr{_refs) r=r?rvnew_ref committer timestamptimezonemessagerwr*r*r. set_if_equals9s     z$SwiftInfoRefsContainer.set_if_equalscCsB|dkrdS|||}t|tsdS||=|||j|=dS)z5Remove a refname only if it currently equals old_ref.rTFr|)r=r?rvrrrrrwr*r*r.remove_if_equalsNs   z'SwiftInfoRefsContainer.remove_if_equalscCs2z |jd|jd<Wn tyYnw|jS)Nzrefs/heads/masterr)rrLkeysrr*r*r.allkeys\s   zSwiftInfoRefsContainer.allkeysr)NNNN) rCrDrErrryr{rrrrr r*r*rr.rss"    rsc@s:eZdZd ddZdefddZd ddZed d ZdS) SwiftReporxNcCs|d|_||_t|j|j|_|j}|s td|jdd|D}t|vr4td|jdd|_|j|_ t |j}t |j|}t |||dS) abInit a Git bare Repository on top of a Swift container. References are managed in info/refs objects by `SwiftInfoRefsContainer`. The root attribute is the Swift container that contain the Git bare repository. Args: root: The container which contains the bare repo conf: A ConfigParser object rz!There is not any GIT repo here : cSsg|] }|dddqS)r?rr)rr%r*r*r.r/vsz&SwiftRepo.__init__..zThis repository (z) is not bare.TN)lstriprrRrwrWrrMr$bare _controldirr!rsr%r)r=rrRr*r5rwr*r*r.res    zSwiftRepo.__init__cCrL)zProbe the file-system to determine whether permissions can be trusted. Returns: True if permissions can be trusted, False otherwise. Fr*rr*r*r._determine_file_modeszSwiftRepo._determine_file_modecCsDt}|||j||WddS1swYdS)zPut an object in a Swift container. Args: filename: the path to the object to put on Swift contents: the content as bytestring N)rrhrWr)r=rXcontentsrYr*r*r._put_named_files "zSwiftRepo._put_named_filecCsP|tttttdfD] }||tdq||j|}| d|S)zCreate a new bare repository. Args: scon: a `SwiftConnector` instance conf: a ConfigParser object Returns: a `SwiftRepo` instance rwrT) rrrr$r r rrr _init_files)clsrWrRrnrr*r*r. init_bares    zSwiftRepo.init_barer) rCrDrErrrr classmethodrr*r*r*r.rds    rc@seZdZdddZddZdS)SwiftSystemBackendrxNcCs||_||_dSrq)rRlogger)r=rrRr*r*r.rs zSwiftSystemBackend.__init__cCs|jd|t||jS)Nzopening repository at %s)rrArrR)r=rNr*r*r.open_repositorys z"SwiftSystemBackend.open_repositoryr)rCrDrErrr*r*r*r.rs  rrxc Csddl}|}|jdddddd|jd d d ttd d |jdddddd||\}}z ddl}ddl}WntyJt dt dYnwddl }|j ddlm}|t}t|j}t||} |t| |j|jd} | dS)z*Entry point for starting a TCP git server.rNz-lz--listen_addresslisten_addressz 127.0.0.1zBinding IP address.destdefaulthelpz-pz--portportzBinding TCP port.)rrrr-c--swift_config swift_configr)1Path to the configuration file for Swift backend.zOgevent and geventhttpclient libraries are mandatory for use the Swift backend.r0) log_utils)r)optparse OptionParser add_optionrr parse_argsgeventgeventhttpclient ImportErrorprintsysexit gevent.monkeymonkey patch_socketdulwichr getLoggerrCrUrrdefault_logging_configr'rr serve_forever) rrparseroptionsrrrrrRbackendserverr*r*r. cmd_daemonsV        rcCspddl}|}|jdddddd||\}}t|j}|gkr'|d|d}t||}t ||dS) Nrrrrr)rrzmissing repository name) rrrrrUrerrorrwrr)rrrrrRreporWr*r*r.cmd_inits    rcCsttd}ttjdkr!tdtjdd|t dtjd}||vr6td|t d||tjdddS)N)initdaemonrzUsage: {} <{}> [OPTIONS...]r|r0zNo such subcommand: ) rrr<rargvrformatrrr)rcommandscmdr*r*r.mains   r__main__)NNr)PrrirJrr[rrT urllib.parseparserrg configparserriortypingrrr greenthreadsr lru_cacher r5r r r r*r rrrrrmrrrrrrrrrrrrrprotocolrrwr r!r"r#rr$r%rr&r'r(rUrZrprurMrvrwrrr r!rsrrrrrrrCr*r*r*r.s\       <    I,BHB 4