o U h-@sdZddlmZddlmZddlmZdZdZddZ d d Z d d Z d dZ ddZ ddZGdddZddZGdddeZdS)aAll line-ending related functions, from conversions to config processing. Line-ending normalization is a complex beast. Here is some notes and details about how it seems to work. The normalization is a two-fold process that happens at two moments: - When reading a file from the index and to the working directory. For example when doing a ``git clone`` or ``git checkout`` call. We call this process the read filter in this module. - When writing a file to the index from the working directory. For example when doing a ``git add`` call. We call this process the write filter in this module. Note that when checking status (getting unstaged changes), whether or not normalization is done on write depends on whether or not the file in the working dir has also been normalized on read: - For autocrlf=true all files are always normalized on both read and write. - For autocrlf=input files are only normalized on write if they are newly "added". Since files which are already committed are not normalized on checkout into the working tree, they are also left alone when staging modifications into the index. One thing to know is that Git does line-ending normalization only on text files. How does Git know that a file is text? We can either mark a file as a text file, a binary file or ask Git to automatically decides. Git has an heuristic to detect if a file is a text file or a binary file. It seems based on the percentage of non-printable characters in files. The code for this heuristic is here: https://git.kernel.org/pub/scm/git/git.git/tree/convert.c#n46 Dulwich have an implementation with a slightly different heuristic, the `dulwich.patch.is_binary` function. The binary detection heuristic implementation is close to the one in JGit: https://github.com/eclipse/jgit/blob/f6873ffe522bbc3536969a3a3546bf9a819b92bf/org.eclipse.jgit/src/org/eclipse/jgit/diff/RawText.java#L300 There is multiple variables that impact the normalization. First, a repository can contains a ``.gitattributes`` file (or more than one...) that can further customize the operation on some file patterns, for example: \*.txt text Force all ``.txt`` files to be treated as text files and to have their lines endings normalized. \*.jpg -text Force all ``.jpg`` files to be treated as binary files and to not have their lines endings converted. \*.vcproj text eol=crlf Force all ``.vcproj`` files to be treated as text files and to have their lines endings converted into ``CRLF`` in working directory no matter the native EOL of the platform. \*.sh text eol=lf Force all ``.sh`` files to be treated as text files and to have their lines endings converted into ``LF`` in working directory no matter the native EOL of the platform. If the ``eol`` attribute is not defined, Git uses the ``core.eol`` configuration value described later. \* text=auto Force all files to be scanned by the text file heuristic detection and to have their line endings normalized in case they are detected as text files. Git also have a obsolete attribute named ``crlf`` that can be translated to the corresponding text attribute value. Then there are some configuration option (that can be defined at the repository or user level): - core.autocrlf - core.eol ``core.autocrlf`` is taken into account for all files that doesn't have a ``text`` attribute defined in ``.gitattributes``; it takes three possible values: - ``true``: This forces all files on the working directory to have CRLF line-endings in the working directory and convert line-endings to LF when writing to the index. When autocrlf is set to true, eol value is ignored. - ``input``: Quite similar to the ``true`` value but only force the write filter, ie line-ending of new files added to the index will get their line-endings converted to LF. - ``false`` (default): No normalization is done. ``core.eol`` is the top-level configuration to define the line-ending to use when applying the read_filer. It takes three possible values: - ``lf``: When normalization is done, force line-endings to be ``LF`` in the working directory. - ``crlf``: When normalization is done, force line-endings to be ``CRLF`` in the working directory. - ``native`` (default): When normalization is done, force line-endings to be the platform's native line ending. One thing to remember is when line-ending normalization is done on a file, Git always normalize line-ending to ``LF`` when writing to the index. There are sources that seems to indicate that Git won't do line-ending normalization when a file contains mixed line-endings. I think this logic might be in text / binary detection heuristic but couldn't find it yet. Sources: - https://git-scm.com/docs/git-config#git-config-coreeol - https://git-scm.com/docs/git-config#git-config-coreautocrlf - https://git-scm.com/docs/gitattributes#_checking_out_and_checking_in - https://adaptivepatchwork.com/2012/03/01/mind-the-end-of-your-line/ )iter_tree_contents)Blob) is_binarys  cCs |ttS)zConvert CRLF in text hunk into LF. Args: text_hunk: A bytes string representing a text hunk Returns: The text hunk with the same type, with CRLF replaced into LF replaceCRLFLF) text_hunkr g/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/dulwich/line_ending.pyconvert_crlf_to_lfs r cCs|tt}|ttS)zConvert LF in text hunk into CRLF. Args: text_hunk: A bytes string representing a text hunk Returns: The text hunk with the same type, with LF replaced into CRLF r)r intermediaryr r r convert_lf_to_crlfs  rcCt|S)zBReturns the correct checkout filter based on the passed arguments.)get_checkout_filter_autocrlfcore_eol core_autocrlfgit_attributesr r r get_checkout_filterrcCr)zAReturns the correct checkin filter based on the passed arguments.)get_checkin_filter_autocrlfrr r r get_checkin_filterrrcCs|dkrtSdS)aGReturns the correct checkout filter base on autocrlf value. Args: core_autocrlf: The bytes configuration value of core.autocrlf. Valid values are: b'true', b'false' or b'input'. Returns: Either None if no filter has to be applied or a function accepting a single argument, a binary text hunk trueN)rrr r r rs rcCs|dks|dkr tSdS)aFReturns the correct checkin filter base on autocrlf value. Args: core_autocrlf: The bytes configuration value of core.autocrlf. Valid values are: b'true', b'false' or b'input'. Returns: Either None if no filter has to be applied or a function accepting a single argument, a binary text hunk rsinputN)r rr r r rs rc@s*eZdZdZd ddZddZdd ZdS) BlobNormalizerzAn object to store computation result of which filter to apply based on configuration, gitattributes, path and operation (checkin or checkout). returnNcCs||_||_z|dd}Wn tyd}Ynwz |dd}Wn ty/d}Ynwt|||j|_t|||j|_dS)NcoreeolnativeautocrlfF) config_stack gitattributesgetKeyErrorlowerrfallback_read_filterrfallback_write_filter)selfr"r#rrr r r __init__s$   zBlobNormalizer.__init__cC|jdur t||jddS|S)z,Normalize a blob during a checkin operation.NTbinary_detection)r(normalize_blobr)blob tree_pathr r r checkin_normalize z BlobNormalizer.checkin_normalizecCr+)z-Normalize a blob during a checkout operation.NTr,)r'r.r/r r r checkout_normalizer3z!BlobNormalizer.checkout_normalizerN)__name__ __module__ __qualname____doc__r*r2r4r r r r rs   rcCs2|j}|dur t|r |S||}t}||_|S)zTakes a blob as input returns either the original blob if binary_detection is True and the blob content looks like binary, else return a new blob with converted data. T)datarr)r0 conversionr-r:converted_datanew_blobr r r r.sr.cs,eZdZddfdd ZfddZZS) TreeBlobNormalizerNrcs8t|||rddt||D|_dSt|_dS)NcSsh|]\}}}|qSr r ).0name_r r r #s z.TreeBlobNormalizer.__init__..)superr*rexisting_pathsset)r)r"r object_storetree __class__r r r* s  zTreeBlobNormalizer.__init__cs&|jdus ||jvrt||S|SN)r'rDrCr2r/rHr r r2)s  z$TreeBlobNormalizer.checkin_normalizerJr5)r6r7r8r*r2 __classcell__r r rHr r>s r>N)r9rFrobjectsrpatchrrr r rrrrrrr.r>r r r r s w    .