o  h @sddlZddlmZddlZddlmZddlmZmZddlm Z e e Z Gdddej ZGd d d ej ZGd d d ej ZGd ddej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddeZided ed!d"d#fd$ed%ed&ed'd(ifd)ed*ed+ed,ejd-ed.ed/ed0ejd1ed2ejd3ejd4ejejejej d5Z!ee!Z"d6d7Z#e#d&Z$e#d%Z%e#dZ&e#d$Z'e#d/Z(e#d4Z)e#d.Z*e#d-Z+dS)8N) OrderedDict)version)Tensornn)loggingc2eZdZdZfddZdedefddZZS)PytorchGELUTanha A fast C implementation of the tanh approximation of the GeLU activation function. See https://arxiv.org/abs/1606.08415. This implementation is equivalent to NewGELU and FastGELU but much faster. However, it is not an exact numerical match due to rounding errors. cs6tttjtdkrtdtjddS)Nz1.12.0zYou are using torch==zM, but torch>=1.12.0 is required to use PytorchGELUTanh. Please upgrade torch.)super__init__rparsetorch __version__ ImportErrorself __class__l/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/activations.pyr %s  zPytorchGELUTanh.__init__inputreturncCstjj|ddS)Ntanh) approximate)r functionalgelurrrrrforward-szPytorchGELUTanh.forward__name__ __module__ __qualname____doc__r rr __classcell__rrrrr s r c@"eZdZdZdedefddZdS)NewGELUActivationz Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see the Gaussian Error Linear Units paper: https://arxiv.org/abs/1606.08415 rrc Cs6d|dttdtj|dt|dS)N??@Hm?g@)r rmathsqrtpipowrrrrr7s6zNewGELUActivation.forwardNrr r!r"rrrrrrr%1sr%csLeZdZdZd deffdd Zdedefdd Zdedefd d ZZ S) GELUActivationa Original Implementation of the GELU activation function in Google BERT repo when initially created. For information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional Also see the Gaussian Error Linear Units paper: https://arxiv.org/abs/1606.08415 Fuse_gelu_pythoncs(t|r |j|_dStjj|_dSN)r r _gelu_pythonactrrr)rr0rrrr Cs  zGELUActivation.__init__rrcCs |ddt|tdS)Nr&r'r()r erfr*r+rrrrr2Js zGELUActivation._gelu_pythoncC ||Sr1r3rrrrrM zGELUActivation.forward)F) rr r!r"boolr rr2rr#rrrrr/;s r/c@r$)FastGELUActivationz} Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs rrcCs*d|dt|ddd||S)Nr&r'g3E?r))r rrrrrrVs*zFastGELUActivation.forwardNr.rrrrr9Qr9c@r$)QuickGELUActivationzr Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs rrcCs|td|S)NgZd;?)r sigmoidrrrrr_szQuickGELUActivation.forwardNr.rrrrr;Zr:r;cs<eZdZdZdedeffdd Zdedefdd ZZS) ClippedGELUActivationa Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to https://arxiv.org/abs/2004.09602. Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://arxiv.org/abs/1606.08415 minmaxcs8||krtd|d|dt||_||_dS)Nzmin should be < max (got min: z, max: )) ValueErrorr r r>r?)rr>r?rrrr ps   zClippedGELUActivation.__init__xrcCstt||j|jSr1)r cliprr>r?)rrBrrrrxzClippedGELUActivation.forward) rr r!r"floatr rrr#rrrrr=cs r=cr)AccurateGELUActivationz Applies GELU approximation that is faster than default and more accurate than QuickGELU. See: https://github.com/hendrycks/GELUs Implemented along with MEGA (Moving Average Equipped Gated Attention) cs ttdtj|_dS)N)r r r*r+r,precomputed_constantrrrrr s zAccurateGELUActivation.__init__rrc Cs,d|dt|j|dt|dS)Nr&rr))r rrHr-rrrrrs,zAccurateGELUActivation.forwardrrrrrrF|s rFcsDeZdZdZfddZdedefddZdedefdd ZZS) MishActivationz See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://arxiv.org/abs/1908.08681). Also visit the official repository for the paper: https://github.com/digantamisra98/Mish cs:tttjtdkr|j|_dStj j |_dS)Nz1.9.0) r r rr r r _mish_pythonr3rrmishrrrrr s  zMishActivation.__init__rrcCs|ttj|Sr1)r rrrsoftplusrrrrrKrDzMishActivation._mish_pythoncCr5r1r6rrrrrr7zMishActivation.forward) rr r!r"r rrKrr#rrrrrJs  rJc@r$)LinearActivationz[ Applies the linear activation function, i.e. forwarding input directly to output. rrcCs|Sr1rrrrrrszLinearActivation.forwardNr.rrrrrNr:rNc@seZdZdZdddZdS)LaplaceActivationz Applies elementwise activation based on Laplace function, introduced in MEGA as an attention activation. See https://arxiv.org/abs/2209.10655 Inspired by squared relu, but with bounded range and gradient for better stability 绹۞? ^/ ?cCs*|||td}ddt|S)Nr(r&r')divr*r+r r4)rrmusigmarrrrszLaplaceActivation.forwardN)rPrQrr r!r"rrrrrrOsrOc@seZdZdZddZdS)ReLUSquaredActivationzX Applies the relu^2 activation introduced in https://arxiv.org/abs/2109.08668v2 cCstj|}t|}|Sr1)rrrelur square)rr relu_appliedsquaredrrrrs  zReLUSquaredActivation.forwardNrUrrrrrVs rVcseZdZfddZZS)ClassInstantiercs4t|}t|tr |n|if\}}|di|S)Nr)r __getitem__ isinstancetuple)rkeycontentclskwargsrrrr\s zClassInstantier.__getitem__)rr r!r\r#rrrrr[sr[rgelu_10i )r>r? gelu_fastgelu_new gelu_pythonr0Tgelu_pytorch_tanh gelu_accuratelaplace leaky_relulinearrL quick_gelurWrelu2relu6r<silu)swishrprelucCs,|tvrt|Std|dtt)Nz function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)activation_stringrrrget_activationsrx),r* collectionsrr packagingrrrutilsr get_loggerrloggerModuler r%r/r9r;r=rFrJrNrOrVr[ LeakyReLUReLUReLU6SigmoidSiLUTanhPReLUACT2CLSrsrxrgrfrrermrprL linear_actrrrrs