o  hQ @s ddlZddlZddlmZzddlZWneefy/ddlZeej j dkr-e dYnwddZ ddZ d d Zd d Zd dZddZdddZeejjedkrbddZejjZeZne Ze Zeeeeeeeejjejjejjejjejjd ZddZdS)N)parsezYour currently installed version of Keras is Keras 3, but this is not yet supported in Transformers. Please install the backwards-compatible tf-keras package with `pip install tf-keras`.c Cs:t|}ddtj|ttd|j}||S)av Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) Also see https://arxiv.org/abs/1606.08415 ??@)tfconvert_to_tensormatherfcastsqrtdtype)xcdfro/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/transformers/activations_tf.py_gelu"s (rc Cs`t|}ttj|j}td|j}ddttd|||t|d}||S)z Gaussian Error Linear Unit. This is a smoother version of the GELU. Original paper: https://arxiv.org/abs/1606.0841 Args: x: float Tensor to perform activation Returns: `x` with the GELU activation applied. Hm?rrr) rrr r pir tanhr pow)rrcoeffrrrr _gelu_new/s 0rcCs t|}|ttj|S)N)rrrr softplusrrrrmishAs rcCsPt|}td|j}td|j}d|dt||d|||S)Nrg3E?rr)rrr r r)rcoeff1coeff2rrr gelu_fastGs *rcCs,t|}td|j}|tj||S)NgZd;?)rrr r r sigmoid)rrrrr quick_geluOs r!cCstt|ddS)a Clip the range of possible GeLU outputs between [-10, 10]. This is especially useful for quantization purpose, as it allows mapping 2 negatives values in the GeLU spectrum. For more information on this trick, please refer to https://arxiv.org/abs/2004.09602 Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) Also see https://arxiv.org/abs/1606.08415 :param x: :return: i )r clip_by_valuerrrrrgelu_10Us r$cCs$tj|d|d\}}|tj|S)a Gated Linear Unit. Implementation as defined in the original paper (see https://arxiv.org/abs/1612.08083), where the input `x` is split in two halves across a dimension (`axis`), A and B, returning A * sigmoid(B). Args: `x`: float Tensor to perform activation `axis`: dimension across which `x` be split in half Returns: `x` with the GLU activation applied (with its size halved across the dimension `axis`). r)axis)rsplitr r )rr&abrrrglucs r*z2.4cCstjj|ddS)NT) approximate)keras activationsgelurrrrapproximate_gelu_wrapusr/) r.r$rgelu_newr*rr!relur siluswishrcCs,|tvrt|Std|dtt)Nz function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)activation_stringrrrget_tf_activationsr9)r%)r tensorflowrpackaging.versionrtf_kerasr,ModuleNotFoundError ImportError __version__major ValueErrorrrrrr!r$r*versionVERSIONr/r-r.r0r1r r3rr4r9rrrrsP