I ran some numbers, to be double-checked because I’m tired.
I’m generating Float16 domain HFD above 1 and finding the point at which the difference between a compressed value and the next one is 0, tanh is looking a bit sad here, not that the other ones are great either, atan is a good compromise:
argmax(f(HFD[x]) - f(HFD[x + 1]))
[ Threshold 0.0 ]
tanh 6.58594
atan 7376.0
simple 23168.0
[ Threshold 0.1 ]
tanh 6.02734
atan 6636.0
simple 14744.0
[ Threshold 0.2 ]
tanh 5.46875
atan 5900.0
simple 13104.0
[ Threshold 0.3 ]
tanh 4.91016
atan 3650.0
simple 8108.0
[ Threshold 0.4 ]
tanh 4.35156
atan 3130.0
simple 6952.0
[ Threshold 0.5 ]
tanh 3.61914
atan 1844.0
simple 5792.0
[ Threshold 0.6 ]
tanh 3.0957
atan 1476.0
simple 3278.0
[ Threshold 0.7 ]
tanh 2.57227
atan 783.0
simple 1738.0
[ Threshold 0.8 ]
tanh 1.97852
atan 369.5
simple 820.0
[ Threshold 0.9 ]
tanh 1.48926
atan 93.0625
simple 205.625
[ Threshold 1.0 ]
tanh 1
atan 1
simple 1
https://colab.research.google.com/drive/1f-5A-u7hqklDHYoWNLsQVSsy8MCUk6Js?usp=sharing
PS: Keep in mind that no particular distance is set, e.g. tanh defaults to inf and that the threshold values between the functions are not really comparable, tanh and atan can go lower than simple.