sigmoid, tanh 함수 미분

Logistic Sigmoid (Activation) Function 미분

$$g\left(a\right) = \frac{1}{1+e^{-a}}$$

Using

$$ \frac{d}{dx} \left( \frac{u}{v} \right) = \frac{v\left(du/dx\right) - u\left(dv/dx\right)}{v^{2}},$$

$$ g'\left(a\right) = \frac{e^{-a}}{\left( 1 + e^{-a} \right)^2} = \frac{1 + e^{-a} - 1}{\left( 1 + e^{-a} \right)^2} = \frac{1}{\left( 1 + e^{-a} \right)} - \frac{1}{\left( 1 + e^{-a} \right)^2}$$

$$= \frac{1}{\left( 1 + e^{-a} \right)} \left( 1 - \frac{1}{\left( 1 + e^{-a} \right)} \right) $$

$$= g(a) \left( 1 - g(a) \right)$$

Hyperbolic Tangent (Activation) Function 미분

$$ g(a) = tanh(a) = \frac{e^a - e^{-a}}{e^a + e^{-a}} = \frac{1 - e^{-2a}}{1 + e^{-2a}} $$

$$ g'(a) = \frac{2e^{-2a} \left( 1 + e^{-2a} \right) - \left( 1 - e^{-2a}\right) \left( -2e^{-2a}\right) }{\left( 1 + e^{-2a} \right)^2 } $$

$$ = \frac{2e^{-2a} \left( 1 + e^{-2a} + 1 - e^{-2a}\right)}{\left( 1 + e^{-2a} \right)^2} = \frac{4e^{-2a}}{\left(1 + e^{-2a} \right)^2}$$

$$ = \frac{\left(1+e^{-2a} \right)^2 - \left(1-e^{-2a} \right)^2}{\left(1 + e^{-2a} \right)^2} = \frac{\left(1 + e^{-2a} \right)^2}{\left(1 + e^{-2a} \right)^2} - \frac{\left(1 - e^{-2a} \right)^2}{\left(1 + e^{-2a} \right)^2} $$

$$ = 1 - \left( \frac{1 - e^{-2a}}{1 + e^{-2a}}\right)^2 = 1 - g(a)^2 = \left(1+g(a)\right)\left(1-g(a)\right) $$

where,

$$ 4e^{-2a} = \left(1 + 2e^{-2a} + e^{-4a} \right) - \left(1 - 2e^{-2a} + e^{-4a} \right) $$

$$ = \left(1 + e^{-2a}\right)^2 - \left(1 - e^{-2a}\right)^2 $$