relu[source]

relu(inplace=True)

a = torch.randn(3, 3)
a, relu()(a)

(tensor([[0.0000, 0.0000, 0.1396],
         [0.7610, 0.0000, 0.1118],
         [0.0000, 0.0000, 0.0000]]),
 tensor([[0.0000, 0.0000, 0.1396],
         [0.7610, 0.0000, 0.1118],
         [0.0000, 0.0000, 0.0000]]))

x = torch.arange(-5, 3, 0.05)
y = Relu(inplace=False)(x)
plt.plot(x, y);

LeakyReLU[source]

LeakyReLU(inplace=True, negative_slope=0.01)

a = torch.randn(3, 3)
a, leaky_relu()(a)

(tensor([[ 0.4458,  0.0084,  1.1934],
         [-0.0066, -0.0087, -0.0084],
         [ 0.1442,  0.6694,  0.2855]]),
 tensor([[ 0.4458,  0.0084,  1.1934],
         [-0.0066, -0.0087, -0.0084],
         [ 0.1442,  0.6694,  0.2855]]))

x = torch.arange(-5, 3, 0.05)
y = LeakyReLU(inplace=False, negative_slope=0.5)(x)
plt.plot(x, y);

Mish

$$ \begin{align} \text{mish}(x) &= \text{tanh}\left(\text{softplus}(x)\right)\\ \text{tanh}(x) &= \frac{\exp(2x) - 1}{\exp(2x) + 1}\\ \text{softplus}(x) &= \ln(1 + \exp(x)) \end{align} $$

class Mish[source]

Mish() :: Module

Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

x = torch.arange(-5, 3, 0.05)
y = Mish()(x)
plt.plot(x, y);

Sigmoid Activation

Usually used to convert a number into a probability: $$ \sigma(x) = \frac{1}{1+\exp(-x)} $$

sigmoid()(torch.randn((3,1)))

tensor([[0.3305],
        [0.4534],
        [0.6104]])

Softmax Activation

Converts multicategorical numerical numbers to a probability distribution. Suppose $y_ij$ is the output for the i-th instance for the j-th category. Then the probability is: $$ p(y_{ij}) = \frac{\exp(y_{ij})}{\sum_j \exp(y_{ij})} $$

y = torch.randn((2,3))
p = softmax(y)
print(p)
print(p.sum(dim=-1, keepdim=True))

tensor([[0.2092, 0.1568, 0.6340],
        [0.2197, 0.3284, 0.4519]])
tensor([[1.],
        [1.]])

get_activation[source]

get_activation(activation)