optimizer got an empty parameter list-CodePudding

If I use optim.SGD(model_conv.fc.parameters() I'm getting an error:

optimizer got an empty parameter list

This error is, when model_conv.fc is nn.Hardtanh(...) (also when I try to use ReLu). But with nn.Linear it works fine.

What could be the reason?

model_conv.fc = nn.Hardtanh(min_val=0.0, max_val=1.0) #not ok --> optimizer got an empy parameter list
#model_conv.fc = nn.ReLU() #also Not OK
# num_ftrs = model_conv.fc.in_features
# model_conv.fc = nn.Linear(num_ftrs, 1) #it works fine


model_conv = model_conv.to(config.device())

optimizer_conv = optim.SGD(model_conv.fc.parameters(), lr=config.learning_rate, momentum=config.momentum) #error is here

CodePudding user response：

Hardtanh and ReLU are parameter-free layers but Linear has parameters.

CodePudding user response：

Activation functions are used to add non-linearity to your model which is parameter-free, so you should pass a nn.Linear as a fully-connected (FC) layer

num_ftrs = model_conv.fc.in_features
model_conv.fc = nn.Sequential( list_of_FC_layers )

e.g

model_conv.fc = nn.Sequential( nn.Linear(in_features, hidden_neurons),
                               nn.Linear(hidden_neurons, out_channels) )

model_conv.fc = nn.Linear( in_features, out_channels)

out_channels in case of binary-classification task is 1 and in case of multi-class classification is num_classes

NOTE.1: for multi-class classification, Do Not use the softmax layer as it is done in CrossEntropyLoss

NOTE.2 : You can use Sigmoid activation for binary classification use