RuntimeError: 0D or 1D target tensor expected, multi-target not supported error is raised when the target tensor (labels) has an incorrect shape. In PyTorch, the CrossEntropyLoss expects the target tensor to have a 1D shape.
To fix this error, make sure your labels tensor has the correct shape. The tensor should be (batch_size,), where batch_size is the number of samples in your input. The example assumes a single input sample, so the shape should be (1,).
Each element in this target tensor should be a class index in the range [0, C-1], where C is the number of classes.
If the target tensor is not in this format (for example, if it’s a 2D tensor or one-hot encoded), this error will occur.
import torch
from transformers import BertTokenizer, BertForSequenceClassification
from torch.nn import CrossEntropyLoss
# Loading the model and tokenizer
model_name = "bert-base-uncased"
tokenizer = BertTokenizer.from_pretrained(model_name)
model = BertForSequenceClassification.from_pretrained(model_name)
# Prepare your input
input_text = "Your input text goes here"
tokens = tokenizer(input_text, return_tensors="pt")
input_ids = tokens["input_ids"]
attention_mask = tokens["attention_mask"]
# Forward pass to get the logits
logits = model(input_ids, attention_mask=attention_mask).logits
# Prepare your labels
labels = torch.tensor([0]) # Assuming you have a binary classification task and a single input sample
# Compute the loss
loss_function = CrossEntropyLoss()
loss = loss_function(logits, labels)
print(loss)
Output
For nn.CrossEntropyLoss, the target has to be a single number from the interval [0, #classes] instead of a one-hot encoded target vector. Your target is [1, 0]. Thus, PyTorch thinks you want multiple labels per input, which is unsupported.
You can replace your one-hot-encoded targets like this:
[1, 0] --> 0
[0, 1] --> 1
Correct Target Tensor Shape for a Single Sample
If you have a single input sample, your target tensor should be of shape (1,) with the single element being the class index.
For example: target = torch.tensor([class_index]) where class_index is the integer representing the class.
Correct Target Tensor Shape for Batch:
For a batch of samples, the target tensor should be a 1D tensor, with each element corresponding to the class index of each sample in the batch.
For example: target = torch.tensor([class_index1, class_index2, …, class_indexN]) for a batch of N samples.