LW-DETR Object Detection Models

Construct LW-DETR model variants for real-time object detection. LW-DETR is a lightweight Detection Transformer that combines a ViT encoder, a C2f multi-scale projector, and a shallow DETR decoder with deformable cross-attention.

model_lw_detr_tiny(
  pretrained = FALSE,
  progress = TRUE,
  num_classes = 91L,
  num_select = 100L,
  ...
)

model_lw_detr_small(
  pretrained = FALSE,
  progress = TRUE,
  num_classes = 91L,
  num_select = 300L,
  ...
)

model_lw_detr_medium(
  pretrained = FALSE,
  progress = TRUE,
  num_classes = 91L,
  num_select = 300L,
  ...
)

model_lw_detr_large(
  pretrained = FALSE,
  progress = TRUE,
  num_classes = 91L,
  num_select = 300L,
  ...
)

Arguments

pretrained: Logical. If TRUE, loads COCO pretrained weights.
progress: Logical. Show progress bar during download (unused).
num_classes: Integer. Number of object classes (default: 91 for COCO).
num_select: Integer. Number of top-scoring detections to return per image.
...: Additional arguments (unused).

Value

An lw_detr nn_module.

Functions

model_lw_detr_tiny(): LW-DETR tiny — ViT-tiny, 6 layers, 100 queries
model_lw_detr_small(): LW-DETR small — ViT-tiny, 10 layers, 300 queries
model_lw_detr_medium(): LW-DETR medium — ViT-small, 10 layers, 300 queries
model_lw_detr_large(): LW-DETR large — ViT-small, 10 layers, 2-scale, 300 queries

Input Format

The forward method is model(images, pixel_mask = NULL), where images is an ImageNet-normalised torch_tensor of shape (batch_size, 3, H, W) with H and W divisible by 64 (recommended 640 x 640). Normalise with mean = c(0.485, 0.456, 0.406), std = c(0.229, 0.224, 0.225).

The reference evaluation resizes every image to 640×640 without keeping the aspect ratio. Alternatively, resize the longest side to 640, pad to 640×640 and pass a pixel_mask of shape (batch_size, H, W) (logical, TRUE over real pixels and FALSE over padding), as with the reference model's padded NestedTensor input. The padded region is then excluded from attention and boxes are returned in the coordinates of the unpadded image. When pixel_mask is omitted the whole frame is treated as valid.

Output Format

Returns a list with element detections: a list (one per image) of lists with:

boxes — tensor (k, 4) in xyxy pixel coordinates
labels — integer tensor (k,) of COCO category ids (e.g. 17 = cat); pass to coco_classes() for names
scores — float tensor (k,) — confidence scores

References

Chen et al. (2024). LW-DETR: A Transformer Replacement to YOLO for Real-Time Detection. https://arxiv.org/abs/2406.03459

Examples

if (FALSE) { # \dontrun{
norm_mean <- c(0.485, 0.456, 0.406)
norm_std  <- c(0.229, 0.224, 0.225)

# A non-square demo image from the LW-DETR repository
url <- "https://raw.githubusercontent.com/Atten4Vis/LW-DETR/main/demo/000000496954.jpg"
img <- base_loader(url) |> transform_to_tensor()
h <- img$shape[2]
w <- img$shape[3]

# Letterbox the longest side to 640 and build the matching pixel mask
s  <- 640 / max(h, w)
nh <- round(h * s)
nw <- round(w * s)
resized <- img |> transform_resize(c(nh, nw))
canvas  <- torch::torch_zeros(c(3, 640, 640))
canvas[, 1:nh, 1:nw] <- resized
mask <- torch::torch_zeros(c(640, 640), dtype = torch::torch_bool())
mask[1:nh, 1:nw] <- TRUE

input <- canvas |> transform_normalize(norm_mean, norm_std)

model <- model_lw_detr_tiny(pretrained = TRUE)
model$eval()
pred <- torch::with_no_grad(
  model(input$unsqueeze(1), pixel_mask = mask$unsqueeze(1))
)$detections[[1]]

# Draw the most confident detections on the letterboxed image
topk   <- pred$scores$topk(k = 5L)[[2]]
boxes  <- pred$boxes[topk, ]
labels <- coco_classes(as.integer(pred$labels[topk]))
boxed  <- draw_bounding_boxes(canvas, boxes, labels = labels)
tensor_image_browse(boxed)
} # }