This issue tracks the candidate methods that could maybe be implemented as Scalarizer in torchjd.scalarization (see #666).
| Status |
Name |
Ref |
Stateful |
Existing implementations |
Special Remarks |
| 🟢 To do |
Sum |
- |
No |
(trivial) |
|
| 🟢 To do |
Mean |
- |
No |
(trivial) |
Sometimes called Equal Weights (EW) in research papers. |
| 🟢 To do |
Linear |
- |
No |
(trivial) |
Sometimes called Linear Scalarization (LS) in research papers. Name it Linear or Constant? |
| 🟢 To do |
Random |
Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learning (TMLR 2022, 200 citations) |
No |
LibMTL (official) |
Sometimes called Random Loss Weighting (RLW) in research papers. Name it RLW or Random? |
| 🔵 To investigate |
STCH (Smooth TCHebyshev) |
Smooth Tchebycheff Scalarization for Multi-Objective Optimization (ICML 2024, 87 citations) |
No |
official, LibMTL |
|
| 🔵 To investigate |
GLS (Geometric Loss Strategy) |
MultiNet++: Multi-Stream Feature Aggregation and Geometric Loss Strategy for Multi-Task Learning (CVPR workshop 2019, 149 citations) |
No |
LibMTL |
|
| 🔵 To investigate |
IMTL-L |
Towards Impartial Multi-Task Learning (ICLR 2021, 279 citations) |
? |
official, LibMTL (maybe this is IMTL-G) |
Need more investigation |
| 🔵 To investigate |
FAMO |
FAMO: Fast Adaptive Multitask Optimization (NeurIPS 2023, 124 citations) |
? |
official, LibMTL |
Not sure this is a Scalarizer, need more investigation. |
| 🔵 To investigate |
GradNorm |
GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks (ICML 2018, 2334 citations) |
Yes, trainable state |
unofficial, LibMTL |
Not sure this is a Scalarizer, need more investigation. |
| 🔵 To investigate |
UW (UncertaintyWeighting) |
Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics (CVPR 2018, 5535 citations) |
? |
unofficial, LibMTL |
Not sure this is a Scalarizer, need more investigation. |
| 🔵 To investigate |
DWA (Dynamic Weight Averaging) |
End-to-End Multi-Task Learning with Attention (CVPR 2019, 1836 citations) |
? |
official, LibMTL |
Not sure this is a Scalarizer, need more investigation. |
This issue tracks the candidate methods that could maybe be implemented as
Scalarizerintorchjd.scalarization(see #666).LinearorConstant?RLWorRandom?