Performance Optimization for DDP sharded
🚀 Feature

Motivation

Experiments show that enabling buckets and compressing in FP16 before broadcasting improves performance for multi-nodes.
20% for 1B parameter model enabling buckets
5% for 0.6B parameter model 2 nodes compressing FP16 before broadcasting
Pitch

set smarter default for DDP sharded
If single-node: reduce_buffer_size = 0 (disable buckets)
If multi-nodes: reduce_buffer_size = 8M, if training use precision 16, broadcast_fp16 = True
Alternatives

Additional context