FP8 quantization

Memory Requirements for Llama 3.1-405BRunning Llama 3.1-405B requires substantial memory and computational resources:GPU Memory: The 405B model can utilize as much as 80GB of GPU memory per A100 GPU for efficient inference. Using Tensor...