653 B
653 B
Examples
To run example scripts in this folder, one must first install auto_gptq as described in this
Basic Usage
Run the following code to execute basic_usage.py:
python basic_usage.py
Quantize with Alpaca
To Run this script, one also need to install datasets via pip install datasets.
Then Execute quant_with_alpaca.py using command like this:
CUDA_VISIBLE_DEVICES=0 python quant_with_alpaca.py --pretrained_model_dir "facebook/opt-125m"
The alpaca dataset used in here is a cleaned version provided by gururise in AlpacaDataCleaned