simcop2387/AutoGPTQ
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity

update another example

Browse source
This commit is contained in:
Forkoz 2023-05-17 07:27:49 -05:00 • committed by GitHub
parent 922ec02998
commit 2d0aaa423f
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
examples/quantization/basic_usage_wikitext2.py
View file

@ -137,6 +137,7 @@ def main():
quantize_config = BaseQuantizeConfig( quantize_config = BaseQuantizeConfig(
bits=4, # quantize model to 4-bit bits=4, # quantize model to 4-bit
group_size=128, # it is recommended to set the value to 128 group_size=128, # it is recommended to set the value to 128
desc_act=False, # desc_act and group size only works on triton
) )
# load un-quantized model, the model will always be force loaded into cpu # load un-quantized model, the model will always be force loaded into cpu