85 commits

Author	SHA1	Message	Date
oobabooga	bae1aa34aa	Fix loading Llama-3_3-Nemotron-Super-49B-v1 and similar models (closes #7012)	2025-05-25 17:19:26 -07:00
oobabooga	5d00574a56	Minor UI fixes	2025-05-20 16:20:49 -07:00
oobabooga	9ec46b8c44	Remove the HQQ loader (HQQ models can be loaded through Transformers)	2025-05-19 09:23:24 -07:00
oobabooga	61276f6a37	Merge remote-tracking branch 'refs/remotes/origin/dev' into dev	2025-05-17 07:22:51 -07:00
oobabooga	4800d1d522	More robust VRAM calculation	2025-05-17 07:20:38 -07:00
mamei16	052c82b664	Fix KeyError: 'gpu_layers' when loading existing model settings (#6991)	2025-05-17 11:19:13 -03:00
oobabooga	0f77ff9670	UI: Use total VRAM (not free) for layers calculation when a model is loaded	2025-05-16 19:19:22 -07:00
oobabooga	71fa046c17	Minor changes after 1c549d176b	2025-05-16 17:38:08 -07:00
oobabooga	d99fb0a22a	Add backward compatibility with saved n_gpu_layers values	2025-05-16 17:29:18 -07:00
oobabooga	1c549d176b	Fix GPU layers slider: honor saved settings and show true maximum	2025-05-16 17:26:13 -07:00
oobabooga	38c50087fe	Prevent a crash on systems without an NVIDIA GPU	2025-05-16 11:55:30 -07:00
oobabooga	253e85a519	Only compute VRAM/GPU layers for llama.cpp models	2025-05-16 10:02:30 -07:00
oobabooga	ee7b3028ac	Always cache GGUF metadata calls	2025-05-16 09:12:36 -07:00
oobabooga	4925c307cf	Auto-adjust GPU layers on context size and cache type changes + many fixes	2025-05-16 09:07:38 -07:00
oobabooga	5534d01da0	Estimate the VRAM for GGUF models + autoset gpu-layers (#6980)	2025-05-16 00:07:37 -03:00
oobabooga	3fa1a899ae	UI: Fix gpu-layers being ignored (closes #6973)	2025-05-13 12:07:59 -07:00
oobabooga	d9de14d1f7	Restructure the repository (#6904)	2025-04-26 08:56:54 -03:00
oobabooga	d4b1e31c49	Use --ctx-size to specify the context size for all loaders ... Old flags are still recognized as alternatives.	2025-04-25 16:59:03 -07:00
oobabooga	78aeabca89	Fix the transformers loader	2025-04-21 18:33:14 -07:00
oobabooga	ae02ffc605	Refactor the transformers loader (#6859)	2025-04-20 13:33:47 -03:00
oobabooga	ae54d8faaa	New llama.cpp loader (#6846)	2025-04-18 09:59:37 -03:00
Googolplexed	d78abe480b	Allow for model subfolder organization for GGUF files (#6686) ... --------- Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>	2025-04-18 02:53:59 -03:00
oobabooga	2c2d453c8c	Revert "Use ExLlamaV2 (instead of the HF one) for EXL2 models for now" ... This reverts commit 0ef1b8f8b4.	2025-04-17 21:31:32 -07:00
oobabooga	0ef1b8f8b4	Use ExLlamaV2 (instead of the HF one) for EXL2 models for now ... It doesn't seem to have the "OverflowError" bug	2025-04-17 05:47:40 -07:00
oobabooga	682c78ea42	Add back detection of GPTQ models (closes #6841)	2025-04-11 21:00:42 -07:00
oobabooga	8b8d39ec4e	Add ExLlamaV3 support (#6832)	2025-04-09 00:07:08 -03:00
oobabooga	a5855c345c	Set context lengths to at most 8192 by default (to prevent out of memory errors) (#6835)	2025-04-07 21:42:33 -03:00
oobabooga	7157257c3f	Remove the AutoGPTQ loader (#6641)	2025-01-08 19:28:56 -03:00
oobabooga	e6181e834a	Remove AutoAWQ as a standalone loader ... (it works better through transformers)	2024-07-23 15:31:17 -07:00
oobabooga	907137a13d	Automatically set bf16 & use_eager_attention for Gemma-2	2024-07-01 21:46:35 -07:00
mefich	a85749dcbe	Update models_settings.py: add default alpha_value, add proper compress_pos_emb for newer GGUFs (#6111)	2024-06-26 22:17:56 -03:00
oobabooga	577a8cd3ee	Add TensorRT-LLM support (#5715)	2024-06-24 02:30:03 -03:00
Forkoz	1576227f16	Fix GGUFs with no BOS token present, mainly qwen2 models. (#6119) ... --------- Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>	2024-06-14 13:51:01 -03:00
oobabooga	2d196ed2fe	Remove obsolete pre_layer parameter	2024-06-12 18:56:44 -07:00
oobabooga	9e189947d1	Minor fix after bd7cc4234d (thanks @belladoreai)	2024-05-21 10:37:30 -07:00
oobabooga	bd7cc4234d	Backend cleanup (#6025)	2024-05-21 13:32:02 -03:00
oobabooga	a38a37b3b3	llama.cpp: default n_gpu_layers to the maximum value for the model automatically	2024-05-19 10:57:42 -07:00
oobabooga	64e2a9a0a7	Fix the Phi-3 template when used in the UI	2024-04-24 01:34:11 -07:00
oobabooga	f27e1ba302	Add a /v1/internal/chat-prompt endpoint (#5879)	2024-04-19 00:24:46 -03:00
oobabooga	17c4319e2d	Fix loading command-r context length metadata	2024-04-10 21:39:59 -07:00
oobabooga	64a76856bd	Metadata: Fix loading Command R+ template with multiple options	2024-04-06 07:32:17 -07:00
oobabooga	d423021a48	Remove CTransformers support (#5807)	2024-04-04 20:23:58 -03:00
oobabooga	9ab7365b56	Read rope_theta for DBRX model (thanks turboderp)	2024-04-01 20:25:31 -07:00
oobabooga	db5f6cd1d8	Fix ExLlamaV2 loaders using unnecessary "bits" metadata	2024-03-30 21:51:39 -07:00
oobabooga	624faa1438	Fix ExLlamaV2 context length setting (closes #5750)	2024-03-30 21:33:16 -07:00
oobabooga	4039999be5	Autodetect llamacpp_HF loader when tokenizer exists	2024-02-16 09:29:26 -08:00
oobabooga	76d28eaa9e	Add a menu for customizing the instruction template for the model (#5521)	2024-02-16 14:21:17 -03:00
oobabooga	2a1063eff5	Revert "Remove non-HF ExLlamaV2 loader (#5431)" ... This reverts commit cde000d478.	2024-02-06 06:21:36 -08:00
oobabooga	cde000d478	Remove non-HF ExLlamaV2 loader (#5431)	2024-02-04 01:15:51 -03:00
oobabooga	2734ce3e4c	Remove RWKV loader (#5130)	2023-12-31 02:01:40 -03:00