Code

Dataset Processing Args#1006

Code

Draft

kylesayrs wants to merge 56 commits intomainvllm-project/llm-compressor:mainfrom kylesayrs/tokenizer-kwargs-argumentvllm-project/llm-compressor:kylesayrs/tokenizer-kwargs-argumentCopy head branch name to clipboard

+57-19

Commits

Commits on Dec 2, 2024

• clean up CustomDataset

  Show description for a62617c
  kylesayrs

  committed
  a62617c

  Copy full SHA for a62617c

• chchchchanges

  Show description for 57b5e02
  kylesayrs

  committed
  57b5e02

  Copy full SHA for 57b5e02

• wip: use rename to processor, going through tests

  Show description for fa317fd
  kylesayrs

  committed
  fa317fd

  Copy full SHA for fa317fd

• remove labels from calibration dataset rather than assuming that all tokenized datasets should not be given labels

  Show description for f3f5875
  kylesayrs

  committed
  f3f5875

  Copy full SHA for f3f5875

• cleanup

  Show description for 58c3afe
  kylesayrs

  committed
  58c3afe

  Copy full SHA for 58c3afe

• cleanup, etc

  Show description for 72aecfc
  kylesayrs

  committed
  72aecfc

  Copy full SHA for 72aecfc

• Merge remote-tracking branch 'origin' into kylesayrs/cleanup-custom-dataset

  kylesayrs

  committed
  77217fb

  Copy full SHA for 77217fb

• fix typehinting

  Show description for 4461a3e
  kylesayrs

  committed
  4461a3e

  Copy full SHA for 4461a3e

• add typechecking imports

  Show description for fb33001
  kylesayrs

  committed
  fb33001

  Copy full SHA for fb33001

Commits on Dec 3, 2024

• remove sparseml utilities

  Show description for bf4744a
  kylesayrs

  committed
  bf4744a

  Copy full SHA for bf4744a

• Merge branch 'kylesayrs/remove-sparseml-utilities' into kylesayrs/cleanup-custom-dataset

  kylesayrs

  committed
  62ae31d

  Copy full SHA for 62ae31d

• use in model_load

  Show description for 7e516c1
  kylesayrs

  committed
  7e516c1

  Copy full SHA for 7e516c1

• remove use of RECIPE FILE NAME

  Show description for 9e33641
  kylesayrs

  committed
  9e33641

  Copy full SHA for 9e33641

• rename to RECIPE_FILE_NAME, avoid circular import

  Show description for 58c0fba
  kylesayrs

  committed
  58c0fba

  Copy full SHA for 58c0fba

• Merge branch 'kylesayrs/remove-sparseml-utilities' into kylesayrs/cleanup-custom-dataset

  kylesayrs

  committed
  b28aaae

  Copy full SHA for b28aaae

• image dataset collation

  kylesayrs

  committed
  8d13013

  Copy full SHA for 8d13013

• cleanup, do not handle case where processor is None

  Show description for 163ee8f
  kylesayrs

  committed
  163ee8f

  Copy full SHA for 163ee8f

• remove qa ignore

  Show description for 1180b34
  kylesayrs

  committed
  1180b34

  Copy full SHA for 1180b34

• Merge branch 'kylesayrs/remove-sparseml-utilities' into kylesayrs/cleanup-custom-dataset

  kylesayrs

  committed
  ad20ae7

  Copy full SHA for ad20ae7

• add documentation

  Show description for c431958
  kylesayrs

  committed
  c431958

  Copy full SHA for c431958

• add data collator arg

  Show description for b48d55d
  kylesayrs

  committed
  b48d55d

  Copy full SHA for b48d55d

• use default factor

  Show description for 0ed5c2c
  kylesayrs

  committed
  0ed5c2c

  Copy full SHA for 0ed5c2c

Commits on Dec 4, 2024

• validate flickr

  Show description for 4576712
  kylesayrs

  committed
  4576712

  Copy full SHA for 4576712

• discover bug, tests and multimodal working

  kylesayrs

  committed
  5276c58

  Copy full SHA for 5276c58

• dataset split fallbacks

  Show description for dffcbc3
  kylesayrs

  committed
  dffcbc3

  Copy full SHA for dffcbc3

• Merge branch 'kylesayrs/dataset-split-fallbacks' into kylesayrs/cleanup-custom-dataset

  kylesayrs

  committed
  779c9a2

  Copy full SHA for 779c9a2

• cleanup, depreciate remove_columns argument

  kylesayrs

  committed
  d061567

  Copy full SHA for d061567

Commits on Dec 5, 2024

• silently assign tokenizer to processor

  kylesayrs

  committed
  55a31ca

  Copy full SHA for 55a31ca

• replace tokenizer with processor

  Show description for 1aba16d
  kylesayrs

  committed
  1aba16d

  Copy full SHA for 1aba16d

• Merge branch 'kylesayrs/processor-replaces-tokenizer' into kylesayrs/cleanup-custom-dataset

  kylesayrs

  committed
  135e459

  Copy full SHA for 135e459

• typehinting, add not-implemented error

  Show description for bc505bf
  kylesayrs

  committed
  bc505bf

  Copy full SHA for bc505bf

• remove todos

  Show description for c91ba77
  kylesayrs

  committed
  c91ba77

  Copy full SHA for c91ba77

• update dataset manager api in tests

  Show description for 0a573a1
  kylesayrs

  committed
  0a573a1

  Copy full SHA for 0a573a1

• Delete examples/multimodal_vision/qwen_vl2.py

  kylesayrs

  authored
  acb1a18

  Copy full SHA for acb1a18

• Delete examples/multimodal_vision/mllama.py

  kylesayrs

  authored
  56b5d12

  Copy full SHA for 56b5d12

Commits on Dec 9, 2024

• Merge remote-tracking branch 'origin' into kylesayrs/cleanup-custom-dataset

  kylesayrs

  committed
  c1f5cb2

  Copy full SHA for c1f5cb2

Commits on Dec 17, 2024

• handle columns better

  Show description for 7667998
  kylesayrs

  committed
  7667998

  Copy full SHA for 7667998

Commits on Dec 18, 2024

• filter_tokenizer_args

  Show description for af86f45
  kylesayrs

  committed
  af86f45

  Copy full SHA for af86f45

• Merge remote-tracking branch 'origin' into kylesayrs/cleanup-custom-dataset

  kylesayrs

  committed
  0438e17

  Copy full SHA for 0438e17

• more tests

  Show description for f4fa9c3
  kylesayrs

  committed
  f4fa9c3

  Copy full SHA for f4fa9c3

• Merge remote-tracking branch 'origin' into kylesayrs/cleanup-custom-dataset

  kylesayrs

  committed
  6bd1721

  Copy full SHA for 6bd1721

• remove duplicate file

  Show description for e757e61
  kylesayrs

  committed
  e757e61

  Copy full SHA for e757e61

• better help texts

  Show description for bdfa3d4
  kylesayrs

  committed
  bdfa3d4

  Copy full SHA for bdfa3d4

Commits on Dec 19, 2024

• rvert data split fallbacks

  kylesayrs

  committed
  601cb0e

  Copy full SHA for 601cb0e

Commits on Dec 20, 2024

• handle non-fast tokenizers

  kylesayrs

  committed
  7f6e8cd

  Copy full SHA for 7f6e8cd

• address nits, add logging

  Show description for 3a9816c
  kylesayrs

  committed
  3a9816c

  Copy full SHA for 3a9816c

• add back copyrights

  kylesayrs

  committed
  7be0c88

  Copy full SHA for 7be0c88

• correctly update helptext

  Show description for bedbf8c
  kylesayrs

  committed
  bedbf8c

  Copy full SHA for bedbf8c

• Merge remote-tracking branch 'origin' into kylesayrs/cleanup-custom-dataset

  kylesayrs

  committed
  7c54bed

  Copy full SHA for 7c54bed

• use rename_columns and processor_kwargs args

  Show description for f2b3c99
  kylesayrs

  committed
  f2b3c99

  Copy full SHA for f2b3c99

• ensure prompt key

  Show description for 5b4ee22
  kylesayrs

  committed
  5b4ee22

  Copy full SHA for 5b4ee22

• syntax

  Show description for bbe3eb6
  kylesayrs

  committed
  bbe3eb6

  Copy full SHA for bbe3eb6

Commits on Dec 27, 2024

• Merge remote-tracking branch 'origin' into kylesayrs/tokenizer-kwargs-argument

  kylesayrs

  committed
  6f2a246

  Copy full SHA for 6f2a246

• account for models which improperly do not override the abstract methods

  Show description for 148e617
  kylesayrs

  committed
  148e617

  Copy full SHA for 148e617

• Merge branch 'kylesayrs/patch-mal-models' into kylesayrs/tokenizer-kwargs-argument

  kylesayrs

  committed
  ca9757d

  Copy full SHA for ca9757d

Commits on Jan 25, 2025

• Merge branch 'main' into kylesayrs/tokenizer-kwargs-argument

  kylesayrs

  authored
  ddc7d54

  Copy full SHA for ddc7d54