[Code Refactor] Large Updates Since Aug (#293) · Luodian/Otter@1c056f1

Commit

[Code Refactor] Large Updates Since Aug (#293)

* update black format

* update keep symbol logic and revert changes made to self.images

* update

* update mean/var for idefics model

* update

* stash changes to merge

* Revert "Merge branch 'dev/idefics' of https://github.com/Luodian/Otter-2 into dev/idefics"

This reverts commit 5947cfb998c3d6b7752fae01718b3ed06a89fcd3, reversing
changes made to 8caad9c9afd08f63ddefc8a9a509bc83bdc92024.

* update

* update

* add more images to test

* update flamingo's logic to handle freeze and unfreeze params

* add to flamingo training

* update

* Conflicts:
	pipeline/train/instruction_following.py
	src/otter_ai/models/flamingo/modeling_flamingo.py

* add handling logic for text-only data

* update populate args

* update llama2 model

* add task description via data recipe yaml

* black lint update

* update with black lint

* fix bug

* update special tokens and lm_head

* add tokenizer saving behavior

* update

* add controller for resize embeddnig

* update

* change codellama prompt strategy

* make mosaicgpt work

* hardcode for sort inconsistent data

* update

* blank lint

* refactor preprocessing logic

* update

* clean

* fix some bugs

* update with more evals

* masking for single

* masking media_token_id

* fix

* fix missing eos

* add git ignore for config

* new_masking

* new_masking

* new_masking

* update config

* clean_mask_functions

* put new scripts and data yaml

* fix masking issues for idefics

* black lint

* change_labels_init

* change_labels_init

* allow multiple task descriptions

* lint

* black lint

* typo fix

* prevalidation in dataloading

* fix slurm config

* fix slurm config

* update

* updates

* black lint

* explicitly assign constant scheduler

* eval update

* black lint

* benchmark eval updates

* update

* update logging with wandb

* update wandb logging

* [Eval] MMVet Eval (#18)

* prepare dataset

* MME

* add mmvet dataset

* git ignore

* MMVet

* lint

* Squashed commit of the following:

commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
Author: Li Bo <[email protected]>
Date:   Mon Oct 9 02:02:10 2023 +0000

    update wandb logging

commit 2d29a417d9e96251be33edda0b0637647882c31b
Author: Li Bo <[email protected]>
Date:   Mon Oct 9 01:45:05 2023 +0000

    update logging with wandb

commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
Author: Li Bo <[email protected]>
Date:   Mon Oct 9 01:35:25 2023 +0000

    update

commit b8312afc54915b615fd8319bcffc9e0781af346a
Author: Li Bo <[email protected]>
Date:   Sun Oct 8 14:41:23 2023 +0000

    benchmark eval updates

commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
Author: Li Bo <[email protected]>
Date:   Sun Oct 8 13:28:24 2023 +0000

    black lint

commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
Author: Li Bo <[email protected]>
Date:   Sun Oct 8 13:27:22 2023 +0000

    eval update

commit cff3f32262bcdba980f84e82b874aa03c05c9027
Author: Bo Li <[email protected]>
Date:   Sun Oct 8 18:44:22 2023 +0800

    explicitly assign constant scheduler

* Squashed commit of the following:

commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
Author: Li Bo <[email protected]>
Date:   Mon Oct 9 02:02:10 2023 +0000

    update wandb logging

commit 2d29a417d9e96251be33edda0b0637647882c31b
Author: Li Bo <[email protected]>
Date:   Mon Oct 9 01:45:05 2023 +0000

    update logging with wandb

commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
Author: Li Bo <[email protected]>
Date:   Mon Oct 9 01:35:25 2023 +0000

    update

commit b8312afc54915b615fd8319bcffc9e0781af346a
Author: Li Bo <[email protected]>
Date:   Sun Oct 8 14:41:23 2023 +0000

    benchmark eval updates

commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
Author: Li Bo <[email protected]>
Date:   Sun Oct 8 13:28:24 2023 +0000

    black lint

commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
Author: Li Bo <[email protected]>
Date:   Sun Oct 8 13:27:22 2023 +0000

    eval update

commit cff3f32262bcdba980f84e82b874aa03c05c9027
Author: Bo Li <[email protected]>
Date:   Sun Oct 8 18:44:22 2023 +0800

    explicitly assign constant scheduler

* Squashed commit of the following:

commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
Author: Li Bo <[email protected]>
Date:   Mon Oct 9 02:02:10 2023 +0000

    update wandb logging

commit 2d29a417d9e96251be33edda0b0637647882c31b
Author: Li Bo <[email protected]>
Date:   Mon Oct 9 01:45:05 2023 +0000

    update logging with wandb

commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
Author: Li Bo <[email protected]>
Date:   Mon Oct 9 01:35:25 2023 +0000

    update

commit b8312afc54915b615fd8319bcffc9e0781af346a
Author: Li Bo <[email protected]>
Date:   Sun Oct 8 14:41:23 2023 +0000

    benchmark eval updates

commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
Author: Li Bo <[email protected]>
Date:   Sun Oct 8 13:28:24 2023 +0000

    black lint

commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
Author: Li Bo <[email protected]>
Date:   Sun Oct 8 13:27:22 2023 +0000

    eval update

commit cff3f32262bcdba980f84e82b874aa03c05c9027
Author: Bo Li <[email protected]>
Date:   Sun Oct 8 18:44:22 2023 +0800

    explicitly assign constant scheduler

---------

Co-authored-by: Li Bo <[email protected]>

* seedbench_eval (#19)

* seedbench_eval

* update seedbench eval for idefics

* update

---------

Co-authored-by: Li Bo <[email protected]>

* Dev/refactor (#20)

* seedbench_eval

* update seedbench eval for idefics

* update

* resolve input for seedbench

* update

---------

Co-authored-by: Joshua Adrian Cahyono <[email protected]>

* resolve idefics for seedbench eval

* prepare

* fix mme

* update mme saving logic

* Dev/mathvista (#21)

* for mathvista

* add time info

* black lint code

* update

* update test config

* stage updates

* update mathvista eval

* update mathvista

* better format prompt

* fix bug in mme eval

* update naming rule

* update output path

* Dev/train (#22)

* Merge commit '2d29a417d9e96251be33edda0b0637647882c31b'

* Conflicts:
	pipeline/mimicit_utils/mimicit_dataset.py

* update mpt config

* fix training padding

* update fix

* Squashed commit of the following:

commit e4f1ec3b287c026a6212a88316d7341883195294
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 13:47:47 2023 +0000

    update output path

commit 1a8e12bfa21697188eaa7a4ed6169223e0f92258
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 12:23:02 2023 +0000

    update naming rule

commit 537d239e3b4109c552a94b80a7bde3bb8a7b107d
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 09:10:47 2023 +0000

    fix bug in mme eval

commit 0c7eacb6ebef4d55187d67b06c88cd504ef17dbd
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 14:36:55 2023 +0800

    Dev/mathvista (#21)

    * for mathvista

    * add time info

    * black lint code

    * update

    * update test config

    * stage updates

    * update mathvista eval

    * update mathvista

    * better format prompt

commit c07f73f6c868444fd784e5844134dc0cccaa566d
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 06:33:08 2023 +0000

    update mme saving logic

commit dce1b0f30a390ace45bcf579d93764847848ee51
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 06:28:29 2023 +0000

    fix mme

commit b970f617bea2b418cf726a50b4cd008dc7eb4907
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 06:11:03 2023 +0000

    prepare

commit 33e6cb0f0212ed845ca17956743926b4d53ec0f3
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 06:09:57 2023 +0000

    resolve idefics for seedbench eval

commit fc2a87bc853a34ce8bfb16b7f6a504a1d63af0e5
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 12:19:06 2023 +0800

    Dev/refactor (#20)

    * seedbench_eval

    * update seedbench eval for idefics

    * update

    * resolve input for seedbench

    * update

    ---------

    Co-authored-by: Joshua Adrian Cahyono <[email protected]>

commit edb2440eec77c6c07ed5457a1251e567e57b88f2
Author: JvThunder <[email protected]>
Date:   Tue Oct 10 02:42:27 2023 +0800

    seedbench_eval (#19)

    * seedbench_eval

    * update seedbench eval for idefics

    * update

    ---------

    Co-authored-by: Li Bo <[email protected]>

commit 39264fc8f0781057de820c122f31fc60ac2f301f
Author: Pu Fanyi <[email protected]>
Date:   Mon Oct 9 20:08:29 2023 +0800

    [Eval] MMVet Eval (#18)

    * prepare dataset

    * MME

    * add mmvet dataset

    * git ignore

    * MMVet

    * lint

    * Squashed commit of the following:

    commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 02:02:10 2023 +0000

        update wandb logging

    commit 2d29a417d9e96251be33edda0b0637647882c31b
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 01:45:05 2023 +0000

        update logging with wandb

    commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 01:35:25 2023 +0000

        update

    commit b8312afc54915b615fd8319bcffc9e0781af346a
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 14:41:23 2023 +0000

        benchmark eval updates

    commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 13:28:24 2023 +0000

        black lint

    commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 13:27:22 2023 +0000

        eval update

    commit cff3f32262bcdba980f84e82b874aa03c05c9027
    Author: Bo Li <[email protected]>
    Date:   Sun Oct 8 18:44:22 2023 +0800

        explicitly assign constant scheduler

    * Squashed commit of the following:

    commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 02:02:10 2023 +0000

        update wandb logging

    commit 2d29a417d9e96251be33edda0b0637647882c31b
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 01:45:05 2023 +0000

        update logging with wandb

    commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 01:35:25 2023 +0000

        update

    commit b8312afc54915b615fd8319bcffc9e0781af346a
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 14:41:23 2023 +0000

        benchmark eval updates

    commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 13:28:24 2023 +0000

        black lint

    commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 13:27:22 2023 +0000

        eval update

    commit cff3f32262bcdba980f84e82b874aa03c05c9027
    Author: Bo Li <[email protected]>
    Date:   Sun Oct 8 18:44:22 2023 +0800

        explicitly assign constant scheduler

    * Squashed commit of the following:

    commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 02:02:10 2023 +0000

        update wandb logging

    commit 2d29a417d9e96251be33edda0b0637647882c31b
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 01:45:05 2023 +0000

        update logging with wandb

    commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 01:35:25 2023 +0000

        update

    commit b8312afc54915b615fd8319bcffc9e0781af346a
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 14:41:23 2023 +0000

        benchmark eval updates

    commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 13:28:24 2023 +0000

        black lint

    commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 13:27:22 2023 +0000

        eval update

    commit cff3f32262bcdba980f84e82b874aa03c05c9027
    Author: Bo Li <[email protected]>
    Date:   Sun Oct 8 18:44:22 2023 +0800

        explicitly assign constant scheduler

    ---------

    Co-authored-by: Li Bo <[email protected]>

commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
Author: Li Bo <[email protected]>
Date:   Mon Oct 9 02:02:10 2023 +0000

    update wandb logging

commit 2d29a417d9e96251be33edda0b0637647882c31b
Author: Li Bo <[email protected]>
Date:   Mon Oct 9 01:45:05 2023 +0000

    update logging with wandb

commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
Author: Li Bo <[email protected]>
Date:   Mon Oct 9 01:35:25 2023 +0000

    update

commit b8312afc54915b615fd8319bcffc9e0781af346a
Author: Li Bo <[email protected]>
Date:   Sun Oct 8 14:41:23 2023 +0000

    benchmark eval updates

commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
Author: Li Bo <[email protected]>
Date:   Sun Oct 8 13:28:24 2023 +0000

    black lint

commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
Author: Li Bo <[email protected]>
Date:   Sun Oct 8 13:27:22 2023 +0000

    eval update

* fix video training

* also train vision encoder

* idefics full finetune

* fix images input for idefics model

* [evaluation] Fix mmvet output (#23)

* fix mmvet

* lint

* hardcode enabling flash

* better design for idefics model to integrate multi-types data

* black lint

* generate results.json for seedbench (#24)

* mathvista

* update configs and demo evals

* migrate

* Dev/refactor (#26)

* seedbench_eval

* update seedbench eval for idefics

* update

* resolve input for seedbench

* update

* make_submission_seedbench

* pope_eval

* update_pope_eval

* update_pope

* black linta

---------

Co-authored-by: Li Bo <[email protected]>

* idefics bath gen

* [Eval] (#27)

* Merge commit '2d29a417d9e96251be33edda0b0637647882c31b'

* Conflicts:
	pipeline/mimicit_utils/mimicit_dataset.py

* update mpt config

* fix training padding

* update fix

* Squashed commit of the following:

commit e4f1ec3b287c026a6212a88316d7341883195294
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 13:47:47 2023 +0000

    update output path

commit 1a8e12bfa21697188eaa7a4ed6169223e0f92258
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 12:23:02 2023 +0000

    update naming rule

commit 537d239e3b4109c552a94b80a7bde3bb8a7b107d
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 09:10:47 2023 +0000

    fix bug in mme eval

commit 0c7eacb6ebef4d55187d67b06c88cd504ef17dbd
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 14:36:55 2023 +0800

    Dev/mathvista (#21)

    * for mathvista

    * add time info

    * black lint code

    * update

    * update test config

    * stage updates

    * update mathvista eval

    * update mathvista

    * better format prompt

commit c07f73f6c868444fd784e5844134dc0cccaa566d
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 06:33:08 2023 +0000

    update mme saving logic

commit dce1b0f30a390ace45bcf579d93764847848ee51
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 06:28:29 2023 +0000

    fix mme

commit b970f617bea2b418cf726a50b4cd008dc7eb4907
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 06:11:03 2023 +0000

    prepare

commit 33e6cb0f0212ed845ca17956743926b4d53ec0f3
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 06:09:57 2023 +0000

    resolve idefics for seedbench eval

commit fc2a87bc853a34ce8bfb16b7f6a504a1d63af0e5
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 12:19:06 2023 +0800

    Dev/refactor (#20)

    * seedbench_eval

    * update seedbench eval for idefics

    * update

    * resolve input for seedbench

    * update

    ---------

    Co-authored-by: Joshua Adrian Cahyono <[email protected]>

commit edb2440eec77c6c07ed5457a1251e567e57b88f2
Author: JvThunder <[email protected]>
Date:   Tue Oct 10 02:42:27 2023 +0800

    seedbench_eval (#19)

    * seedbench_eval

    * update seedbench eval for idefics

    * update

    ---------

    Co-authored-by: Li Bo <[email protected]>

commit 39264fc8f0781057de820c122f31fc60ac2f301f
Author: Pu Fanyi <[email protected]>
Date:   Mon Oct 9 20:08:29 2023 +0800

    [Eval] MMVet Eval (#18)

    * prepare dataset

    * MME

    * add mmvet dataset

    * git ignore

    * MMVet

    * lint

    * Squashed commit of the following:

    commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 02:02:10 2023 +0000

        update wandb logging

    commit 2d29a417d9e96251be33edda0b0637647882c31b
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 01:45:05 2023 +0000

        update logging with wandb

    commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 01:35:25 2023 +0000

        update

    commit b8312afc54915b615fd8319bcffc9e0781af346a
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 14:41:23 2023 +0000

        benchmark eval updates

    commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 13:28:24 2023 +0000

        black lint

    commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 13:27:22 2023 +0000

        eval update

    commit cff3f32262bcdba980f84e82b874aa03c05c9027
    Author: Bo Li <[email protected]>
    Date:   Sun Oct 8 18:44:22 2023 +0800

        explicitly assign constant scheduler

    * Squashed commit of the following:

    commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 02:02:10 2023 +0000

        update wandb logging

    commit 2d29a417d9e96251be33edda0b0637647882c31b
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 01:45:05 2023 +0000

        update logging with wandb

    commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 01:35:25 2023 +0000

        update

    commit b8312afc54915b615fd8319bcffc9e0781af346a
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 14:41:23 2023 +0000

        benchmark eval updates

    commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 13:28:24 2023 +0000

        black lint

    commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 13:27:22 2023 +0000

        eval update

    commit cff3f32262bcdba980f84e82b874aa03c05c9027
    Author: Bo Li <[email protected]>
    Date:   Sun Oct 8 18:44:22 2023 +0800

        explicitly assign constant scheduler

    * Squashed commit of the following:

    commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 02:02:10 2023 +0000

        update wandb logging

    commit 2d29a417d9e96251be33edda0b0637647882c31b
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 01:45:05 2023 +0000

        update logging with wandb

    commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 01:35:25 2023 +0000

        update

    commit b8312afc54915b615fd8319bcffc9e0781af346a
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 14:41:23 2023 +0000

        benchmark eval updates

    commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 13:28:24 2023 +0000

        black lint

    commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 13:27:22 2023 +0000

        eval update

    commit cff3f32262bcdba980f84e82b874aa03c05c9027
    Author: Bo Li <[email protected]>
    Date:   Sun Oct 8 18:44:22 2023 +0800

        explicitly assign constant scheduler

    ---------

    Co-authored-by: Li Bo <[email protected]>

commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
Author: Li Bo <[email protected]>
Date:   Mon Oct 9 02:02:10 2023 +0000

    update wandb logging

commit 2d29a417d9e96251be33edda0b0637647882c31b
Author: Li Bo <[email protected]>
Date:   Mon Oct 9 01:45:05 2023 +0000

    update logging with wandb

commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
Author: Li Bo <[email protected]>
Date:   Mon Oct 9 01:35:25 2023 +0000

    update

commit b8312afc54915b615fd8319bcffc9e0781af346a
Author: Li Bo <[email protected]>
Date:   Sun Oct 8 14:41:23 2023 +0000

    benchmark eval updates

commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
Author: Li Bo <[email protected]>
Date:   Sun Oct 8 13:28:24 2023 +0000

    black lint

commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
Author: Li Bo <[email protected]>
Date:   Sun Oct 8 13:27:22 2023 +0000

    eval update

* migrates

* Squashed commit of the following:

commit 52bacca3d858d2ac32f44f8b1619ce5064ae1a28
Author: Li Bo <[email protected]>
Date:   Thu Oct 12 10:44:46 2023 +0000

    update configs and demo evals

commit a41247d193b3ca0967c38d0b3b11e2a04ebf9ca7
Merge: 9ff5957 34b70f8
Author: Li Bo <[email protected]>
Date:   Thu Oct 12 10:02:24 2023 +0000

    Merge branch 'main_dev' of https://github.com/Luodian/Otter-2 into main_dev

commit 9ff5957c01349e58b60154f469ae3ee6c8c30555
Author: Li Bo <[email protected]>
Date:   Thu Oct 12 08:52:22 2023 +0000

    mathvista

commit 34b70f8068cfd5fbdc36980e6ac5215942849723
Author: Pu Fanyi <[email protected]>
Date:   Thu Oct 12 15:56:08 2023 +0800

    generate results.json for seedbench (#24)

commit 19fdf40a3f0cd05551f482f9834e38f9210256a3
Author: Li Bo <[email protected]>
Date:   Thu Oct 12 02:04:38 2023 +0000

    black lint

commit 01d56cec48f65307ed33e2a2dbd853717464e51f
Author: Li Bo <[email protected]>
Date:   Thu Oct 12 02:02:52 2023 +0000

    better design for idefics model to integrate multi-types data

commit 1a0972e6cc283af4099f0c85d907c3ec61a36224
Author: Li Bo <[email protected]>
Date:   Wed Oct 11 18:28:19 2023 +0000

    hardcode enabling flash

commit cf11867f2bb1a7c50c863eb2b438a588acf4f733
Author: Pu Fanyi <[email protected]>
Date:   Thu Oct 12 00:55:23 2023 +0800

    [evaluation] Fix mmvet output (#23)

    * fix mmvet

    * lint

commit 54362e9de5c0ead2d622544dea8802fd62db19ce
Author: Li Bo <[email protected]>
Date:   Wed Oct 11 07:47:32 2023 +0000

    fix images input for idefics model

commit 96e467c0480cca1abd86ac723f8b33260c49b1a1
Author: Bo Li <[email protected]>
Date:   Wed Oct 11 15:20:04 2023 +0800

    idefics full finetune

commit 11be13deb2c5e57b392d289670237714cdb5b012
Author: Bo Li <[email protected]>
Date:   Wed Oct 11 13:22:32 2023 +0800

    also train vision encoder

commit 3dfcf3e9a468897d1a251163d13a4a5de7542a64
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 14:13:40 2023 +0000

    fix video training

commit b8a086bcbe43e5debd85fab2db11610df513558f
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 21:53:29 2023 +0800

    Dev/train (#22)

    * Merge commit '2d29a417d9e96251be33edda0b0637647882c31b'

    * Conflicts:
    	pipeline/mimicit_utils/mimicit_dataset.py

    * update mpt config

    * fix training padding

    * update fix

    * Squashed commit of the following:

    commit e4f1ec3b287c026a6212a88316d7341883195294
    Author: Li Bo <[email protected]>
    Date:   Tue Oct 10 13:47:47 2023 +0000

        update output path

    commit 1a8e12bfa21697188eaa7a4ed6169223e0f92258
    Author: Li Bo <[email protected]>
    Date:   Tue Oct 10 12:23:02 2023 +0000

        update naming rule

    commit 537d239e3b4109c552a94b80a7bde3bb8a7b107d
    Author: Li Bo <[email protected]>
    Date:   Tue Oct 10 09:10:47 2023 +0000

        fix bug in mme eval

    commit 0c7eacb6ebef4d55187d67b06c88cd504ef17dbd
    Author: Li Bo <[email protected]>
    Date:   Tue Oct 10 14:36:55 2023 +0800

        Dev/mathvista (#21)

        * for mathvista

        * add time info

        * black lint code

        * update

        * update test config

        * stage updates

        * update mathvista eval

        * update mathvista

        * better format prompt

    commit c07f73f6c868444fd784e5844134dc0cccaa566d
    Author: Li Bo <[email protected]>
    Date:   Tue Oct 10 06:33:08 2023 +0000

        update mme saving logic

    commit dce1b0f30a390ace45bcf579d93764847848ee51
    Author: Li Bo <[email protected]>
    Date:   Tue Oct 10 06:28:29 2023 +0000

        fix mme

    commit b970f617bea2b418cf726a50b4cd008dc7eb4907
    Author: Li Bo <[email protected]>
    Date:   Tue Oct 10 06:11:03 2023 +0000

        prepare

    commit 33e6cb0f0212ed845ca17956743926b4d53ec0f3
    Author: Li Bo <[email protected]>
    Date:   Tue Oct 10 06:09:57 2023 +0000

        resolve idefics for seedbench eval

    commit fc2a87bc853a34ce8bfb16b7f6a504a1d63af0e5
    Author: Li Bo <[email protected]>
    Date:   Tue Oct 10 12:19:06 2023 +0800

        Dev/refactor (#20)

        * seedbench_eval

        * update seedbench eval for idefics

        * update

        * resolve input for seedbench

        * update

        ---------

        Co-authored-by: Joshua Adrian Cahyono <[email protected]>

    commit edb2440eec77c6c07ed5457a1251e567e57b88f2
    Author: JvThunder <[email protected]>
    Date:   Tue Oct 10 02:42:27 2023 +0800

        seedbench_eval (#19)

        * seedbench_eval

        * update seedbench eval for idefics

        * update

        ---------

        Co-authored-by: Li Bo <[email protected]>

    commit 39264fc8f0781057de820c122f31fc60ac2f301f
    Author: Pu Fanyi <[email protected]>
    Date:   Mon Oct 9 20:08:29 2023 +0800

        [Eval] MMVet Eval (#18)

        * prepare dataset

        * MME

        * add mmvet dataset

        * git ignore

        * MMVet

        * lint

        * Squashed commit of the following:

        commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
        Author: Li Bo <[email protected]>
        Date:   Mon Oct 9 02:02:10 2023 +0000

            update wandb logging

        commit 2d29a417d9e96251be33edda0b0637647882c31b
        Author: Li Bo <[email protected]>
        Date:   Mon Oct 9 01:45:05 2023 +0000

            update logging with wandb

        commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
        Author: Li Bo <[email protected]>
        Date:   Mon Oct 9 01:35:25 2023 +0000

            update

        commit b8312afc54915b615fd8319bcffc9e0781af346a
        Author: Li Bo <[email protected]>
        Date:   Sun Oct 8 14:41:23 2023 +0000

            benchmark eval updates

        commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
        Author: Li Bo <[email protected]>
        Date:   Sun Oct 8 13:28:24 2023 +0000

            black lint

        commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
        Author: Li Bo <[email protected]>
        Date:   Sun Oct 8 13:27:22 2023 +0000

            eval update

        commit cff3f32262bcdba980f84e82b874aa03c05c9027
        Author: Bo Li <[email protected]>
        Date:   Sun Oct 8 18:44:22 2023 +0800

            explicitly assign constant scheduler

        * Squashed commit of the following:

        commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
        Author: Li Bo <[email protected]>
        Date:   Mon Oct 9 02:02:10 2023 +0000

            update wandb logging

        commit 2d29a417d9e96251be33edda0b0637647882c31b
        Author: Li Bo <[email protected]>
        Date:   Mon Oct 9 01:45:05 2023 +0000

            update logging with wandb

        commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
        Author: Li Bo <[email protected]>
        Date:   Mon Oct 9 01:35:25 2023 +0000

            update

        commit b8312afc54915b615fd8319bcffc9e0781af346a
        Author: Li Bo <[email protected]>
        Date:   Sun Oct 8 14:41:23 2023 +0000

            benchmark eval updates

        commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
        Author: Li Bo <[email protected]>
        Date:   Sun Oct 8 13:28:24 2023 +0000

            black lint

        commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
        Author: Li Bo <[email protected]>
        Date:   Sun Oct 8 13:27:22 2023 +0000

            eval update

        commit cff3f32262bcdba980f84e82b874aa03c05c9027
        Author: Bo Li <[email protected]>
        Date:   Sun Oct 8 18:44:22 2023 +0800

            explicitly assign constant scheduler

        * Squashed commit of the following:

        commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
        Author: Li Bo <[email protected]>
        Date:   Mon Oct 9 02:02:10 2023 +0000

            update wandb logging

        commit 2d29a417d9e96251be33edda0b0637647882c31b
        Author: Li Bo <[email protected]>
        Date:   Mon Oct 9 01:45:05 2023 +0000

            update logging with wandb

        commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
        Author: Li Bo <[email protected]>
        Date:   Mon Oct 9 01:35:25 2023 +0000

            update

        commit b8312afc54915b615fd8319bcffc9e0781af346a
        Author: Li Bo <[email protected]>
        Date:   Sun Oct 8 14:41:23 2023 +0000

            benchmark eval updates

        commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
        Author: Li Bo <[email protected]>
        Date:   Sun Oct 8 13:28:24 2023 +0000

            black lint

        commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
        Author: Li Bo <[email protected]>
        Date:   Sun Oct 8 13:27:22 2023 +0000

            eval update

        commit cff3f32262bcdba980f84e82b874aa03c05c9027
        Author: Bo Li <[email protected]>
        Date:   Sun Oct 8 18:44:22 2023 +0800

            explicitly assign constant scheduler

        ---------

        Co-authored-by: Li Bo <[email protected]>

    commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 02:02:10 2023 +0000

        update wandb logging

    commit 2d29a417d9e96251be33edda0b0637647882c31b
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 01:45:05 2023 +0000

        update logging with wandb

    commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 01:35:25 2023 +0000

        update

    commit b8312afc54915b615fd8319bcffc9e0781af346a
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 14:41:23 2023 +0000

        benchmark eval updates

    commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 13:28:24 2023 +0000

        black lint

    commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 13:27:22 2023 +0000

        eval update

* update

* Squashed commit of the following:

commit 94177061ff29e8c395ae63d408f64297cfa09211
Author: JvThunder <[email protected]>
Date:   Fri Oct 13 11:42:56 2023 +0800

    Dev/refactor (#26)

    * seedbench_eval

    * update seedbench eval for idefics

    * update

    * resolve input for seedbench

    * update

    * make_submission_seedbench

    * pope_eval

    * update_pope_eval

    * update_pope

    * black linta

    ---------

    Co-authored-by: Li Bo <[email protected]>

commit 52bacca3d858d2ac32f44f8b1619ce5064ae1a28
Author: Li Bo <[email protected]>
Date:   Thu Oct 12 10:44:46 2023 +0000

    update configs and demo evals

commit a41247d193b3ca0967c38d0b3b11e2a04ebf9ca7
Merge: 9ff5957 34b70f8
Author: Li Bo <[email protected]>
Date:   Thu Oct 12 10:02:24 2023 +0000

    Merge branch 'main_dev' of https://github.com/Luodian/Otter-2 into main_dev

commit 9ff5957c01349e58b60154f469ae3ee6c8c30555
Author: Li Bo <[email protected]>
Date:   Thu Oct 12 08:52:22 2023 +0000

    mathvista

commit 34b70f8068cfd5fbdc36980e6ac5215942849723
Author: Pu Fanyi <[email protected]>
Date:   Thu Oct 12 15:56:08 2023 +0800

    generate results.json for seedbench (#24)

commit 19fdf40a3f0cd05551f482f9834e38f9210256a3
Author: Li Bo <[email protected]>
Date:   Thu Oct 12 02:04:38 2023 +0000

    black lint

commit 01d56cec48f65307ed33e2a2dbd853717464e51f
Author: Li Bo <[email protected]>
Date:   Thu Oct 12 02:02:52 2023 +0000

    better design for idefics model to integrate multi-types data

commit 1a0972e6cc283af4099f0c85d907c3ec61a36224
Author: Li Bo <[email protected]>
Date:   Wed Oct 11 18:28:19 2023 +0000

    hardcode enabling flash

commit cf11867f2bb1a7c50c863eb2b438a588acf4f733
Author: Pu Fanyi <[email protected]>
Date:   Thu Oct 12 00:55:23 2023 +0800

    [evaluation] Fix mmvet output (#23)

    * fix mmvet

    * lint

commit 54362e9de5c0ead2d622544dea8802fd62db19ce
Author: Li Bo <[email protected]>
Date:   Wed Oct 11 07:47:32 2023 +0000

    fix images input for idefics model

commit 96e467c0480cca1abd86ac723f8b33260c49b1a1
Author: Bo Li <[email protected]>
Date:   Wed Oct 11 15:20:04 2023 +0800

    idefics full finetune

commit 11be13deb2c5e57b392d289670237714cdb5b012
Author: Bo Li <[email protected]>
Date:   Wed Oct 11 13:22:32 2023 +0800

    also train vision encoder

commit 3dfcf3e9a468897d1a251163d13a4a5de7542a64
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 14:13:40 2023 +0000

    fix video training

commit b8a086bcbe43e5debd85fab2db11610df513558f
Author: Li Bo <[email protected]>
Date:   Tue Oct 10 21:53:29 2023 +0800

    Dev/train (#22)

    * Merge commit '2d29a417d9e96251be33edda0b0637647882c31b'

    * Conflicts:
    	pipeline/mimicit_utils/mimicit_dataset.py

    * update mpt config

    * fix training padding

    * update fix

    * Squashed commit of the following:

    commit e4f1ec3b287c026a6212a88316d7341883195294
    Author: Li Bo <[email protected]>
    Date:   Tue Oct 10 13:47:47 2023 +0000

        update output path

    commit 1a8e12bfa21697188eaa7a4ed6169223e0f92258
    Author: Li Bo <[email protected]>
    Date:   Tue Oct 10 12:23:02 2023 +0000

        update naming rule

    commit 537d239e3b4109c552a94b80a7bde3bb8a7b107d
    Author: Li Bo <[email protected]>
    Date:   Tue Oct 10 09:10:47 2023 +0000

        fix bug in mme eval

    commit 0c7eacb6ebef4d55187d67b06c88cd504ef17dbd
    Author: Li Bo <[email protected]>
    Date:   Tue Oct 10 14:36:55 2023 +0800

        Dev/mathvista (#21)

        * for mathvista

        * add time info

        * black lint code

        * update

        * update test config

        * stage updates

        * update mathvista eval

        * update mathvista

        * better format prompt

    commit c07f73f6c868444fd784e5844134dc0cccaa566d
    Author: Li Bo <[email protected]>
    Date:   Tue Oct 10 06:33:08 2023 +0000

        update mme saving logic

    commit dce1b0f30a390ace45bcf579d93764847848ee51
    Author: Li Bo <[email protected]>
    Date:   Tue Oct 10 06:28:29 2023 +0000

        fix mme

    commit b970f617bea2b418cf726a50b4cd008dc7eb4907
    Author: Li Bo <[email protected]>
    Date:   Tue Oct 10 06:11:03 2023 +0000

        prepare

    commit 33e6cb0f0212ed845ca17956743926b4d53ec0f3
    Author: Li Bo <[email protected]>
    Date:   Tue Oct 10 06:09:57 2023 +0000

        resolve idefics for seedbench eval

    commit fc2a87bc853a34ce8bfb16b7f6a504a1d63af0e5
    Author: Li Bo <[email protected]>
    Date:   Tue Oct 10 12:19:06 2023 +0800

        Dev/refactor (#20)

        * seedbench_eval

        * update seedbench eval for idefics

        * update

        * resolve input for seedbench

        * update

        ---------

        Co-authored-by: Joshua Adrian Cahyono <[email protected]>

    commit edb2440eec77c6c07ed5457a1251e567e57b88f2
    Author: JvThunder <[email protected]>
    Date:   Tue Oct 10 02:42:27 2023 +0800

        seedbench_eval (#19)

        * seedbench_eval

        * update seedbench eval for idefics

        * update

        ---------

        Co-authored-by: Li Bo <[email protected]>

    commit 39264fc8f0781057de820c122f31fc60ac2f301f
    Author: Pu Fanyi <[email protected]>
    Date:   Mon Oct 9 20:08:29 2023 +0800

        [Eval] MMVet Eval (#18)

        * prepare dataset

        * MME

        * add mmvet dataset

        * git ignore

        * MMVet

        * lint

        * Squashed commit of the following:

        commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
        Author: Li Bo <[email protected]>
        Date:   Mon Oct 9 02:02:10 2023 +0000

            update wandb logging

        commit 2d29a417d9e96251be33edda0b0637647882c31b
        Author: Li Bo <[email protected]>
        Date:   Mon Oct 9 01:45:05 2023 +0000

            update logging with wandb

        commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
        Author: Li Bo <[email protected]>
        Date:   Mon Oct 9 01:35:25 2023 +0000

            update

        commit b8312afc54915b615fd8319bcffc9e0781af346a
        Author: Li Bo <[email protected]>
        Date:   Sun Oct 8 14:41:23 2023 +0000

            benchmark eval updates

        commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
        Author: Li Bo <[email protected]>
        Date:   Sun Oct 8 13:28:24 2023 +0000

            black lint

        commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
        Author: Li Bo <[email protected]>
        Date:   Sun Oct 8 13:27:22 2023 +0000

            eval update

        commit cff3f32262bcdba980f84e82b874aa03c05c9027
        Author: Bo Li <[email protected]>
        Date:   Sun Oct 8 18:44:22 2023 +0800

            explicitly assign constant scheduler

        * Squashed commit of the following:

        commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
        Author: Li Bo <[email protected]>
        Date:   Mon Oct 9 02:02:10 2023 +0000

            update wandb logging

        commit 2d29a417d9e96251be33edda0b0637647882c31b
        Author: Li Bo <[email protected]>
        Date:   Mon Oct 9 01:45:05 2023 +0000

            update logging with wandb

        commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
        Author: Li Bo <[email protected]>
        Date:   Mon Oct 9 01:35:25 2023 +0000

            update

        commit b8312afc54915b615fd8319bcffc9e0781af346a
        Author: Li Bo <[email protected]>
        Date:   Sun Oct 8 14:41:23 2023 +0000

            benchmark eval updates

        commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
        Author: Li Bo <[email protected]>
        Date:   Sun Oct 8 13:28:24 2023 +0000

            black lint

        commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
        Author: Li Bo <[email protected]>
        Date:   Sun Oct 8 13:27:22 2023 +0000

            eval update

        commit cff3f32262bcdba980f84e82b874aa03c05c9027
        Author: Bo Li <[email protected]>
        Date:   Sun Oct 8 18:44:22 2023 +0800

            explicitly assign constant scheduler

        * Squashed commit of the following:

        commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
        Author: Li Bo <[email protected]>
        Date:   Mon Oct 9 02:02:10 2023 +0000

            update wandb logging

        commit 2d29a417d9e96251be33edda0b0637647882c31b
        Author: Li Bo <[email protected]>
        Date:   Mon Oct 9 01:45:05 2023 +0000

            update logging with wandb

        commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
        Author: Li Bo <[email protected]>
        Date:   Mon Oct 9 01:35:25 2023 +0000

            update

        commit b8312afc54915b615fd8319bcffc9e0781af346a
        Author: Li Bo <[email protected]>
        Date:   Sun Oct 8 14:41:23 2023 +0000

            benchmark eval updates

        commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
        Author: Li Bo <[email protected]>
        Date:   Sun Oct 8 13:28:24 2023 +0000

            black lint

        commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
        Author: Li Bo <[email protected]>
        Date:   Sun Oct 8 13:27:22 2023 +0000

            eval update

        commit cff3f32262bcdba980f84e82b874aa03c05c9027
        Author: Bo Li <[email protected]>
        Date:   Sun Oct 8 18:44:22 2023 +0800

            explicitly assign constant scheduler

        ---------

        Co-authored-by: Li Bo <[email protected]>

    commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 02:02:10 2023 +0000

        update wandb logging

    commit 2d29a417d9e96251be33edda0b0637647882c31b
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 01:45:05 2023 +0000

        update logging with wandb

    commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
    Author: Li Bo <[email protected]>
    Date:   Mon Oct 9 01:35:25 2023 +0000

        update

    commit b8312afc54915b615fd8319bcffc9e0781af346a
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 14:41:23 2023 +0000

        benchmark eval updates

    commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 13:28:24 2023 +0000

        black lint

    commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
    Author: Li Bo <[email protected]>
    Date:   Sun Oct 8 13:27:22 2023 +0000

        eval update

* update saved dir

* prompt fix

* avoid divide zero error

* visible response

* Make otter public dataset runnable

* Make otter public dataset runnable

* lint

* format instructions

* format instructions

* Squashed commit of the following:

commit 6baf46d1041680a3e39058a661aebd7752f3e647
Author: Fanyi Pu <[email protected]>
Date:   Fri Oct 13 18:13:42 2023 +0800

    lint

commit 29f9835babc40642c887c57812aa9ed86d03a301
Author: Fanyi Pu <[email protected]>
Date:   Fri Oct 13 18:12:31 2023 +0800

    idefics batch generator

commit f5a4905c4bd072808c79354696a02191aed212aa
Author: Fanyi Pu <[email protected]>
Date:   Fri Oct 13 17:05:40 2023 +0800

    add idefics class

commit d99f61f3b585909899fdb07f33bbf1e2af674e35
Author: Fanyi Pu <[email protected]>
Date:   Fri Oct 13 13:21:49 2023 +0800

    Squashed commit of the following:

    commit a65945c57cdb8156eb16c17081f6a162ce950993
    Author: Li Bo <[email protected]>
    Date:   Fri Oct 13 04:36:44 2023 +0000

        avoid divide zero error

    commit 74143e50677a85d37fb4f8430aa84f02b3184092
    Merge: 2dd2d17 7b88300
    Author: Li Bo <[email protected]>
    Date:   Fri Oct 13 04:21:07 2023 +0000

        Merge branch 'dev/train' of https://github.com/Luodian/Otter-2 into dev/train

    commit 2dd2d1788934b15d4de01377c59ebefa15182cc1
    Author: Li Bo <[email protected]>
    Date:   Fri Oct 13 04:21:05 2023 +0000

        prompt fix

    commit 7b883005a874edac7819ea23797d416ebac5b22b
    Merge: 7d3a713 9417706
    Author: Li Bo <[email protected]>
    Date:   Fri Oct 13 12:18:02 2023 +0800

        Merge branch 'main_dev' into dev/train

    commit 7d3a713ce0816dddae094287d926a24b361a79a1
    Author: Li Bo <[email protected]>
    Date:   Fri Oct 13 04:15:36 2023 +0000

        update saved dir

    commit 36bfb402a0aecfeed0fe9699f091bdd0eb198a6a
    Author: Li Bo <[email protected]>
    Date:   Fri Oct 13 03:43:07 2023 +0000

        Squashed commit of the following:

        commit 94177061ff29e8c395ae63d408f64297cfa09211
        Author: JvThunder <[email protected]>
        Date:   Fri Oct 13 11:42:56 2023 +0800

            Dev/refactor (#26)

            * seedbench_eval

            * update seedbench eval for idefics

            * update

            * resolve input for seedbench

            * update

            * make_submission_seedbench

            * pope_eval

            * update_pope_eval

            * update_pope

            * black linta

            ---------

            Co-authored-by: Li Bo <[email protected]>

        commit 52bacca3d858d2ac32f44f8b1619ce5064ae1a28
        Author: Li Bo <[email protected]>
        Date:   Thu Oct 12 10:44:46 2023 +0000

            update configs and demo evals

        commit a41247d193b3ca0967c38d0b3b11e2a04ebf9ca7
        Merge: 9ff5957 34b70f8
        Author: Li Bo <[email protected]>
        Date:   Thu Oct 12 10:02:24 2023 +0000

            Merge branch 'main_dev' of https://github.com/Luodian/Otter-2 into main_dev

        commit 9ff5957c01349e58b60154f469ae3ee6c8c30555
        Author: Li Bo <[email protected]>
        Date:   Thu Oct 12 08:52:22 2023 +0000

            mathvista

        commit 34b70f8068cfd5fbdc36980e6ac5215942849723
        Author: Pu Fanyi <[email protected]>
        Date:   Thu Oct 12 15:56:08 2023 +0800

            generate results.json for seedbench (#24)

        commit 19fdf40a3f0cd05551f482f9834e38f9210256a3
        Author: Li Bo <[email protected]>
        Date:   Thu Oct 12 02:04:38 2023 +0000

            black lint

        commit 01d56cec48f65307ed33e2a2dbd853717464e51f
        Author: Li Bo <[email protected]>
        Date:   Thu Oct 12 02:02:52 2023 +0000

            better design for idefics model to integrate multi-types data

        commit 1a0972e6cc283af4099f0c85d907c3ec61a36224
        Author: Li Bo <[email protected]>
        Date:   Wed Oct 11 18:28:19 2023 +0000

            hardcode enabling flash

        commit cf11867f2bb1a7c50c863eb2b438a588acf4f733
        Author: Pu Fanyi <[email protected]>
        Date:   Thu Oct 12 00:55:23 2023 +0800

            [evaluation] Fix mmvet output (#23)

            * fix mmvet

            * lint

        commit 54362e9de5c0ead2d622544dea8802fd62db19ce
        Author: Li Bo <[email protected]>
        Date:   Wed Oct 11 07:47:32 2023 +0000

            fix images input for idefics model

        commit 96e467c0480cca1abd86ac723f8b33260c49b1a1
        Author: Bo Li <[email protected]>
        Date:   Wed Oct 11 15:20:04 2023 +0800

            idefics full finetune

        commit 11be13deb2c5e57b392d289670237714cdb5b012
        Author: Bo Li <[email protected]>
        Date:   Wed Oct 11 13:22:32 2023 +0800

            also train vision encoder

        commit 3dfcf3e9a468897d1a251163d13a4a5de7542a64
        Author: Li Bo <[email protected]>
        Date:   Tue Oct 10 14:13:40 2023 +0000

            fix video training

        commit b8a086bcbe43e5debd85fab2db11610df513558f
        Author: Li Bo <[email protected]>
        Date:   Tue Oct 10 21:53:29 2023 +0800

            Dev/train (#22)

            * Merge commit '2d29a417d9e96251be33edda0b0637647882c31b'

            * Conflicts:
            	pipeline/mimicit_utils/mimicit_dataset.py

            * update mpt config

            * fix training padding

            * update fix

            * Squashed commit of the following:

            commit e4f1ec3b287c026a6212a88316d7341883195294
            Author: Li Bo <[email protected]>
            Date:   Tue Oct 10 13:47:47 2023 +0000

                update output path

            commit 1a8e12bfa21697188eaa7a4ed6169223e0f92258
            Author: Li Bo <[email protected]>
            Date:   Tue Oct 10 12:23:02 2023 +0000

                update naming rule

            commit 537d239e3b4109c552a94b80a7bde3bb8a7b107d
            Author: Li Bo <[email protected]>
            Date:   Tue Oct 10 09:10:47 2023 +0000

                fix bug in mme eval

            commit 0c7eacb6ebef4d55187d67b06c88cd504ef17dbd
            Author: Li Bo <[email protected]>
            Date:   Tue Oct 10 14:36:55 2023 +0800

                Dev/mathvista (#21)

                * for mathvista

                * add time info

                * black lint code

                * update

                * update test config

                * stage updates

                * update mathvista eval

                * update mathvista

                * better format prompt

            commit c07f73f6c868444fd784e5844134dc0cccaa566d
            Author: Li Bo <[email protected]>
            Date:   Tue Oct 10 06:33:08 2023 +0000

                update mme saving logic

            commit dce1b0f30a390ace45bcf579d93764847848ee51
            Author: Li Bo <[email protected]>
            Date:   Tue Oct 10 06:28:29 2023 +0000

                fix mme

            commit b970f617bea2b418cf726a50b4cd008dc7eb4907
            Author: Li Bo <[email protected]>
            Date:   Tue Oct 10 06:11:03 2023 +0000

                prepare

            commit 33e6cb0f0212ed845ca17956743926b4d53ec0f3
            Author: Li Bo <[email protected]>
            Date:   Tue Oct 10 06:09:57 2023 +0000

                resolve idefics for seedbench eval

            commit fc2a87bc853a34ce8bfb16b7f6a504a1d63af0e5
            Author: Li Bo <[email protected]>
            Date:   Tue Oct 10 12:19:06 2023 +0800

                Dev/refactor (#20)

                * seedbench_eval

                * update seedbench eval for idefics

                * update

                * resolve input for seedbench

                * update

                ---------

                Co-authored-by: Joshua Adrian Cahyono <[email protected]>

            commit edb2440eec77c6c07ed5457a1251e567e57b88f2
            Author: JvThunder <[email protected]>
            Date:   Tue Oct 10 02:42:27 2023 +0800

                seedbench_eval (#19)

                * seedbench_eval

                * update seedbench eval for idefics

                * update

                ---------

                Co-authored-by: Li Bo <[email protected]>

            commit 39264fc8f0781057de820c122f31fc60ac2f301f
            Author: Pu Fanyi <[email protected]>
            Date:   Mon Oct 9 20:08:29 2023 +0800

                [Eval] MMVet Eval (#18)

                * prepare dataset

                * MME

                * add mmvet dataset

                * git ignore

                * MMVet

                * lint

                * Squashed commit of the following:

                commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
                Author: Li Bo <[email protected]>
                Date:   Mon Oct 9 02:02:10 2023 +0000

                    update wandb logging

                commit 2d29a417d9e96251be33edda0b0637647882c31b
                Author: Li Bo <[email protected]>
                Date:   Mon Oct 9 01:45:05 2023 +0000

                    update logging with wandb

                commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
                Author: Li Bo <[email protected]>
                Date:   Mon Oct 9 01:35:25 2023 +0000

                    update

                commit b8312afc54915b615fd8319bcffc9e0781af346a
                Author: Li Bo <[email protected]>
                Date:   Sun Oct 8 14:41:23 2023 +0000

                    benchmark eval updates

                commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
                Author: Li Bo <[email protected]>
                Date:   Sun Oct 8 13:28:24 2023 +0000

                    black lint

                commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
                Author: Li Bo <[email protected]>
                Date:   Sun Oct 8 13:27:22 2023 +0000

                    eval update

                commit cff3f32262bcdba980f84e82b874aa03c05c9027
                Author: Bo Li <[email protected]>
                Date:   Sun Oct 8 18:44:22 2023 +0800

                    explicitly assign constant scheduler

                * Squashed commit of the following:

                commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
                Author: Li Bo <[email protected]>
                Date:   Mon Oct 9 02:02:10 2023 +0000

                    update wandb logging

                commit 2d29a417d9e96251be33edda0b0637647882c31b
                Author: Li Bo <[email protected]>
                Date:   Mon Oct 9 01:45:05 2023 +0000

                    update logging with wandb

                commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
                Author: Li Bo <[email protected]>
                Date:   Mon Oct 9 01:35:25 2023 +0000

                    update

                commit b8312afc54915b615fd8319bcffc9e0781af346a
                Author: Li Bo <[email protected]>
                Date:   Sun Oct 8 14:41:23 2023 +0000

                    benchmark eval updates

                commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
                Author: Li Bo <[email protected]>
                Date:   Sun Oct 8 13:28:24 2023 +0000

                    black lint

                commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
                Author: Li Bo <[email protected]>
                Date:   Sun Oct 8 13:27:22 2023 +0000

                    eval update

                commit cff3f32262bcdba980f84e82b874aa03c05c9027
                Author: Bo Li <[email protected]>
                Date:   Sun Oct 8 18:44:22 2023 +0800

                    explicitly assign constant scheduler

                * Squashed commit of the following:

                commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
                Author: Li Bo <[email protected]>
                Date:   Mon Oct 9 02:02:10 2023 +0000

                    update wandb logging

                commit 2d29a417d9e96251be33edda0b0637647882c31b
                Author: Li Bo <[email protected]>
                Date:   Mon Oct 9 01:45:05 2023 +0000

                    update logging with wandb

                commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
                Author: Li Bo <[email protected]>
                Date:   Mon Oct 9 01:35:25 2023 +0000

                    update

                commit b8312afc54915b615fd8319bcffc9e0781af346a
                Author: Li Bo <[email protected]>
                Date:   Sun Oct 8 14:41:23 2023 +0000

                    benchmark eval updates

                commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
                Author: Li Bo <[email protected]>
                Date:   Sun Oct 8 13:28:24 2023 +0000

                    black lint

                commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
                Author: Li Bo <[email protected]>
                Date:   Sun Oct 8 13:27:22 2023 +0000

                    eval update

                commit cff3f32262bcdba980f84e82b874aa03c05c9027
                Author: Bo Li <[email protected]>
                Date:   Sun Oct 8 18:44:22 2023 +0800

                    explicitly assign constant scheduler

                ---------

                Co-authored-by: Li Bo <[email protected]>

            commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
            Author: Li Bo <[email protected]>
            Date:   Mon Oct 9 02:02:10 2023 +0000

                update wandb logging

            commit 2d29a417d9e96251be33edda0b0637647882c31b
            Author: Li Bo <[email protected]>
            Date:   Mon Oct 9 01:45:05 2023 +0000

                update logging with wandb

            commit 3b55adba8d52b19fe5fc8acb3c64706793682b02
            Author: Li Bo <[email protected]>
            Date:   Mon Oct 9 01:35:25 2023 +0000

                update

            commit b8312afc54915b615fd8319bcffc9e0781af346a
            Author: Li Bo <[email protected]>
            Date:   Sun Oct 8 14:41:23 2023 +0000

                benchmark eval updates

            commit 956a7cc1f968768b19fbd97490b1a8ff9f6ad1d1
            Author: Li Bo <[email protected]>
            Date:   Sun Oct 8 13:28:24 2023 +0000

                black lint

            commit 048778f0de04cb4f08d74f2aaab5da33ce8e69c6
            Author: Li Bo <[email protected]>
            Date:   Sun Oct 8 13:27:22 2023 +0000

                eval update

    commit 94177061ff29e8c395ae63d408f64297cfa09211
    Author: JvThunder <[email protected]>
    Date:   Fri Oct 13 11:42:56 2023 +0800

        Dev/refactor (#26)

        * seedbench_eval

        * update seedbench eval for idefics

        * update

        * resolve input for seedbench

        * update

        * make_submission_seedbench

        * pope_eval

        * update_pope_eval

        * update_pope

        * black linta

        ---------

        Co-authored-by: Li Bo <[email protected]>

    commit 0907722ca82b38b3eacc897140f8f1b1236d8ef9
    Author: Li Bo <[email protected]>
    Date:   Thu Oct 12 14:33:21 2023 +0000

        update

    commit b1760445eda1d0199aab1c6f93cd663bb68087e9
    Author: Li Bo <[email protected]>
    Date:   Thu Oct 12 11:10:10 2023 +0000

        Squashed commit of the following:

        commit 52bacca3d858d2ac32f44f8b1619ce5064ae1a28
        Author: Li Bo <[email protected]>
        Date:   Thu Oct 12 10:44:46 2023 +0000

            update configs and demo evals

        commit a41247d193b3ca0967c38d0b3b11e2a04ebf9ca7
        Merge: 9ff5957 34b70f8
        Author: Li Bo <[email protected]>
        Date:   Thu Oct 12 10:02:24 2023 +0000

            Merge branch 'main_dev' of https://github.com/Luodian/Otter-2 into main_dev

        commit 9ff5957c01349e58b60154f469ae3ee6c8c30555
        Author: Li Bo <[email protected]>
        Date:   Thu Oct 12 08:52:22 2023 +0000

            mathvista

        commit 34b70f8068cfd5fbdc36980e6ac5215942849723
        Author: Pu Fanyi <[email protected]>
        Date:   Thu Oct 12 15:56:08 2023 +0800

            generate results.json for seedbench (#24)

        commit 19fdf40a3f0cd05551f482f9834e38f9210256a3
        Author: Li Bo <[email protected]>
        Date:   Thu Oct 12 02:04:38 2023 +0000

            black lint

        commit 01d56cec48f65307ed33e2a2dbd853717464e51f
        Author: Li Bo <[email protected]>
        Date:   Thu Oct 12 02:02:52 2023 +0000

            better design for idefics model to integrate multi-types data

        commit 1a0972e6cc283af4099f0c85d907c3ec61a36224
        Author: Li Bo <[email protected]>
        Date:   Wed Oct 11 18:28:19 2023 +0000

            hardcode enabling flash

        commit cf11867f2bb1a7c50c863eb2b438a588acf4f733
        Author: Pu Fanyi <[email protected]>
        Date:   Thu Oct 12 00:55:23 2023 +0800

            [evaluation] Fix mmvet output (#23)

            * fix mmvet

            * lint

        commit 54362e9de5c0ead2d622544dea8802fd62db19ce
        Author: Li Bo <[email protected]>
        Date:   Wed Oct 11 07:47:32 2023 +0000

            fix images input for idefics model

        commit 96e467c0480cca1abd86ac723f8b33260c49b1a1
        Author: Bo Li <[email protected]>
        Date:   Wed Oct 11 15:20:04 2023 +0800

            idefics full finetune

        commit 11be13deb2c5e57b392d289670237714cdb5b012
        Author: Bo Li <[email protected]>
        Date:   Wed Oct 11 13:22:32 2023 +0800

            also train vision encoder

        commit 3dfcf3e9a468897d1a251163d13a4a5de7542a64
        Author: Li Bo <[email protected]>
        Date:   Tue Oct 10 14:13:40 2023 +0000

            fix video training

        commit b8a086bcbe43e5debd85fab2db11610df513558f
        Author: Li Bo <[email protected]>
        Date:   Tue Oct 10 21:53:29 2023 +0800

            Dev/train (#22)

            * Merge commit '2d29a417d9e96251be33edda0b0637647882c31b'

            * Conflicts:
            	pipeline/mimicit_utils/mimicit_dataset.py

            * update mpt config

            * fix training padding

            * update fix

            * Squashed commit of the following:

            commit e4f1ec3b287c026a6212a88316d7341883195294
            Author: Li Bo <[email protected]>
            Date:   Tue Oct 10 13:47:47 2023 +0000

                update output path

            commit 1a8e12bfa21697188eaa7a4ed6169223e0f92258
            Author: Li Bo <[email protected]>
            Date:   Tue Oct 10 12:23:02 2023 +0000

                update naming rule

            commit 537d239e3b4109c552a94b80a7bde3bb8a7b107d
            Author: Li Bo <[email protected]>
            Date:   Tue Oct 10 09:10:47 2023 +0000

                fix bug in mme eval

            commit 0c7eacb6ebef4d55187d67b06c88cd504ef17dbd
            Author: Li Bo <[email protected]>
            Date:   Tue Oct 10 14:36:55 2023 +0800

                Dev/mathvista (#21)

                * for mathvista

                * add time info

                * black lint code

                * update

                * update test config

                * stage updates

                * update mathvista eval

                * update mathvista

                * better format prompt

            commit c07f73f6c868444fd784e5844134dc0cccaa566d
            Author: Li Bo <[email protected]>
            Date:   Tue Oct 10 06:33:08 2023 +0000

                update mme saving logic

            commit dce1b0f30a390ace45bcf579d93764847848ee51
            Author: Li Bo <[email protected]>
            Date:   Tue Oct 10 06:28:29 2023 +0000

                fix mme

            commit b970f617bea2b418cf726a50b4cd008dc7eb4907
            Author: Li Bo <[email protected]>
            Date:   Tue Oct 10 06:11:03 2023 +0000

                prepare

            commit 33e6cb0f0212ed845ca17956743926b4d53ec0f3
            Author: Li Bo <[email protected]>
            Date:   Tue Oct 10 06:09:57 2023 +0000

                resolve idefics for seedbench eval

            commit fc2a87bc853a34ce8bfb16b7f6a504a1d63af0e5
            Author: Li Bo <[email protected]>
            Date:   Tue Oct 10 12:19:06 2023 +0800

                Dev/refactor (#20)

                * seedbench_eval

                * update seedbench eval for idefics

                * update

                * resolve input for seedbench

                * update

                ---------

                Co-authored-by: Joshua Adrian Cahyono <[email protected]>

            commit edb2440eec77c6c07ed5457a1251e567e57b88f2
            Author: JvThunder <[email protected]>
            Date:   Tue Oct 10 02:42:27 2023 +0800

                seedbench_eval (#19)

                * seedbench_eval

                * update seedbench eval for idefics

                * update

                ---------

                Co-authored-by: Li Bo <[email protected]>

            commit 39264fc8f0781057de820c122f31fc60ac2f301f
            Author: Pu Fanyi <[email protected]>
            Date:   Mon Oct 9 20:08:29 2023 +0800

                [Eval] MMVet Eval (#18)

                * prepare dataset

                * MME

                * add mmvet dataset

                * git ignore

                * MMVet

                * lint

                * Squashed commit of the following:

                commit 1da20cf87fccae6e379463e96fffbcfa9ad9c592
                Author: Li Bo <[email protected]>
                Date:   Mon Oct 9 02:02:10 2023 +0000

                    update wandb logging

                commit 2d29a417d9e96251be33edda0b0637647882c31b
                Author: Li Bo <drlu…

Loading branch information

7 people authored Nov 8, 2023

1 parent 0d12192 commit 1c056f1

.gitattributes

100644 → 100755

Empty file.

.github/issue_template.md

100644 → 100755

Empty file.

.github/pull_request_template.md

100644 → 100755

Empty file.

.github/workflows/black.yml

100644 → 100755

-Original file line number
+Diff line change
@@ Expand Up / @@ -7,4 +7,4 @@ jobs: @@
         runs-on: ubuntu-latest
         steps:
           - uses: actions/checkout@v3
-          - uses: psf/black@stable
+          - uses: psf/black@stable

.gitignore

100644 → 100755

-Original file line number
+Diff line change
@@ Expand Up / @@ -14,6 +14,7 @@ __pycache__/ @@
     *.dir
     *.html
     *.mp4
+    *.MP4
     *.png
     *.pt
     *.bin
@@ Expand Down Expand Up / @@ -116,7 +117,6 @@ logs/ @@
     ofa_compress/
     train_*.sh
     gpt_playground/
-    evaluation/
     data/
     azure/
     .deepspeed_env
@@ Expand All / @@ -134,4 +134,22 @@ USE_POLICY.md @@
     *.pyc
     open_flamingo.egg-info
     llama-7b-hf/*
-    cache/
+    cache/
+    *.code-workspace
+    pipeline/benchmarks/ckpts/*
+    pipeline/benchmarks/models/Ask_Anything
+    pipeline/benchmarks/models/FrozenBiLM
+    pipeline/benchmarks/models/LLaMA_Adapter
+    shared_scripts/gcp_instance/**
+    shared_scripts/shai_instance/**
+    pipeline/benchmarks/models/Video_ChatGPT
+    pipeline/benchmarks/models/LLaVA
+    pipeline/benchmarks/models/llava
+    pipeline/benchmarks/evaluation_result
+    pipeline/serve/user_logs
+    config.yaml
+    azure_storage/
+    checkpoints/
+    *.ttf

CODE_OF_CONDUCT.md

100644 → 100755

Empty file.

LICENSE

100644 → 100755

Empty file.

README.md

100644 → 100755

Large diffs are not rendered by default.

conftest.py

-Original file line number
+Diff line change
@@ -0,0 +1,15 @@
+    import pytest
+    def pytest_addoption(parser):
+        parser.addoption(
+            "--yaml-path",
+            action="store",
+            default="default_yaml_path.yaml",
+            help="Path to the YAML file",
+        )
+    @pytest.fixture
+    def yaml_path(request):
+        return request.config.getoption("--yaml-path")

docs/OtterHD.md

-Original file line number
+Diff line change
@@ -0,0 +1,109 @@
+    ## OtterHD
+    <p align="center" width="100%">
+    <img src="https://i.postimg.cc/1zhXhzTr/otterhd-title.png"  width="80%" height="80%">
+    </p>
+    <div>
+    <div align="center">
+        <a href='https://brianboli.com/' target='_blank'>Bo Li<sup>*</sup></a>&emsp;
+        <a href='https://veiled-texture-20c.notion.site/Peiyuan-Zhang-ab24b48621c9491db767a76df860873a?pvs=4' target='_blank'>Peiyuan Zhang<sup>*</sup></a>&emsp;
+        </br>
+        <a href='https://jingkang50.github.io/' target='_blank'>Jingkang Yang<sup>♠</sup></a>&emsp;
+        <a href='https://zhangyuanhan-ai.github.io/' target='_blank'>Yuanhan Zhang<sup>♠</sup></a>&emsp;
+        <a href='https://pufanyi.github.io/' target='_blank'>Fanyi Pu<sup>♠</sup></a>&emsp;
+        <a href='https://liuziwei7.github.io/' target='_blank'>Ziwei Liu<sup>&#x2709</sup></a>
+    </div>
+    <div>
+    <div align="center">
+        S-Lab, Nanyang Technological University&emsp;
+        </br>
+        <sup>*</sup> Equal Contribution&emsp;
+        <sup>♠</sup> Equal appreciation on assistance&emsp;
+        <sup>&#x2709</sup> Corresponding Author
+    </div>
+    [Technical Report](link) | [Demo](https://huggingface.co/spaces/Otter-AI/OtterHD-8B-demo) | [Benchmarks](https://huggingface.co/spaces/Otter-AI)
+    We introduce OtterHD-8B, a multimodal model fine-tuned from [Fuyu-8B](https://huggingface.co/adept/fuyu-8b) to facilitate a more fine-grained interpretation of high-resolution visual input without requiring a vision encoder. OtterHD-8B also supports flexible input sizes at test time, ensuring adaptability to diverse inference budgets.
+    We improve the native HuggingFace implementation of Fuyu-8B is highly unoptimized with [FlashAttention-2](https://github.com/Dao-AILab/flash-attention) and other fused operators including fused layernorm, fused square ReLU, and fused rotary positional embedding. Fuyu's simplified architecture facilitates us to do this in a fairly convenient way. As illustrated in the following, the modifications substantially enhance GPU utilization and training throughput (> 5 times larger than the vanilla HF implementation of Fuyu). Checkout the details at [here](../src/otter_ai/models/fuyu/modeling_fuyu.py).
+    To our best knowledge and experiment trials, OtterHD achieves fastest training throughput among current leading LMMs, as it can be fully optimized and benefit from the simplified architecture.
+    <p align="center" width="100%">
+    <img src="https://i.postimg.cc/c43PkMqC/tokens-throughput.png"  width="80%" height="80%">
+    </p>
+    ### Installation
+    On top of the regular Otter environment, we need to install Flash-Attention 2 and other fused operators:
+    ```bash
+    pip uninstall -y ninja && pip install ninja
+    git clone https://github.com/Dao-AILab/flash-attention
+    cd flash-attention
+    python setup.py install
+    cd csrc/rotary && pip install .
+    cd ../csrc/fused_dense_lib && pip install .
+    cd ../layer_norm && pip install .
+    cd ../xentropy && pip install .
+    cd ../.. && rm -rf flash-attention
+    ```
+    ### How to Finetune
+    ```bash
+    accelerate launch \
+    --config_file=pipeline/accelerate_configs/accelerate_config_zero2.yaml \
+    --num_processes=8 \
+    --main_process_port=25000 \
+    pipeline/train/instruction_following.py \
+    --pretrained_model_name_or_path=adept/fuyu-8b \
+    --training_data_yaml=./Demo_Data.yaml \
+    --model_name=fuyu \
+    --instruction_format=fuyu \
+    --batch_size=8 \
+    --gradient_accumulation_steps=2 \
+    --num_epochs=3 \
+    --wandb_entity=ntu-slab \
+    --external_save_dir=./checkpoints \
+    --save_hf_model \
+    --run_name=OtterHD_Tester \
+    --wandb_project=Fuyu \
+    --report_to_wandb \
+    --workers=1 \
+    --lr_scheduler=linear \
+    --learning_rate=1e-5 \
+    --warmup_steps_ratio=0.01 \
+    --dynamic_resolution \
+    --weight_decay 0.1 \
+    ```
+    ## MagnifierBench
+    <p align="center" width="100%">
+    <img src="https://i.postimg.cc/fL8pSXK7/Screenshot-2023-11-05-at-3-52-58-PM.png"  width="80%" height="80%">
+    </p>
+    The human visual system can naturally perceive the details of small objects within a wide field of view, but current benchmarks for testing LMMs have not specifically focused on assessing this ability. This may be because the input sizes of mainstream Vision-Language models are constrained to relatively small resolutions. With the advent of the Fuyu and OtterHD models, we can extend the input resolution to a much larger range. Therefore, there is an urgent need for a benchmark that can test the ability to discern the details of small objects (often 1% image size) in high-resolution input images.
+    ### Evaluation
+    Create a yaml file `benchmark.yaml` with below content:
+    ```yaml
+    datasets:
+      - name: magnifierbench
+        split: test
+        data_path: Otter-AI/MagnifierBench
+        prompt: Answer with the option letter from the given choices directly.
+        api_key: [You GPT-4 API]
+    models:
+      - name: fuyu
+        model_path: azure_storage/fuyu-8b
+        resolution: 1440
+    ```
+    Then run
+    ```python
+    python -m pipeline.benchmarks.evaluate --confg benchmark.yaml
+    ```

docs/benchmark_eval.md

-Original file line number
+Diff line change
@@ -0,0 +1,59 @@
+    # Welcome to the benchmark evaluation page!
+    The evaluation pipeline is designed to be one-clickable and easy to use. However, you may encounter some problems when running the models (e.g. LLaVA, LLaMA-Adapter) that require you to clone their repo to local path. Please feel free to contact us if you have any questions.
+    We support the following benchmarks:
+    - MagnifierBench
+    - MMBench
+    - MM-VET
+    - MathVista
+    - POPE
+    - MME
+    - SicenceQA
+    - SeedBench
+    And following models:
+    - LLaVA
+    - Fuyu
+    - OtterHD
+    - Otter-Image
+    - Otter-Video
+    - Idefics
+    - LLaMA-Adapter
+    - Qwen-VL
+    many more, see `/pipeline/benchmarks/models`
+    Create a yaml file `benchmark.yaml` with below content:
+    ```yaml
+    datasets:
+      - name: magnifierbench
+        split: test
+        data_path: Otter-AI/MagnifierBench
+        prompt: Answer with the option letter from the given choices directly.
+        api_key: [You GPT-4 API]
+      - name: mme
+        split: test
+      - name: pope
+        split: test
+        default_output_path: ./logs
+      - name: mmvet
+        split: test
+        api_key: [You GPT-4 API]
+        gpt_model: gpt-4-0613
+      - name: mathvista
+        split: test
+        api_key: [You GPT-4 API]
+        gpt_model: gpt-4-0613
+      - name: mmbench
+        split: test
+    models:
+      - name: fuyu
+        model_path: adept/fuyu-8b
+    ```
+    Then run
+    ```python
+    python -m pipeline.benchmarks.evaluate --confg benchmark.yaml
+    ```

docs/huggingface_compatible.md

100644 → 100755

Empty file.

docs/mimicit_format.md

-Original file line number
+Diff line change
@@ -0,0 +1,89 @@
+    # Breaking Down the MIMIC-IT Format
+    We mainly use one integrate dataset format and we refer it to MIMIC-IT format since.
+    The mimic-it format contains the following data yaml file. Within this data yaml file, you could assign the path of the instruction json file and the image parquet file, and also the number of samples you want to use. The number of samples within each group will be uniformly sampled, and the `number_samples / total_numbers`` will decide sampling ratio of each dataset.
+    ```yaml
+    IMAGE_TEXT: # Group name should be in [IMAGE_TEXT, TEXT_ONLY, IMAGE_TEXT_IN_CONTEXT]
+      LADD: # Dataset name can be assigned at any name you want
+        mimicit_path: azure_storage/json/LA/LADD_instructions.json # Path of the instruction json file
+        images_path: azure_storage/Parquets/LA.parquet # Path of the image parquet file
+        num_samples: -1 # Number of samples you want to use, -1 means use all samples, if not set, default is -1.
+      LACR_T2T:
+        mimicit_path: azure_storage/json/LA/LACR_T2T_instructions.json
+        images_path: azure_storage/Parquets/LA.parquet
+        num_samples: -1
+      M3IT_CAPTIONING:
+        mimicit_path: azure_storage/json/M3IT/captioning/coco/coco_instructions.json
+        images_path: azure_storage/Parquets/coco.parquet
+        num_samples: 20000
+    TEXT_ONLY:
+      LIMA:
+        mimicit_path: azure_storage/json/LANG_Only/LIMA/LIMA_instructions_max_1K_tokens.json
+        num_samples: 20000
+      SHAREGPT:
+        mimicit_path: azure_storage/json/LANG_Only/SHAREGPT/SHAREGPT_instructions_max_1K_tokens.json
+        num_samples: 10000
+      AL:
+        mimicit_path: azure_storage/json/LANG_Only/AL/AL_instructions_max_1K_tokens.json
+        num_samples: 20000
+    ```
+    The data yaml file mainly include two groups of data (1) IMAGE_TEXT (2) TEXT_ONLY.
+    For each group, one dataset contains the `instruction.json` file and `images.parquet` file. You can browse the `instruction.json` file at [here](https://entuedu-my.sharepoint.com/:f:/g/personal/libo0013_e_ntu_edu_sg/Eo9bgNV5cjtEswfA-HfjNNABiKsjDzSWAl5QYAlRZPiuZA?e=nNUhJH) and the `images.parquet` file at [here](https://entuedu-my.sharepoint.com/:f:/g/personal/libo0013_e_ntu_edu_sg/EmwHqgRtYtBNryTcFmrGWCgBjvWQMo1XeCN250WuM2_51Q?e=sCymXx). We will provide more at the same Onedrive folder gradually due to the limited internet bandwith, you send emails to push us.
+    You are also welcome to make your own data into this format, let's breakdown what's inside them:
+    ## DallE3_instructions.json
+    ```
+    {
+    	"meta": { "version": "0.0.1", "time": "2023-10-29", "author": "Jingkang Yang" },
+    	"data": {
+    		"D3_INS_000000": {
+    			"instruction": "What do you think is the prompt for this AI-generated picture?",
+    			"answer": "photo of a gigantic hand coming from the sky reaching out people who are holding hands at a beach, there is also a giant eye in the sky look at them",
+    			"image_ids": ["D3_IMG_000000"],
+    			"rel_ins_ids": []
+    		},
+    		"D3_INS_000001": {
+    			"instruction": "This is an AI generated image, can you infer what's the prompt behind this image?",
+    			"answer": "photography of a a soccer stadium on the moon, players are dressed as astronauts",
+    			"image_ids": ["D3_IMG_000001"],
+    			"rel_ins_ids": []
+    		}...
+        }
+    }
+    ```
+    Note that the `image_ids` is the key of the `DallE3_images.parquet` file, you can use the `image_ids` to index the `base64` string of the image.
+    ## DallE3_images.parquet
+    ```
+    import pandas as pd
+    images = "./DallE3_images.parquet"
+    image_parquet = pd.read_parquet(images)
+    image_parquet.head()
+    	                                                            base64
+    D3_IMG_000000	/9j/4AAQSkZJRgABAQEASABIAAD/2wBDAAEBAQEBAQEBAQ...
+    D3_IMG_000001	/9j/4AAQSkZJRgABAQEASABIAAD/5FolU0NBTEFETwAAAg...
+    ```
+    Note that before September, we mainly use `images.json` to store the `key:base64_str` pairs, but we found it causes too much CPU memory during decoding large json files. So we switch to parquet, the parquet file is the same as previous json file and you can use the script to convert it from json to parquet.
+    ```python
+    json_file_path = "LA.json"
+    with open(json_file_path, "r") as f:
+        data_dict = json.load(f)
+    df = pd.DataFrame.from_dict(resized_data_dict, orient="index", columns=["base64"])
+    parquet_file_path = os.path.join(
+        parquet_root_path, os.path.basename(json_file_path).split(".")[0].replace("_image", "") + ".parquet"
+    )
+    df.to_parquet(parquet_file_path, engine="pyarrow")
+    ```

docs/server_host.md

100644 → 100755

Empty file.

environment.yml

100644 → 100755

Empty file.

mimic-it/README.md

100644 → 100755

Empty file.

mimic-it/convert-it/README.md

100644 → 100755

Empty file.

mimic-it/convert-it/__init__.py

100644 → 100755

Empty file.

mimic-it/convert-it/abstract_dataset.py

100644 → 100755

Empty file.

mimic-it/convert-it/datasets/2d.py

100644 → 100755

Empty file.

mimic-it/convert-it/datasets/3d.py

100644 → 100755

Empty file.

mimic-it/convert-it/datasets/__init__.py

100644 → 100755

Empty file.

mimic-it/convert-it/datasets/change.py

100644 → 100755

Empty file.

mimic-it/convert-it/datasets/fpv.py

100644 → 100755

-Original file line number
+Diff line change
@@ Expand Up / @@ -56,7 +56,11 @@ def get_image(video_path): @@
             final_images_dict = {}
             with ThreadPoolExecutor(max_workers=num_thread) as executor:
-                process_bar = tqdm(total=len(video_paths), unit="video", desc="Processing videos into images")
+                process_bar = tqdm(
+                    total=len(video_paths),
+                    unit="video",
+                    desc="Processing videos into images",
+                )
                 for images_dict in executor.map(get_image, video_paths):
                     final_images_dict.update(images_dict)
                     process_bar.update()
@@ Expand Down @@

mimic-it/convert-it/datasets/utils/scene_navigation_utils.py

100644 → 100755

Empty file.

mimic-it/convert-it/datasets/utils/visual_story_telling_utils.py

100644 → 100755

Empty file.

mimic-it/convert-it/datasets/video.py

100644 → 100755

Empty file.

mimic-it/convert-it/image_utils.py

100644 → 100755

Empty file.

mimic-it/convert-it/main.py

100644 → 100755

Empty file.

mimic-it/syphus/abstract_dataset.py

100644 → 100755

Empty file.

mimic-it/syphus/datasets/3d.py

100644 → 100755

Empty file.

mimic-it/syphus/datasets/__init__.py

100644 → 100755

Empty file.

mimic-it/syphus/datasets/change.py

100644 → 100755

Empty file.

mimic-it/syphus/datasets/fpv.py

100644 → 100755

Empty file.

mimic-it/syphus/datasets/funqa.py

100644 → 100755

Empty file.

mimic-it/syphus/datasets/translate.py

100644 → 100755

Empty file.

mimic-it/syphus/datasets/video.py

100644 → 100755

Empty file.

mimic-it/syphus/file_utils.py

100644 → 100755

Empty file.

mimic-it/syphus/main.py

100644 → 100755

Empty file.

mimic-it/syphus/prompts/coco_spot_the_difference_prompt.py

100644 → 100755

Empty file.

mimic-it/syphus/prompts/dense_captions.json

100644 → 100755

Empty file.

mimic-it/syphus/prompts/ego4d.json

100644 → 100755

Empty file.

mimic-it/syphus/prompts/funqa_dia.json

100644 → 100755

Empty file.

mimic-it/syphus/prompts/funqa_mcqa.json

100644 → 100755

Empty file.

mimic-it/syphus/prompts/funqa_translation.json

100644 → 100755

Empty file.

mimic-it/syphus/prompts/scene_navigation.json

100644 → 100755

Empty file.

mimic-it/syphus/prompts/spot_the_difference.json

100644 → 100755

Empty file.

mimic-it/syphus/prompts/translation_prompt.py

100644 → 100755

Empty file.

mimic-it/syphus/prompts/tv_captions.json

100644 → 100755

Empty file.

mimic-it/syphus/prompts/visual_story_telling.json

100644 → 100755

Empty file.

pipeline/accelerate_configs/accelerate_config_zero1.yaml

-Original file line number
+Diff line change
@@ -0,0 +1,18 @@
+    compute_environment: LOCAL_MACHINE
+    deepspeed_config:
+      gradient_accumulation_steps: 1
+      gradient_clipping: 1.0
+      offload_optimizer_device: none
+      offload_param_device: none
+      zero3_init_flag: false
+      zero_stage: 1
+    distributed_type: DEEPSPEED
+    fsdp_config: {}
+    machine_rank: 0
+    main_process_ip: null
+    main_process_port: null
+    main_training_function: main
+    mixed_precision: bf16
+    num_machines: 1
+    num_processes: 8
+    use_cpu: false

pipeline/accelerate_configs/accelerate_config_zero2.yaml

-Original file line number
+Diff line change
@@ -1,6 +1,6 @@
     compute_environment: LOCAL_MACHINE
     deepspeed_config:
-      gradient_accumulation_steps: 1
+      gradient_accumulation_steps: 4
       gradient_clipping: 1.0
       offload_optimizer_device: none
       offload_param_device: none
@@ Expand Down @@

pipeline/accelerate_configs/accelerate_config_zero3.yaml

100644 → 100755

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -1,5 +1,5 @@
  
    compute_environment: LOCAL_MACHINE

    deepspeed_config:

    deepspeed_config: 

      gradient_accumulation_steps: 1

      gradient_clipping: 1.0

      offload_optimizer_device: none

    @@ -11,9 +11,9 @@ distributed_type: DEEPSPEED
  
    fsdp_config: {}

    machine_rank: 0

    main_process_ip: null

    main_process_port: 20222

    main_process_port: 20333

    main_training_function: main

    mixed_precision: bf16

    num_machines: 1

    num_processes: 8

    use_cpu: false

    use_cpu: false

0 comments on commit `1c056f1`

Please sign in to comment.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Commit

There are no files selected for viewing

0 comments on commit `1c056f1`

Commit

There are no files selected for viewing

0 comments on commit 1c056f1

0 comments on commit `1c056f1`