Large Multimoal Models Evaluation Interface

We introduce lmms-eval, an evaluation framework meticulously crafted for consistent and efficient evaluation of Large Multimoal Models (LMMs).

This framework simplifies the evaluation process across a diverse array of evaluation datasets. You can evaluate the models on multiple datasets with a single command. We prepared all the evaluation datasets at Huggingface Datasets. No preparation is needed, just one command line, few minutes, and get the results. Not just numbers, but also the detailed logs and samples, including the model args, input question, model response, and ground truth answer.

Utilizing our framework in your model development cycle at GitHub.