[Roadmap] API eval version

[LukeLIN-web]

Let model to adapt benchmark is still heavy workload. Like https://github.com/hokindeng/VMEvalKit/blob/main/vmevalkit/models/openai_inference.py

Can we let video model run on API like port 8800.

Then we only need the API to evaluate it ?

• Support API
• support hf local dir

[LukeLIN-web]

parallel evaluations for each task,

• docs , how to eval in parallel.

API could be like this
opencompass --models hf_internlm2_5_1_8b_chat --datasets demo_gsm8k_chat_gen -a lmdeploy

https://github.com/open-compass/opencompass/blob/main/opencompass/models/gemini_api.py

TurboMindAPI is streaming, so may differ from non streaming.

[LukeLIN-web]

interface like python run.py opencompass/configs/sdar.py -w outputs

Basically we need do like this, user fill a config file, then automatically download questions, generate_videos.py, bash script/lmdeploy_server.sh ,

[LukeLIN-web]

I will write main, refer opencompass/opencompass/cli/main.py
Finished generated datasets in main.

• run generate video in main.py