CLAMS "app patterns" or "task vocabulary"

[haydenmccormick]

Because

Views are rendered as appropriate tabs by attempting to ascertain the "abstract view type", e.g. OCR, ASR, NER, etc. This is done extremely naively, especially when checking for OCR views, since there is no standardized schema for what the output of a "thumbnail" view should look like. For example, these views could output:

• Alignments between generated TextDocuments and TimePoints in another view
• TimeFrames and TimePoints
• Just TimeFrames
• Just BoundingBoxes
• ... etc.

As a Band-Aid solution, the OCR abstract view type checker just pattern matches against known OCR/CV apps to determine if a view is a thumbnail tab. This is obviously not great for long-term development, since it would require the list to be manually updated with every new OCR app, and it wouldn't support third-party (non-CLAMS-developed) apps.

Done when

This could be addressed in a few different ways:

1. A standardized "abstract view type" as part of the metadata for MMIF views, defined by each CLAMS app separately
2. Allowing the user to add views as tabs manually and specify their abstract type from within the app
3. A much more advanced rule-based type checking system (if there is some consistent and maintained pattern)

There are most likely other solutions that may be more elegant/robust -- I'm leaving this issue open for further discussion on potential strategies/methods for dealing with this problem

Additional context

No response

[keighrim]

I believe this issue to be elevated to the SDK level.

---

I like the idea#1. We can pre-define "app patterns" in clams-python SDK, and each individual app can be subclass(es) of this app patterns (explicitly displayed in app metadata maybe). We can then re-use the app patterns for evaluation repo. Currently we have arbitrary "themes" used in evaluation subdirectories, so we can start from them, incrementally expand the list of patterns, while roughly maintaining the (loose) alignments to the evaluation themes.

@MrSqually I remember you had at some point a similar idea while working on evaluation automation. Can you post your notes from the previous discussion here as well?

[owencking]

"App patterns" seem to me like a good idea!

If the CLAMS project goes ahead with app patterns, then, I would add the "TimePoint captioner" pattern to the list.

That pattern outputs TimePoint-level captioning (TextDocuments with Alignment to TimePoints, with optional additional reference from the TextDocument (via the origin property?) to a TimeFrame that targets the relevant TimePoint. (This is the way that app-smolvlm2-captioner works.)

[keighrim]

Some starting points to get rough "groups" of apps based on their behavior

• how apps are used in evaluation (downstream consuper) names of tasks (or themes?) to evaluate aapb-evaluations#73
• how this idea of patterns can be used in appdir I/O type-based "search" for apps apps#120

---

Some rough sketch of implementation idea;

1. we start compiling patters (from source like the above links)
2. we build a vocabulary for each pattern
3. we either force
   1. an app to pick one of those names from the vocab (appType implementation)
   2. each view to be marked with one from the vocab (viewType implementation)

[keighrim]

"App patterns" seem to me like a good idea!

If the CLAMS project goes ahead with app patterns, then, I would add the "TimePoint captioner" pattern to the list.

That pattern outputs TimePoint-level captioning (TextDocuments with Alignment to TimePoints, with optional additional reference from the TextDocument (via the origin property?) to a TimeFrame that targets the relevant TimePoint. (This is the way that app-smolvlm2-captioner works.)

I don't think that's the case. Isn't that app work on TF level? That's what I see from the code.

Any way, image(s) --> caption should be a solid one for an app pattern. I left some notes in the meeting note. Let's discuss next week.