Using CTranslate with image data (im2text)

[mattndu]

I apologize if this is not the right place to ask a question.

I'm hoping you can provide guidance for adding support for im2text (https://github.com/OpenNMT/Im2Text) to this utility. I have a model trained in im2text, and I'd like to run it through this utility just for the forward pass in C++.

I imagine I'll need to write some code to load the image data and pass it to the utility? Just looking for advice/pointers -- Thanks!

[guillaumekln]

Hi,

Yes, you'll need to add a new API that supports image data and applies a similar processing than Im2Text does. For example, you could add a new type of Translator.

Most importantly, you'll also need to provide an implementation for every modules (or group of modules) introduced by the computation graph. This could be a good amount of work. You could make use of Torch implementation or cuDNN's API to achieve this even though the workflow uses Eigen. For example, here is an experimental module that calls cuBLAS' GEMM.