support parallel encoding ?

[qZhang88]

Notice that encoding is done one by one. Would it be faster if find all pieces first, then encode parallelly?

        while (re2::RE2::FindAndConsume(&input, *_regex, &piece)) {
            auto iter = _encoder.find(piece);
            if (iter != _encoder.end()) {
                last_piece_token_len = 1;
                ret.push_back(iter->second);
                continue;
            }
            auto tokens = _byte_pair_encode(piece, _encoder);
            last_piece_token_len = tokens.size();
            ret.insert(ret.end(), tokens.begin(), tokens.end());
        }

[sewenew]

Yes, you can have a try. Put all pieces into a vector, and then run OpenMPI or TBB threads to do the encoding stuff in parallel.

Regards