Python method to return a textual similarity score for two hadith units

[ahadith]

As we continue enriching our data we need to be able to reliably match the text of hadith units from different sources. At its core, this is a simple string matching task, and something very simple like edit distance or Levenshtein distance will work. However, because the strings of interest are digitized ahadith, it raises a few complications, and the method needs to be cognizant of those. We are NOT looking for advanced document similarity methods that measure overlap of semantic content using embeddings in vector spaces; this is a purely textual match. Here are some requirements for the method - we may add more as the use cases become clearer:

• The ability to specify whether we want to include the tashkil/diacritics in the similarity computation or not
• For words that don't match exactly, compare their roots and have that contribute to a slightly lower similarity score
• ignore spacing and punctuation differences
• strip out HTML tags

These methods will also then need to be extended for different data sources that have their own annotations and hooks in the text.

[suhailmahmood]

The goal of these functions is to determine if any two given hadith texts are actually the same hadith or not, I presume.

1. If the two versions are compared while keeping their diacritics if any (call them diacritical versions), the diacritics may be possibly used incorrectly in one (tampered), but yet the similarity score may be high enough (the diacritics may be used only scantily in the text, so they contribute only slightly to the (dis)similarity) for us to conclude that the two versions are same even though one version is tampered.
2. Also, if we strip the diacritics and then compare, we are essentially ignoring the differences in diacritics altogether, again possibly leading to the scenario described above - we will be concluding the two versions of the hadith are same even when one may be using diacritics very incorrectly.

Can these issues be disregarded? I am not sure whether the purpose of the functions to be developed will be served even if we disregard these issues, so let's discuss. Thanks.

[ahadith]

The upshot of my answer is that it is valuable to have a method that compares with diacritics and also without, because both are valuable use cases. Also we plan to use this method in conjunction with sequence information to make sure that two units are referring to the same hadith, for example by ensuring they are in the same book, chapter, and the similarity of the preceding and succeeding k hadith units. In summary, it all depends on how the method is used and what for.

A good reason to compare without diacritics is that not all printings or digitizations of hadith mutun have the same "level" of diacritics. Some have the bare minimum, and some have diacritics on almost every letter.

In engaging more with your point, "tampering" is not really a concern here because most diacritics are inferrable without ambiguity, and most variants of ahadith differ in far more than diacritics, in actual letters or words.

[TunifyBasic]

بسم الله الرحمان الرحيم امى بعد

this is a way of structuring data i thought of

each hadith is a page each page has factors to be ranked that contains scores and based on these we may use algorithms like:
BM25, BM25+ and TF-IDF to calc factors and their scores for each hadith (cron/bg job). kinda a search engine you could explore the yandex leaked source code within the Kernel/ directory you'll find interesting stuff or take a look at this it has factors summary with little descriptions and then when the user search something you do an inverse search for the best result (find out how to figure this based on calculated factors and parameters...).
OR you may wanna find an open source transformer model with weights to be used to do some sort of hadith words weighing (transformers cuz they tend to care about context in llm models) and do the same when user input something weight the searched text and do an inverse search by weights in the vector db to get the hadiths (you could store metadata with weights in vector dbs). i'm not an ai dev or vector db experienced dev and i don't know lot about it some i can't help lot.

that maybe a deep searching methods or use this algo i recently found: Levenshtien for lightweight search

صل وسلم على رسول الله محمد عليه الصلاة والسلام.